From nobody Sat Jun 11 22:42:24 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6329283AA87 for ; Sat, 11 Jun 2022 22:53:39 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-21.consmr.mail.gq1.yahoo.com (sonic305-21.consmr.mail.gq1.yahoo.com [98.137.64.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LLCmf3HMWz3mLG for ; Sat, 11 Jun 2022 22:53:38 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1654988010; bh=enaBz31ilzw1rNd9WPuRuQtWH0f+Ms7H8tg3udok9k0=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=hG0CH07wGj18GnVC5gtlbhL7o8Qid1NCJGGxPSjnzYwYC7SABiqo+V7d6WxaArUoRtUERYl7f0Ij6Y8wsCHaiUYr1g8yMlDk7qsXdo6XbCaQ9ToytvBVdkwshdNat5oj5/CJQ+sFaWxsb1Kd982+aZc3IdbytiPk+M2Oy++CPX9YgXBNtSM8IBptka07JAWWt23CvrkYoMGxZAToM7YNG/5om75oUqrHVay/NIaNi1wGbyXYqP4FkHjU8e/2/vc0q4wzUF3Xyfi/t0GMGszz9FkHqkaMDdwjnWfVGLt+9g2suGghOSzPG/3Gv0B0Ox5jVdudC6pY/GRYUbrCuflZJw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1654988010; bh=qSFq+RSYnTxEdHlGsEKnInBhLWhhyMK8p0LRCE9wSUv=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=aSi9ytXHUCfkL5+CsGkX1CnwQMsb4Pr3WwUrmsH0hDAhoAvLnfu0NTbqdqA1zmgeYfSKJLwbMcE4/fHNL71Xglp8QLEoQJ5Id0WF6MYS22RA89FgYE0x8xKpLhySFrgHJEMEXkRSWUpnaFP/H5ZSMlw+Quplybkm9fFGT+O6ohlGwGXdfhhTaRD88LuBkxfIXDhQVYIhjVWdlPC+BBg62Z5J4BWQzlmKl16xrM71bV3hP5DfLWpFsfgYsRpUzqOuj6uyIAlHe9zXwn48biwIazFyp7+k9MEbbjqdPST+FHYVTBVs35AWo6WhukCwA7lhOujXpEAr3K1nJDS/8U/OKw== X-YMail-OSG: mfsRxTwVM1k6hPv7EbfAAYaIVdFh8.TlWzF0_8oJ2Y2ng4KuEv14EjAHWpY1E8F NmKYm89z7F.vSpQ2bH9SDudzKbB8okNZRCsmk3fpZdhIKml5jdVF0DkGIvwNTOn70HRWuwjP7I2m w_BJzB3rpMASKaVlI4P7ZN4_x35MUqRCUqTKuok37a45S80dqoMLvYJBbI_K5nCTwg9EnAdG2CTG Gwaz0JAfeBRN0OvE3s0BAM7KFnC3NcqsGWHB3D1F1IqvsBv.z4Ug3NZ_k6Crcx4quZvVBYflxLx7 GxffHj3z.hJ7R9rl_HdSz9wtWbrSCdAYPdGH7dD9Ujkg3aITtGpnVIr4dk0qi10qUueKgmRx0JSe ETulACPAZqkFqZKohDxsYm9ItI644sSDBO0u1s5712W4UYgXiAp1Kqx7Mqyx4.O81E3mmgLjh1au FpwHZmxHWad_trpnLs.B0nmgxFfo5VKB4eusXSHHCHHSR2KEY1JbUNwBVhUxzc8B4K4gXQ4gOA1U fnv6vLEPJ0oithtPwnwqopxclpR0SCTkpZV0IvwXtGbnbRzHsNUHC87FeEIf7KurlGjaNC3CCBn6 j172xFEa_cANbaabQ0jVVG3W9j8I9VKXvEwGgo_RLSZltcB0_pR8SLt8XGZeZVjmq3ud6UVc9MiZ WKUAAeZywgiSxndsX7R4wE3_dduxunq1.03IDVzYrfjkrWCzfyV9by3ekf7LLfEqBDmH73ad6cU4 Eo6bhUIzxt.el1KKw2BpOm8FbbjxT7_Nfc8Lq7h.LDFuOnrlcOrZrZdjgrEHeD_8KH3PFS0qx79n lOwX889mg_CCkcqrPV8qBkWwQiOsFBAerX9Lo8dK1uvHG.6m9xn0vjYxemu1kEc50IdgKq8cTq_M JIRtKsvt7I0GRGxejSL16JSksrd6QRvHvn9dx5.qgkIPfU1IjSKHD.XO2DNe.Pv4KkGXIHiVCaEA 7l2OCdkfIKiZVslxdSnCV8zqvt4ldWHw4B0Sche.xdsLV2xL4561w_BR68tYrYBZijeSFEuSKpKX qOL0bXKdgF1zDAqv8qcc7pCUSg25dDsPmcIiJvckJe5wt_lwiZgy33dQpiV5PDxoaHX8WzcRqyM6 Sn285HyL1gHB.gwU.VRMecLTgRhLiX2aRXIcDhry1LKK89DI2z.ykyWPunA2Ca7vEpF.sCkJJbaN ZXQs5_Z2LYUgH.zIxVLy4vluH_z0r69QtlY7U5iQmmUaSBn2osULbq24EljebPsoj2LXXr4.QCM8 pYOxsRYcuV_1grFD9dMeTpeRbNC2SI1ASx8.SMy1jeR_fzTNzgzmzAfOlah11pc6V5qb_jV9sIwL zvUNnEXXu6GNMW31d5Qe63A7eQQ.XZ19FuBiTG6MoyyaNOpc7RgJ.bBjQHRE9NuvvHYuS0imNYYg QrccOvNMdpwc090nsfpRSixOtytYG0n7aQSJfCk04dO4V3aVyEDpbHR2ubFfmuL3xDu5AAti3qoR RS82wStRYvR670IxRVD65liJC9wKx7aGdE4I8gIz9GZ.6Fe1qH2shCfX_AxUMUn5Qwv0sikFw2ZU nD3TigrS5EHgp1UnW43uXz5YdEFVwRmRErJp1EFI5PwjuAcqrmFHM2CRRYLcwrYFhDeCBxWMCp3R 0SQCKf3SrkMO.C1NC.WqCpJCKwIPSzxFy_EuwbDdtOMpSV7J4KOeSOzBjVIgqufi8JGxs.1SWUrK vmbTQQM4umvooPV.OwMUWpnY300oEisjI.Igwgi3hIj5eEVCUa_3ppUKShEGHk5zRAfi7tnBq3t8 ZjeKSScVDUcv0O4P0T8dx0cbHeebFIgeO.fTvhecnX_CPYIWQ_orSGHPNaX56ol04RyoxiIrkdqb T1FQ0qF0wp2Yyt8DusPePzmE3gxT5vTCkcL9jWvSOHrPJg3CEXiVywaTYIy8X.cmXeNUXbPHo2Gy a1gP12ICThRTHXH_x8iowFDdll1DWx_VjnlbQBBJFk1Asjd5Y.hT8dmd0P127yQ-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Sat, 11 Jun 2022 22:53:30 +0000 Received: by hermes--canary-production-ne1-799d7bd497-px24v (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 9a685699436fd6f479267466fadaf335; Sat, 11 Jun 2022 22:42:25 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: What can I learn about data that is staying paged out? (There is a more specific poudriere bulk related context given.) From: Mark Millard In-Reply-To: <6EA27152-3355-4356-B246-A083F31452F2@yahoo.com> Date: Sat, 11 Jun 2022 15:42:24 -0700 Cc: Daniel Ebdrup Jensen , David Cross Content-Transfer-Encoding: quoted-printable Message-Id: <840FA150-5623-4BE7-A289-015E7C14436C@yahoo.com> References: <573B8B0C-5209-459D-98AD-EE92DDA4DF83@yahoo.com> <6EA27152-3355-4356-B246-A083F31452F2@yahoo.com> To: FreeBSD Hackers X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4LLCmf3HMWz3mLG X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=hG0CH07w; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-1.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; NEURAL_SPAM_SHORT(1.00)[1.000]; RCPT_COUNT_THREE(0.00)[3]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.84:from]; MLMMJ_DEST(0.00)[freebsd-hackers]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N I got evidence finally. An example is the sequence: # swapoff /dev/gpt/CA72USBswp14 /dev/gpt/CA72USBswp16 # swapon /dev/gpt/CA72USBswp14 /dev/gpt/CA72USBswp16 (Wait for a time letting some "staying paged out" space accumulate.) # du -xsm /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/ 319 /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/ (Wait for a time letting some more "staying paged out" space accumulate.) # du -xsm /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/ 307 /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/ (The size decreases seem to roughly track the sustained Swap Used increases. [There is some other paging activity but it is not a lot.]) # swapoff /dev/gpt/CA72USBswp14 /dev/gpt/CA72USBswp16 # du -xsm /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/ 379 /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/ (Apparently, du -xsm does not count the paged out pages for tmpfs but the pages show up when forced back into RAM.) # swapon /dev/gpt/CA72USBswp14 /dev/gpt/CA72USBswp16 # du -xsm /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/ 379 /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/ (I expect that when I see the swap usage grow in a staying paged out manor, I'll find that this -xsm figure will also have decreased on the retry in the new context.) The primary space in question is actually in a subdirectory: Taking a quick estimate of the file count in . . ./ref/.p/var/cache/ I get: # ls -Tla = /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/var/cache/ | = wc 82146 821452 7942485 So something like 82144 files ( ignoring . and .. ). # ls -Tla = /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/var/cache/ | = more total 301234 drwxr-xr-x 2 root wheel 5257152 May 28 02:31:26 2022 . drwxr-xr-x 4 root wheel 128 May 28 01:56:42 2022 .. -rw-r--r-- 1 root wheel 2 May 28 02:26:40 2022 = builder_active%01 -rw-r--r-- 1 root wheel 2 May 28 02:31:24 2022 = builder_active%02 -rw-r--r-- 1 root wheel 2 May 28 02:31:26 2022 = builder_active%03 -rw-r--r-- 1 root wheel 2 May 28 02:31:26 2022 = builder_active%04 -rw-r--r-- 1 root wheel 18 May 28 01:56:51 2022 = originspec-pkgname%accessibility_accerciser -rw-r--r-- 1 root wheel 19 May 28 01:56:51 2022 = originspec-pkgname%accessibility_at-spi2-atk -rw-r--r-- 1 root wheel 20 May 28 01:56:51 2022 = originspec-pkgname%accessibility_at-spi2-core -rw-r--r-- 1 root wheel 11 May 28 01:56:51 2022 = originspec-pkgname%accessibility_atk -rw-r--r-- 1 root wheel 13 May 28 01:56:51 2022 = originspec-pkgname%accessibility_atkmm -rw-r--r-- 1 root wheel 17 May 28 01:56:51 2022 = originspec-pkgname%accessibility_caribou . . . -rw-r--r-- 1 root wheel 17 May 28 01:56:55 2022 = pkgname-originspec%zutils-1.11 -rw-r--r-- 1 root wheel 15 May 28 02:03:32 2022 = pkgname-originspec%zxfer-1.1.7 -rw-r--r-- 1 root wheel 14 May 28 02:03:00 2022 = pkgname-originspec%zxid-1.42_1 -rw-r--r-- 1 root wheel 19 May 28 02:04:03 2022 = pkgname-originspec%zxing-cpp-1.3.0 -rw-r--r-- 1 root wheel 12 May 28 01:59:41 2022 = pkgname-originspec%zydis-3.1.0 -rw-r--r-- 1 root wheel 18 May 28 01:57:13 2022 = pkgname-originspec%zynaddsubfx-3.0.6,2 -rw-r--r-- 1 root wheel 9 May 28 02:02:07 2022 = pkgname-originspec%zyre-2.0.1 -rw-r--r-- 1 root wheel 14 May 28 01:59:41 2022 = pkgname-originspec%zziplib-0.13.72_1 -rw-r--r-- 1 root wheel 14 May 28 02:03:00 2022 = pkgname-originspec%zzuf-0.13_1 -rw-r--r-- 1 root wheel 13 May 28 01:56:45 2022 = ports_metadata%top_git_hash -rw-r--r-- 1 root wheel 4 May 28 01:56:49 2022 = ports_metadata%top_unclean Each file seems to have a 4096 Byte page used, making for something around 320 MiBytes. Much of it not being heavily used, some of it is gradually paged out to make room for other things. # du -xsm = /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/var/cache/ 326 = /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/var/cache/ (Still no reported swap usage.) For reference: # du -xsAm = /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/var/cache/ 46 = /usr/local/poudriere/data/.m/main-CA7-bulk_a-default/ref/.p/var/cache/ =3D=3D=3D Mark Millard marklmi at yahoo.com