From nobody Sun Jun 5 22:55:21 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 86A991BD2A38 for ; Sun, 5 Jun 2022 22:55:33 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-22.consmr.mail.gq1.yahoo.com (sonic310-22.consmr.mail.gq1.yahoo.com [98.137.69.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LGX5c1K4Pz4g1v for ; Sun, 5 Jun 2022 22:55:32 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1654469724; bh=sasb86wspff5FDuGGe72NhziR5vKps1jeCRImm7RdLI=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=i+7wnONexTMGDjT3xUrzx4sLrN1G/33Vt050nUOwTgxgYNJWyl24EIvhFTChOLnYughF8iQKtjDcUZ3GLNvuaQHqPDkRTenoMnJRM3Z4Ep5o0wP0FE3DgGy+5SyEhtCpYoPlzId1QL7IXGeFK699pgb2cP/rf8kyUUihOVNLnWhYwFuIYI8Ak7CCQgR0RVSQAgOQ9WDWIv8FNn5pYrEpeZjClnf3K+Q8Sfq2eBn2XF7CBavhI+Xk5HROjr2ek4enQZKO/UkTqJUAhkIOevloqnuG+qmgzZO18D8dfXKbLSTFxf6pMKXtPzc8j6b/qjA4ZYC4vrDfEupEg4ISoNEQrA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1654469724; bh=NB6yj4cnya+zzYMiDomCh5TGWnoZU5xNNobQTncpqbR=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=iABdtjqyusgMCYmII5EhtTiU8S8oVu7ArkNW/ky0gG9CNzE+FTHogD6XWfljBE731VzS3MSsZ8POo5WAUCMpisdMSxWGBIr4lFNnzjM4UJGMegtefhsxqpgXYnJMpTmjf7Kp/PqK/Ph8pOD8v348vCLMQ4It6El3eQQAAwmDjpbLGPBrhVMlpyeR/IbklZLUMrXhwYFjPwjdQrRiCwKf4kpFq0Ds1226BtLzrU0FecvEuZhn5bIovohWuF2DCUMT5t+2hIan+GqzCswDztojyulp2ZLKog7coII3OVsCqF6olrW+8bdFQj8isywHqTD3756Obqqau19zgPrs34bKNQ== X-YMail-OSG: EUK.e.IVM1kMCpj75EqrNIPCS5VW6j5hlhbkpmkKUnUsPP_mEIT7sjt04xhXkgS jyvYMhnQvoreZ36PYN0z3TZe77kCaWAOx_VDNpQO78cgQ3WQcWWXf9BJymbzSa8JMQLUEhWg1Xx1 m8j6J95zMpPY9QNsPVIKKIkaHIdBWiu_sF0WYiayG5PYSFReDDfE0XALp8im0VipH8zYb5SlZ7t3 pynOZbm3QW1v2UGGozmV4m.jHzsILGADmD2yVxRzhP6bSPP09QRKnSIsZXHJkwhtKkulzJPaUEHP .CVgSQC9W78NvS.IEde_ANIUiz0YEO_Gs.0xDOrTr_FEp7gLunqhRVEHT.v1t.JJLCNOgeqhjN9C oitql1ScsNubwuuwxxcwCwG6l.ImAfKGl7BTlZRs1eZ8aRf0j8ob12yBKwtRQM7zFqvwnwQbqNGx k5pUuZAKHhI6nJxAbpXSoRvc2geUwBJREVTyrPGmCPlb5OU2yHXTPn7kEYyBSFOkuYtilWZV_Phf kZPM9maJDBkDTDMTymiJu9w5iCzzSpxLfnSEjjMPhKeIV.eC1R4xV.zyru8CaXF9iyWTSdhJkkX_ ZHXQhPTJuJZ23t4w9ij.1UGY.0nb_IaoY2FknUnrrf8HKDXj55SKFS8v0ylQMHEA593Mw8m1LBNU CW_V6DFPchViiXlE5QuqubUlbazMH1zjsjRrTXJIWQ7Huu_lJVzVAfBJGkMCdI1FADmc_GumUx_h N.JxAaiCzbsp1.NmJVQZjzxHUapIjF6fQkn.uRhvWtAV_ze7ufGPuAqnjT80CIya5YnOXTEsid7R ysy3JKOTJNBefdJWT4mRe4GuimrzD7_2gIVcIwLss1PoJijOy6DyWPXfFqDFYXxnpCLgaBvqeEwx m8Gd5Kb9hhQN0.LzmXySceU69lEvTTcWf2Jv09zkXuF89Q_kG3E79Yr9R7Eyc0eXTEWMShyvC7NO evAB5l2hitM_nnLNl9676rCL9gbph3pgw_tTxZ.bEkR00LnTZOjuvktU0iJAshppcEZ2jzZRr3d2 _DuDHRZjF60erAAHoroiBFeBaQvqVIpUpYhxsJT9wqjrI5GIRhlxzHo6Qr9Dcqd5HKn6.osmzQBu O0suwHNxKS4tdUFxbYZogQSbNBxVplhBe55br1Li_2720JWWMhRpz7mnUXvNgXGNWL7oiVqbxW6r mDVigFO4bx89y.tejoc8QEVO28UHnwo3K1t3fqMgcgoc9N0ihbFTtRRUvv94u3cogezPL924JOtm iwcovL2rRH_s5usnYTdsDTxD.lXnsuVz.kc9XvMzz3OFwvrPSO9ofYw4s_ynu00whh6DtjPFrFLA of6dJISLBTox0PDxzko93R4mDnrZgKXmtlHaupNPmzxPlZj78xCDW9uGNULKb84b3rsLVp1FHa9d qOahWmX4ucUK8FD.X4wLVB1B1CUht_3kjL7zTqIN0PHkZ1o6Ps72jVGszP4yPBEReSrvkqXhy4rR qZkVZvxPXtgFzy7ZRlfKTSZuRMzqm0HQOv1p5Q_10JIIoID4Oe9a4AHimSIEgDkRynEQPsIa50NG .XpC33IEenmDPCsExtb_.lXw2p9ZTp064KpmWgRbyNKWu4k_b9uH5wD.yKV1CsBdjY1vmlv9cnaG FPijN.TGFe.UoBiIFBteSqlWjI6d8JN1l6RcUwzGZX_zmQrqo1D12KvW_b2H5gsh6cS3UhdZhMMV 5urG0cwc8NTR6XHNjHpkIusMcUQK6rEcMNqq.wnPQjjnUWWeeAf3c7GOUMxlIrTm1mIR0q8rOMbh 7rixccyHRf5B14C1on7YY_yFU1eRLhkgdi5hbtv9.99dv8DyAbsVt5Pz2Eox9UB5tYZy79zptrgY k5_tqS2wuW1_qb6MkUUOhYwDJXrA18tt.B17h9U5BlUDhlLYPzz7SC8DckZ50RFfNNWqK7bzDpal 0pDO2TR7W30_PQiSOPJ4b7M3t9chfoHUP4aoIYW3D0Z5KHMuJF_hee8uOm88EWEkZ_j.F9SR.0px TWXrrGNPazKT6_WCn4rwayYxT_zyKVyPTHU.dwllIO87cnd0eax7AMIkYKyYRYLCY5ytLQX5w0N2 DEjiy9.tuC.ohdmBYwFjB110OdPlgyE4uPZKof8uEb2FDvpG66pu7zxIOwSabk_G8kKr3AbskV2z x7MJDqjmXyyhAfjgdnfE0Zxv8JWy4ZipZbJ_kjG2ym1QUt3Mj_Ejw0aIyfjkqIz5K8UK.jrvme4f xkX6EDh8dSowYalOP2OOiZn_C3LHNpyfSbPBxnC9KK8Biqu7Dhq.kAgaN0wWROvBcZxkluMkYEsV dpCgQ8ZQV4Cuo X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Sun, 5 Jun 2022 22:55:24 +0000 Received: by hermes--canary-production-ne1-799d7bd497-d2s2t (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 5427057928de86ed1763723dbffe0b59; Sun, 05 Jun 2022 22:55:23 +0000 (UTC) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: What can I learn about data that is staying paged out? (There is a more specific poudriere bulk related context given.) From: Mark Millard In-Reply-To: <27DD20B8-E19B-444B-BAE7-A09CD5390D2F@crossfamilyweb.com> Date: Sun, 5 Jun 2022 15:55:21 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <50163894-DCEB-4033-B555-698523E4D100@yahoo.com> References: <27DD20B8-E19B-444B-BAE7-A09CD5390D2F@crossfamilyweb.com> To: David Cross X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4LGX5c1K4Pz4g1v X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=i+7wnONe; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.148:from]; MLMMJ_DEST(0.00)[freebsd-hackers]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Jun-5, at 15:45, David Cross wrote: > On Jun 5, 2022, at 6:06 PM, Mark Millard wrote: >>=20 >> =EF=BB=BFOn 2022-Jun-5, at 12:42, Mark Millard = wrote: >>=20 >>> I have a poudriere bulk -a -c going on a 8 Gibyte >>> aarch64 system. top has been showing an occasionally >>> increasing swap usage but never any sizable decreases. >>> Over 5800 ports have built so far. The context is UFS >>> only. The system is running a non-debug build of main. >>>=20 >>> Part of the context is ( in /etc/sysctl.conf ): >>>=20 >>> vm.swap_enabled=3D0 >>> vm.swap_idle_enabled=3D0 >>>=20 >>> Also ( in /usr/local/etc/poudriere.conf ): >>>=20 >>> USE_TMPFS=3D"data" >>>=20 >>> poudriere's TMPFS reports normally total under 128 >>> KiBytes across the 4 builders. >>>=20 >>> For reference, example figures . . . >>>=20 >>> A top variant shows: >>>=20 >>> Swap: 30720Mi Total, 306816Ki Used >>>=20 >>> vmstat -s shows: >>>=20 >>> 78152 swap pager pages paged out >>>=20 >>> Note: (78152*4096)/1024 =3D=3D 312608Ki >>>=20 >>> So nearly all of the "swap pager pages paged out" >>> pages are still sitting out in the used swap/paging >>> space. Thus, the usage is not held by user processes >>> or is held via very long running processes or is >>> not directly tied to user processes --or some mix. >>>=20 >>> The variant of top reports never having observed >>> more than: 6658Mi MaxObs(Act+Wir+Lndry). >>> ("MaxObs" is short for "Maximum Observed".) >>> Such high usage is for a bounded time, long past >>> at this point. (Until some combination of port >>> builds ends up active that uses such.) >>>=20 >>> So I'm curious: >>>=20 >>> What can I learn about the data that is staying >>> paged out (and is gradually growing)? How can I >>> learn it? >>>=20 >>>=20 >>> Other notes: >>>=20 >>> The poudriere jail being built is: >>>=20 >>> # poudriere jail -jmain-CA7-bulk_a -i >>> Jail name: main-CA7-bulk_a >>> Jail version: 14.0-CURRENT >>> Jail arch: arm.armv7 >>> Jail method: null >>> Jail mount: /usr/obj/DESTDIRs/main-CA7-poud-bulk_a >>> Jail fs: =20 >>> Jail updated: 2022-05-23 02:21:24 >>> Jail pkgbase: disabled >>>=20 >>> (Just in case the armv7 jail usage or the null method >>> or such is important to the issue.) >>=20 >> Hmm. systat -swap reports a toal for the Devices/Paths Used >> that is somewhat less than the total for what reports for the >> Pid . . . Total figures (not the Pid Swap figures!): >>=20 >> # systat -swap >> /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 = /10 >> Load Average |||||||| =20 >>=20 >> Device/Path Size Used |0% /10 /20 /30 /40 / 60\ 70\ 80\ = 90\ 100| >> gpt/CA72USBswp14 14G 150M >> gpt/CA72USBswp16 16G 150M >> Total 30G 300M >>=20 >> Pid Username Command Swap/Total Per-Process Per-System >> 1453 root nfsd 1M / 15M 9% 0% >> 1451 root mountd 1M / 15M 7% 0% >> 1481 root sshd 912K / 20M 4% 0% >> 1406 root ntpd 740K / 27M 2% 0% >> 1513 root login 724K / 14M 5% 0% >> 1514 root sh 656K / 13M 4% 0% >> 342 _dhcp dhclient 516K / 13M 3% 0% >> 1363 root rpcbind 448K / 13M 3% 0% >> 1454 root nfsd 400K / 12M 3% 0% >> 341 root dhclient 380K / 13M 2% 0% >> 1341 root syslogd 324K / 12M 2% 0% >> 1505 root getty 292K / 12M 2% 0% >> 1510 root getty 292K / 12M 2% 0% >> 1511 root getty 292K / 12M 2% 0% >> 1512 root getty 292K / 12M 2% 0% >> 1509 root getty 292K / 12M 2% 0% >> 1508 root getty 292K / 12M 2% 0% >> 1507 root getty 292K / 12M 2% 0% >> 1506 root getty 288K / 12M 2% 0% >> 1135 root devd 272K / 11M 2% 0% >> 338 root dhclient 264K / 13M 2% 0% >> 1 root init 244K / 11M 2% 0% >> 1486 root cron 188K / 13M 1% 0% >>=20 >> I'm, Still looking for a clear indication of what >> most of the 300 MiBytes or so of swap/paging space >> is in use for. >>=20 >> =3D=3D=3D >> Mark Millard >> marklmi at yahoo.com >=20 > My guess is that this is swapped out buffer cache (files).=20 >=20 Thanks for the idea. Know how I could find an approximation to the amount of paged out buffer cache to see about how much of the ~300 MiBytes it might explain? Mark =3D=3D=3D Mark Millard marklmi at yahoo.com