From nobody Sun Jul 16 15:59:31 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R3qfY09cKz4nFJk for ; Sun, 16 Jul 2023 15:59:49 +0000 (UTC) (envelope-from bc979@lafn.org) Received: from mail.sermon-archive.info (sermon-archive.info [47.181.130.121]) by mx1.freebsd.org (Postfix) with ESMTP id 4R3qfW6xvPz4Zd0 for ; Sun, 16 Jul 2023 15:59:47 +0000 (UTC) (envelope-from bc979@lafn.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of bc979@lafn.org designates 47.181.130.121 as permitted sender) smtp.mailfrom=bc979@lafn.org; dmarc=none Received: from smtpclient.apple (mini [10.0.1.251]) by mail.sermon-archive.info (Postfix) with ESMTPSA id 4R3qfP2qhyz2fjX8; Sun, 16 Jul 2023 08:59:41 -0700 (PDT) Content-Type: text/plain; charset=utf-8 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.7\)) Subject: Re: OT: how to make a ssh/showmount usable with no internet From: Doug Hardie In-Reply-To: Date: Sun, 16 Jul 2023 08:59:31 -0700 Cc: Aryeh Friedman , FreeBSD Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: References: To: User & X-Mailer: Apple Mail (2.3731.600.7) X-Virus-Scanned: clamav-milter 1.0.1 at mail X-Virus-Status: Clean X-Spamd-Result: default: False [-2.70 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; MLMMJ_DEST(0.00)[freebsd-questions@freebsd.org]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:5650, ipnet:47.181.128.0/18, country:US]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[lafn.org: no valid DMARC record]; TAGGED_RCPT(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4R3qfW6xvPz4Zd0 X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N > On Jul 16, 2023, at 03:03, User & wrote: >=20 > Le vendredi 14 juil. 2023 =C3=A0 14:48:50 (-0700), Doug Hardie =C3=A0 = =C3=A9crit: > Hello, >=20 >> I run unbound (full version) on one of my servers. All my systems = are entered into it's database as local-data in local-zone. All the = systems use that system for DNS. If the internet is down, local DNS = still works. I don't the RDNS entries and haven't needed them. If they = are necessary, unbound can do that also. >=20 > Do you mean that there is several version of unbound and, for example = the pkg > local_unboumd did not provide de fuill version of unbound? That = intersting is it > the case. There is a reduced version of unbound in the base system. I tried it = ane it does not have all the functionality of the port version of = unbound. That is the full version. -- Doug From nobody Sun Jul 16 19:07:35 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R3vqM38Pwz4n2bc for ; Sun, 16 Jul 2023 19:07:43 +0000 (UTC) (envelope-from robert@webtent.org) Received: from mx3.webtent.net (mx3.webtent.net [208.38.145.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4R3vqL2XLPz4Kpt for ; Sun, 16 Jul 2023 19:07:42 +0000 (UTC) (envelope-from robert@webtent.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=webtent.org header.s=201611 header.b=oel2C6Iz; spf=pass (mx1.freebsd.org: domain of robert@webtent.org designates 208.38.145.5 as permitted sender) smtp.mailfrom=robert@webtent.org; dmarc=pass (policy=reject) header.from=webtent.org Received: from localhost (localhost [127.0.0.1]) by mx3.webtent.net (WebTent ESMTP Postfix Internet Mail Exchange) with ESMTP id 52037D7A6B for ; Sun, 16 Jul 2023 15:07:39 -0400 (EDT) Received: from mx3.webtent.net ([127.0.0.1]) by localhost (mx3.webtent.net [127.0.0.1]) (maiad, port 10024) with ESMTP id 68636-06 for ; Sun, 16 Jul 2023 15:07:38 -0400 (EDT) Received: from [192.168.1.50] (ns2.webtent.net [144.129.73.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: robert@mx3.webtent.net) by mx3.webtent.net (WebTent ESMTP Postfix Internet Mail Exchange) with ESMTPSA id 8D60AD7A70 for ; Sun, 16 Jul 2023 15:07:38 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=webtent.org; s=201611; t=1689534458; bh=JfGXVC+adjCmT+DENesDlIaKChSzY3DScBqctLmmYck=; h=To:From:Subject:Date; b=oel2C6Iz4EXV2Xoc7KHfvYifU7dUNTvez7UywXw3/rHBHvzg4JYgLBdHgdcZQKSgw mPzDYkm24fkSflbo0Bura/r+cTpM14VqcvGCLFNyyI8wNpnKWs0S+cU3sgwzF15inw UVvNPDNUIGVhBEEjDRsQSMWTmIOPrXr9FJ9EstcHJmJg0ElJRfoNOm7NEOt19BazzL 4vOKTmz8RTGkzPG2Sqkyrfk09z3EAF4xR2uqHFn53hXs+GV310ySydbwrWLcLCqKuv qOMZdbXwsYUZ7HL8sqP3Spq+mxjdPundP9qlEDHOfKDJiHauYjFk4Yau8LKdk2NWSQ hfouyTOTrqosw== To: FreeBSD From: Robert Fitzpatrick Subject: Apache log rotation Message-ID: Date: Sun, 16 Jul 2023 15:07:35 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 PostboxApp/7.0.60 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Virus-Scanned: WebTent Mailguard 1.0.4_3 X-Spamd-Result: default: False [-4.65 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.95)[-0.948]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DMARC_POLICY_ALLOW(-0.50)[webtent.org,reject]; R_DKIM_ALLOW(-0.20)[webtent.org:s=201611]; R_SPF_ALLOW(-0.20)[+mx]; RCVD_IN_DNSWL_MED(-0.20)[208.38.145.5:from]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[webtent.org:+]; MLMMJ_DEST(0.00)[freebsd-questions@freebsd.org]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; ASN(0.00)[asn:16724, ipnet:208.38.128.0/18, country:US]; FREEFALL_USER(0.00)[robert]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4R3vqL2XLPz4Kpt X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N Ever since I upgraded from FreeBSD 13.1 to 13.2, each week when the logrotate happens for my Apache logs, apache dies. I see this in the log... Jul 16 02:00:00 sites kernel: pid 84639 (httpd), jid 0, uid 0: exited on signal 11 (core dumped) If I use gdb on the core output file, I get no stack... root@sites:~ # gdb /httpd.core (gdb) bt No stack. My suspicion after looking around, is this happens during log rotation since it happens exactly at 2am each Sunday morning. I have my logs listed in newsyslog.conf as this .... /var/log/httpd-access.log               644  14    *    $W0D2  JC /var/run/httpd.pid 30 /var/log/httpd-error.log                        644  14    *    $W0D2 JC /var/run/httpd.pid 30 This has worked for a long time up until the last 4 weeks, ever since I upgraded FreeBSD to 13.2 only from 13.1 and all packages using pkg. Not sure where I got the idea for 30 for the sig num, I see posts now suggesting this should be 1? Appreciate any help to determine what needs changed or how to track down the issue. Are there other ways to rotate apache logs? -- Robert From nobody Sun Jul 16 19:37:13 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R3wTR0zFsz4nHd3 for ; Sun, 16 Jul 2023 19:37:15 +0000 (UTC) (envelope-from dalescott@shaw.ca) Received: from omta002.cacentral1.a.cloudfilter.net (omta002.cacentral1.a.cloudfilter.net [3.97.99.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R3wTQ6PPWz3DWK for ; Sun, 16 Jul 2023 19:37:14 +0000 (UTC) (envelope-from dalescott@shaw.ca) Authentication-Results: mx1.freebsd.org; none Received: from shw-obgw-4003a.ext.cloudfilter.net ([10.228.9.183]) by cmsmtp with ESMTP id L2SHqVA346NwhL7YQqGwvg; Sun, 16 Jul 2023 19:37:14 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=shaw.ca; s=s20180605; t=1689536234; bh=rRQcetJroi6C2f869UzFxF5mvU7J7GhrLgo004a8gjg=; h=Date:From:To:Cc:In-Reply-To:References:Subject; b=G2ieu2tuzsuUefsaKb3O+rbB6idpgbPaORmCK+iitS7KbY/UDuIsqpFFWtYGCTCYI o+iOloyzsgcJweDmEXWsACgXVJ+rhEZ156RwMgiCQQvOneM3ZciQ0xk8dtoVRIrYsF Jm8vOEWhkaOsPPnyNI8lmzc7RAxIxJLgidm37nMs/JSYhCFskiEMYLpDqGPXiWtvGh gk9l/kNCQ+BIrihrLghmIjxuTG7b2x51wogqPWV9FIjYRv+vL/FmOlyoeBfKo29Cre 1Z24HM2HDeKYBN3u2F2QPdrVKtpyp4QSsAj1dYOH9dhqKjRsSaWwN/4RWjuqN4i1QZ ZozBzxJ5acv+A== Received: from cds220.dcs.int.inet ([64.59.134.6]) by cmsmtp with ESMTP id L7YPqTVJjcyvuL7YPqROSw; Sun, 16 Jul 2023 19:37:14 +0000 X-Authority-Analysis: v=2.4 cv=VbHkgXl9 c=1 sm=1 tr=0 ts=64b446ea a=9zdlX7M534QhL7mOrorEvQ==:117 a=FKkrIqjQGGEA:10 a=RDteU5_PNoYA:10 a=IkcTkHD0fZMA:10 a=DLsStemsAAAA:8 a=6I5d2MoRAAAA:8 a=5mcg790sAAAA:8 a=jU4qhlNgAAAA:8 a=tYQj8rC9rwZfjgXUId0A:9 a=QEXdDO2ut3YA:10 a=Bxhjgy3MuvLX_Vf20tGr:22 a=IjAAo6Nm_YfFKYjqs926:22 a=IjZwj45LgO3ly-622nXo:22 a=uQeDYW1NI25gHNlrW_eK:22 Date: Sun, 16 Jul 2023 13:37:13 -0600 (MDT) From: Dale Scott To: Robert Fitzpatrick Cc: FreeBSD Message-ID: <254366756.114175105.1689536233529.JavaMail.zimbra@shaw.ca> In-Reply-To: References: Subject: Re: Apache log rotation List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [174.0.43.39, 174.0.43.39] X-Mailer: Zimbra 8.8.15_GA_4508 (ZimbraWebClient - GC114 (Linux)/8.8.15_GA_4508) Thread-Topic: Apache log rotation Thread-Index: 4KW4srTu4F3zj+OrzI0aw5TqLp5ciQ== X-CMAE-Envelope: MS4xfMGSWHuozsrLqT5AOoPaueFe+6u+X2WY2/aUTcLegmWQSsR0qFkcUbX1rng0bhF9H6Y8eiYwsyl8JYyoXyC1PWF6mWw7w9Nc03Zk7viL3NXIsR1Qr4L0 9kIzP57ohEgf61CyanoxvqgKgy+skVpr2VzqmI4xz86SJcnTrzS9hPuvKlH9Yx7UgJ8DywIM4kKOp8rxGrVSjGGUZGf2RpT7k8cmX8liUtrDMQfUoocGg9Oo X-Rspamd-Queue-Id: 4R3wTQ6PPWz3DWK X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:16509, ipnet:3.96.0.0/15, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N ----- Original Message ----- > From: "Robert Fitzpatrick" > To: "FreeBSD" > Sent: Sunday, July 16, 2023 1:07:35 PM > Subject: Apache log rotation > Ever since I upgraded from FreeBSD 13.1 to 13.2, each week when the > logrotate happens for my Apache logs, apache dies. I see this in the log... > Are you using phpopcache? There has been a known issue for a while with a conflit between FreeBSD ASLR and opcache. ASLR was included but not enabled in FreeBSD 13.1, but is enabled in 13.2-RELEASE - hence why we both started seeing issues after upgrading to FreeBSD 13.2. For details see https://forums.freebsd.org/threads/apachectl-graceful-causes-sigsegv-with-latest-php-8-1-x-and-php-8-2-x.88814/#post-611560 --- Dale Scott, P.Eng. Engineering Manager | PLM | NPI | ERP www.dalescott.net https://ca.linkedin.com/in/dalescott From nobody Mon Jul 17 12:19:12 2023 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R4LlB0ZGMz4mtp1 for ; Mon, 17 Jul 2023 12:20:38 +0000 (UTC) (envelope-from freebsd-database@pp.dyndns.biz) Received: from keymaster.home (ns1.xn--wesstrm-f1a.se [81.4.102.176]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "keymaster.home", Issuer "keymaster.home" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R4Ll91VZFz3rvr for ; Mon, 17 Jul 2023 12:20:37 +0000 (UTC) (envelope-from freebsd-database@pp.dyndns.biz) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of freebsd-database@pp.dyndns.biz has no SPF policy when checking 81.4.102.176) smtp.mailfrom=freebsd-database@pp.dyndns.biz; dmarc=none Received: from [192.168.254.0] ([192.168.254.0]) by keymaster.home (8.17.1/8.17.1) with ESMTP id 36HCJCc1004402 for ; Mon, 17 Jul 2023 14:19:13 +0200 (CEST) (envelope-from freebsd-database@pp.dyndns.biz) Message-ID: <334c1023-8283-2c41-f284-ccc44bae6d0a@pp.dyndns.biz> Date: Mon, 17 Jul 2023 14:19:12 +0200 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: Apache log rotation To: questions@freebsd.org References: Content-Language: en-GB From: =?UTF-8?Q?Morgan_Wesstr=c3=b6m?= In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [0.03 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; AUTH_NA(1.00)[]; HFILTER_HELO_IP_A(1.00)[keymaster.home]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_MIXED_CHARSET(0.83)[subject]; HFILTER_HELO_NORES_A_OR_MX(0.30)[keymaster.home]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[questions@freebsd.org]; ASN(0.00)[asn:198203, ipnet:81.4.100.0/22, country:NL]; R_SPF_NA(0.00)[no SPF record]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DMARC_NA(0.00)[pp.dyndns.biz]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; BLOCKLISTDE_FAIL(0.00)[81.4.102.176:server fail]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[questions@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4R4Ll91VZFz3rvr X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N On 2023-07-16 21:07, Robert Fitzpatrick wrote: > Ever since I upgraded from FreeBSD 13.1 to 13.2, each week when the logrotate > happens for my Apache logs, apache dies. I see this in the log... > I don't have an answer for your specific problem but just wanted to suggest an alternative solution. Have you looked at sysutils/cronolog? Instead of relying on logrotate and restarting the apache process you simply pipe the logs to cronolog which splits them according to a pattern you define. Apache will run uninterrupted and logrotate won't be used at all. Snippets from my own conf files that splits the logfiles per year-month (note the pipe sign): ErrorLog "|/usr/local/sbin/cronolog /var/log/apache24/httpd-error.%Y%m.default.log" CustomLog "|/usr/local/sbin/cronolog /var/log/apache24/httpd-access.%Y%m.default.log" combinedio Regards Morgan From nobody Mon Jul 17 15:59:41 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R4Rc51g6hz4ncCF for ; Mon, 17 Jul 2023 15:59:49 +0000 (UTC) (envelope-from robert@webtent.org) Received: from mx3.webtent.net (mx3.webtent.net [208.38.145.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4R4Rc46FS8z40Ls for ; Mon, 17 Jul 2023 15:59:48 +0000 (UTC) (envelope-from robert@webtent.org) Authentication-Results: mx1.freebsd.org; none Received: from localhost (localhost [127.0.0.1]) by mx3.webtent.net (WebTent ESMTP Postfix Internet Mail Exchange) with ESMTP id CCFC3D7997; Mon, 17 Jul 2023 11:59:45 -0400 (EDT) Received: from mx3.webtent.net ([127.0.0.1]) by localhost (mx3.webtent.net [127.0.0.1]) (maiad, port 10024) with ESMTP id 20402-07; Mon, 17 Jul 2023 11:59:45 -0400 (EDT) Received: from [192.168.1.50] (ns2.webtent.net [144.129.73.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: robert@mx3.webtent.net) by mx3.webtent.net (WebTent ESMTP Postfix Internet Mail Exchange) with ESMTPSA id A30ACD793E; Mon, 17 Jul 2023 11:59:44 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=webtent.org; s=201611; t=1689609585; bh=sCEQtEj/qUhnFCgyXPgv/S337azZPDcoroB5jVv07qY=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=NQLQ1Flz10XXjtpTQkjJYcEg/EI6M1ckwM8w3/8dUf1ykXdS7uw8w2FaIiElewIv4 g60BGPboGUoaUMlOJaO+UwpvY3WigCWX8Wht/Yqd7YqyqokxvABSS2Mq7BZERAzSIM g4yrPetnfzxvY8uyXaRrW16ESxslL1TbW67x9XbkZeIWhyHuS1+juxSSGRBtINn18C EGidWisqyQiAdp4Pu8tRyvRNbH87/14xGLMz8nTWVyU31Ej2WnmMfrPG+iTRZPXkC/ M3+TGjB134Fud27rLLSxiWeo8/+xh2QOWcpUFqmJ9MeO7fISggRIwJARPoOTOi+2bs B81dESoSCLQsw== Subject: Re: Apache log rotation To: Dale Scott Cc: FreeBSD References: <254366756.114175105.1689536233529.JavaMail.zimbra@shaw.ca> From: Robert Fitzpatrick Message-ID: <0fe6df30-a669-24cd-386a-d57edf6f4fee@webtent.org> Date: Mon, 17 Jul 2023 11:59:41 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 PostboxApp/7.0.60 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 In-Reply-To: <254366756.114175105.1689536233529.JavaMail.zimbra@shaw.ca> Content-Type: multipart/alternative; boundary="------------3163486D09BB5567F9ED7923" Content-Language: en-US X-Virus-Scanned: WebTent Mailguard 1.0.4_3 X-Rspamd-Queue-Id: 4R4Rc46FS8z40Ls X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:16724, ipnet:208.38.128.0/18, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --------------3163486D09BB5567F9ED7923 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit > Dale Scott > Sunday, July 16, 2023 3:37 PM > ----- Original Message ----- >> From: "Robert Fitzpatrick" >> To: "FreeBSD" >> Sent: Sunday, July 16, 2023 1:07:35 PM >> Subject: Apache log rotation >> Ever since I upgraded from FreeBSD 13.1 to 13.2, each week when the >> logrotate happens for my Apache logs, apache dies. I see this in the log... >> > Are you using phpopcache? There has been a known issue for a while with a conflit between FreeBSD ASLR and opcache. ASLR was included but not enabled in FreeBSD 13.1, but is enabled in 13.2-RELEASE - hence why we both started seeing issues after upgrading to FreeBSD 13.2. > > For details see https://forums.freebsd.org/threads/apachectl-graceful-causes-sigsegv-with-latest-php-8-1-x-and-php-8-2-x.88814/#post-611560 Thanks for the info, we'll follow. We already have a plan to move to PHP-FPM. --------------3163486D09BB5567F9ED7923 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit
Sunday, July 16, 2023 3:37 PM
----- Original Message -----

From: "Robert Fitzpatrick" <robert@webtent.org>
To: "FreeBSD" <freebsd-questions@freebsd.org>
Sent: Sunday, July 16, 2023 1:07:35 PM
Subject: Apache log rotation

Ever since I upgraded from FreeBSD 13.1 to 13.2, each week when the
logrotate happens for my Apache logs, apache dies. I see this in the log...


Are you using phpopcache? There has been a known issue for a while with a conflit between FreeBSD ASLR and opcache. ASLR was included but not enabled in FreeBSD 13.1, but is enabled in 13.2-RELEASE - hence why we both started seeing issues after upgrading to FreeBSD 13.2.

For details see https://forums.freebsd.org/threads/apachectl-graceful-causes-sigsegv-with-latest-php-8-1-x-and-php-8-2-x.88814/#post-611560

Thanks for the info, we'll follow. We already have a plan to move to PHP-FPM.

--------------3163486D09BB5567F9ED7923-- From nobody Mon Jul 17 16:01:56 2023 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R4Rff0kp5z4ndCt for ; Mon, 17 Jul 2023 16:02:02 +0000 (UTC) (envelope-from robert@webtent.org) Received: from mx3.webtent.net (mx3.webtent.net [208.38.145.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4R4Rfd4jjvz42Rt for ; Mon, 17 Jul 2023 16:02:01 +0000 (UTC) (envelope-from robert@webtent.org) Authentication-Results: mx1.freebsd.org; none Received: from localhost (localhost [127.0.0.1]) by mx3.webtent.net (WebTent ESMTP Postfix Internet Mail Exchange) with ESMTP id 73711D7B37; Mon, 17 Jul 2023 12:01:59 -0400 (EDT) Received: from mx3.webtent.net ([127.0.0.1]) by localhost (mx3.webtent.net [127.0.0.1]) (maiad, port 10024) with ESMTP id 98508-09; Mon, 17 Jul 2023 12:01:58 -0400 (EDT) Received: from [192.168.1.50] (ns2.webtent.net [144.129.73.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: robert@mx3.webtent.net) by mx3.webtent.net (WebTent ESMTP Postfix Internet Mail Exchange) with ESMTPSA id 9814AD7B1E; Mon, 17 Jul 2023 12:01:58 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=webtent.org; s=201611; t=1689609718; bh=h6qjAkuTPD9hr0g6RIv9LwvRwR/Ma37qsQ1Xy9JvotM=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=GrJ8Z4jdoGTREzzFPRnoBBTn2CGz1fdmgMXBleN1pbvvDLRR6NcSP13ZDAgrxSz+5 y7MQLLci9B384FuAMJfi4npQlkhF2FPu73baLZO6854874fM+gTCxZfMyKTSJId3n0 ZnlNoexrWUN7J6KUaqE9WABSbR+s4WTLUkOGu+6y48mmwhic45vjSbcweHH0sj5jsu U2CEx1ouLHjuoxlhYzYaOI0kLnqdVrHRyau/SInr3kv5LGruC5+c+8mnIJ1X5/arBc RPKUB+dxVOJ/Ohbwg42keJ1F6vqnglt6QEYERFXWRom4Kxk170TbqUSwXpjstZUlPl KOAiHbL2S3Q2A== Subject: Re: Apache log rotation To: =?UTF-8?Q?Morgan_Wesstr=c3=b6m?= Cc: questions@freebsd.org References: <334c1023-8283-2c41-f284-ccc44bae6d0a@pp.dyndns.biz> From: Robert Fitzpatrick Message-ID: Date: Mon, 17 Jul 2023 12:01:56 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 PostboxApp/7.0.60 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 In-Reply-To: <334c1023-8283-2c41-f284-ccc44bae6d0a@pp.dyndns.biz> Content-Type: multipart/alternative; boundary="------------17F58F1EEA4F7BE8BAB7C956" Content-Language: en-US X-Virus-Scanned: WebTent Mailguard 1.0.4_3 X-Rspamd-Queue-Id: 4R4Rfd4jjvz42Rt X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:16724, ipnet:208.38.128.0/18, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --------------17F58F1EEA4F7BE8BAB7C956 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit > Morgan Wesström > Monday, July 17, 2023 8:19 AM > > > I don't have an answer for your specific problem but just wanted to > suggest an alternative solution. Have you looked at sysutils/cronolog? > Instead of relying on logrotate and restarting the apache process you > simply pipe the logs to cronolog which splits them according to a > pattern you define. Apache will run uninterrupted and logrotate won't > be used at all. > > Snippets from my own conf files that splits the logfiles per > year-month (note the pipe sign): > > ErrorLog "|/usr/local/sbin/cronolog > /var/log/apache24/httpd-error.%Y%m.default.log" > CustomLog "|/usr/local/sbin/cronolog > /var/log/apache24/httpd-access.%Y%m.default.log" combinedio Thanks, could be very helpful. --------------17F58F1EEA4F7BE8BAB7C956 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit
Monday, July 17, 2023 8:19 AM


I don't have an answer for your specific problem but just wanted to suggest an alternative solution. Have you looked at sysutils/cronolog? Instead of relying on logrotate and restarting the apache process you simply pipe the logs to cronolog which splits them according to a pattern you define. Apache will run uninterrupted and logrotate won't be used at all.

Snippets from my own conf files that splits the logfiles per year-month (note the pipe sign):

ErrorLog "|/usr/local/sbin/cronolog /var/log/apache24/httpd-error.%Y%m.default.log"
CustomLog "|/usr/local/sbin/cronolog /var/log/apache24/httpd-access.%Y%m.default.log" combinedio
Thanks, could be very helpful. --------------17F58F1EEA4F7BE8BAB7C956-- From nobody Tue Jul 18 21:16:18 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R5BZv5gGBz4nqJf for ; Tue, 18 Jul 2023 21:16:23 +0000 (UTC) (envelope-from pat@patmaddox.com) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4R5BZv01Flz3rP1 for ; Tue, 18 Jul 2023 21:16:22 +0000 (UTC) (envelope-from pat@patmaddox.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=patmaddox.com header.s=fm2 header.b="q H2+g/f"; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=F02iMr+a; spf=pass (mx1.freebsd.org: domain of pat@patmaddox.com designates 66.111.4.28 as permitted sender) smtp.mailfrom=pat@patmaddox.com; dmarc=none Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id BF9715C00C3; Tue, 18 Jul 2023 17:16:20 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Tue, 18 Jul 2023 17:16:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=patmaddox.com; h=cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:message-id:mime-version:reply-to :sender:subject:subject:to:to; s=fm2; t=1689714980; x= 1689801380; bh=hK5EFcWGAbzO521nyrJUcYBRq32oaNeuHV3thasDaDk=; b=q H2+g/fpEhWyGGVlGW+Kf0QBwCgNnjmZfmIMj7y+R7SvUkr7Sni4wjZ8NylN9F0d2 Uy2JZB0XBq+LYzqxGVloSuhDhfOQSMhk/cPggS+p0+1XJ7tB6DGuX2n1Oaplh3Em qa3APLcWEs4e3iG9vo9UUhSRubd/NK7ayBXPNOePiFlIEfUCanqVlwhR+WCeJ5wX I1PYIi7aSoFVkibuIyYRdwX1HqO4zirk5c0Gg7x87PnxelktEwosVx/zAqJ+rf2I +wO3nc0TMrCA23yez7Ik3rxLLL/TFxYtE66k0E9AfboOCdkAcw/VeNsd8fLqeoWL bcW4L4R1TBn2kePExSiZQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:message-id:mime-version:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1689714980; x=1689801380; bh=hK5EFcWGAbzO5 21nyrJUcYBRq32oaNeuHV3thasDaDk=; b=F02iMr+a2aO0InIfdd3mvgiJkuo1q e8mXHLtjL1oQIdVXmfYX8+0IMEoYlsdfBAatjDohwL5nrpf+40/HUTsv4AEOi8fG 4oCHk/n1ge6Edgri3l4a5XEYG5PcXmU1Mdr+LalFav3MG2IpdrHph8Go6ZXuRwpe siWUAm65OD2kdULG0gMNDb0Fnen9PWwLxegrxWjgfndxmIZ7jFlAgDTMZ1j7me0y xr48BvTWlbtH5AWDiwQGXPIeclHjYQmxeCJI6yNXCwBX2qWbti5/Q11l+Mcn9mXb Zm4IMb6v1Hn02wE+xpqr1G3nCbWB6AcNErkV250dqKvOj7U2IOSbz5uNA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrgeeggdduheeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffoffkgggtgfesthekmhdtredtjeenucfhrhhomhepfdfrrghtucfo rgguughogidfuceophgrthesphgrthhmrgguughogidrtghomheqnecuggftrfgrthhtvg hrnhepkeettedtteeggedvgefgkeetffekffeijeehteektddthffggfekkedtvedugfej necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphgrth esphgrthhmrgguughogidrtghomh X-ME-Proxy: Feedback-ID: i8b6c40f9:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Tue, 18 Jul 2023 17:16:20 -0400 (EDT) From: "Pat Maddox" To: "User Questions" Subject: NFS on ZFS: make NFS export writable, but local mount read-only? Date: Tue, 18 Jul 2023 14:16:18 -0700 X-Mailer: MailMate (1.13.2r5673) Message-ID: <554CEBCE-4F10-4AE4-B8E8-434971B2AE7F@patmaddox.com> List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed; markup=markdown Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.65 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.95)[-0.949]; R_DKIM_ALLOW(-0.20)[patmaddox.com:s=fm2,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.28]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.28:from]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.28:from]; FREEFALL_USER(0.00)[pat]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[patmaddox.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[patmaddox.com:+,messagingengine.com:+]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-questions@freebsd.org] X-Rspamd-Queue-Id: 4R5BZv01Flz3rP1 X-Spamd-Bar: --- I have a dataset that is intended as an NFS export. I have all that working. The thing is, I don’t want to be able to modify files on the host. It is purely a backup volume, so it should only be writable by NFS. Is it possible to mount the dataset as writable for NFS, but read-only for local host? Pat From nobody Wed Jul 19 14:49:23 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R5dyS6B5fz4nPRM; Wed, 19 Jul 2023 14:49:52 +0000 (UTC) (envelope-from scott.gasch@gmail.com) Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R5dyR23WHz3j47; Wed, 19 Jul 2023 14:49:51 +0000 (UTC) (envelope-from scott.gasch@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=QYQ2oHz4; spf=pass (mx1.freebsd.org: domain of scott.gasch@gmail.com designates 2607:f8b0:4864:20::535 as permitted sender) smtp.mailfrom=scott.gasch@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-517bdc9e81dso2959511a12.1; Wed, 19 Jul 2023 07:49:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689778189; x=1692370189; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=PTgeiMZmMM4tVqsuA1+pXtj1HwRYZmx1FW2ejzn6yTg=; b=QYQ2oHz4SHuSOlFGN0AfO22khEhq4ETaMMHBx1jfRJ+ay0DDVHlHcRtIhs5GFpchZx QneLLTxKRUCK/+eRJGq1I8nd/NzOXS9ko9FiUbjUbWkoo02CDIPZrf+SNNbe036kcA8V 30FJyp30iw+4yHtvlLFcoxf7nlXNeTzYvhu5iOIM/OhgwsY1VvLdRPLum0kwSsz//MTk 6SQLkIrz22c0fz4bbRQDm97oG+OMwdLyKLMvoDiGzTaiXjW7AIWZKo/DLjvcHi/e6oQ5 RaQE/omal/zYOOSV7XHeGBmo7b03AmNtQ7nfeCNtxCttxxBVEWQTO4WxH1Ax72DxVsn6 DJzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689778189; x=1692370189; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=PTgeiMZmMM4tVqsuA1+pXtj1HwRYZmx1FW2ejzn6yTg=; b=Z+JKxioxK1seSbSW9IvdMeVowHPiKh8akIu/hB70BeDvgsq30f4J6k2pIxueOPRboe tu9ASkCpdua5114BZI6YfQaHOC3lVDH4SE/jzz8DDlGDztGibdkIxIWjEuiw8mKy4/iY WLhRlKrZ0FPZ6t+VHLI/yQBE7z1O/BenZaEsjkE69nTlDJy8awmmggni6WMQ69MhWSRT UGr/evGpkyujhgveGQU7Up3rQ+8AKRkTbPm15VfmmtsQleuSczG/kymzhLI9OZ8QH3bY ebtZLVek1m4Y2B5Y9S1oPSGMtl6RjWnpu1EEw1HGAErv17t0hB0smRNhQjngl06FFgY5 bQ7A== X-Gm-Message-State: ABy/qLbjNvjmy+nDuqiAzKki5bqW2bnS0QHbiNKsmi+tmevHfUd22ED9 18AJddtUpEZ4OWg1Z75I47nBsjrZZKBYqZyGq4C6MBZo+LI= X-Google-Smtp-Source: APBJJlGrZ+mqUMYWBCMch9xdUjVJ3knIbd6vAHuLZKTWHwXhTSNQgC2WwiXOYwesVoBM0sumfMw8kcoaiMhwmiv0yPk= X-Received: by 2002:a17:90a:b891:b0:263:ba94:b15 with SMTP id o17-20020a17090ab89100b00263ba940b15mr4283409pjr.4.1689778189270; Wed, 19 Jul 2023 07:49:49 -0700 (PDT) List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 From: Scott Gasch Date: Wed, 19 Jul 2023 07:49:23 -0700 Message-ID: Subject: Swap filling up, usermode process swap usage doesn't explain To: freebsd-questions , freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="000000000000b884220600d824cd" X-Spamd-Result: default: False [-2.79 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; NEURAL_SPAM_SHORT(0.11)[0.111]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MIME_BASE64_TEXT(0.10)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::535:from]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-questions@freebsd.org,freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TAGGED_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4R5dyR23WHz3j47 X-Spamd-Bar: -- --000000000000b884220600d824cd Content-Type: text/plain; charset="UTF-8" I am running a 13.2-RELEASE GENERIC kernel and seeing a pattern where, after about 10 days of uptime, my swap begins to fill up. # swapinfo -h Device Size Used Avail Capacity /dev/ada0p3 48G 3.6G 44G 7% /dev/ada1p3 48G 3.6G 44G 7% /dev/ada2p3 48G 3.6G 44G 7% Total 144G 11G 133G 7% So, 11G of total swap space. What's using it? # systat -swap /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10 Load Average |||||| Device/Path Size Used |0% /10 /20 /30 /40 / 60\ 70\ 80\ 90\ 100| ada0p3 48G 3660M XXX ada1p3 48G 3666M XXX ada2p3 48G 3664M XXX Total 144G 11G XXX Pid Username Command Swap/Total Per-Process Per-System 14703 scott python3.8 4M / 154M 2% 0% 2451 scott rclone 4M / 934M 0% 0% 2452 scott rclone 3M / 1G 0% 0% 73827 scott bash 1M / 17M 6% 0% 39416 scott tmux 968K / 54M 1% 0% 41661 scott bash 828K / 17M 4% 0% 15727 scott bash 808K / 17M 4% 0% 39420 scott bash 804K / 17M 4% 0% 2455 scott bash 544K / 15M 3% 0% 39367 scott tmux 512K / 15M 3% 0% 2447 scott bash 376K / 15M 2% 0% 2450 scott bash 364K / 15M 2% 0% 2453 scott bash 324K / 15M 2% 0% 2454 scott bash 316K / 15M 2% 0% 2445 scott bash 312K / 15M 2% 0% 44937 scott bash 304K / 17M 1% 0% 2458 scott bash 72K / 15M 0% 0% At least they agree about it being 11G. Is this kernel memory being paged out to swap? The machine has 128G of physical memory and isn't under very heavy load at the moment. I suspect this is a bug in some kernel module... possibly wireguard because I run wireguard in a vnet jail and didn't observe this problem until setting that up. But I don't have any hard evidence. I've tried to mitigate this via swapoff -a. This works once but the next day swap will be back, even fuller. I've been doing regular reboots to fix this but would like to get to the bottom of it. If left alone, swap will fill up and the machine will get into a "not quite hung" but unusable and useless state. Am I off-base with my suspicion that this is kernel mode memory? Can someone teach me how to diagnose the status of kernel mode memory heap? Thx, Scott --000000000000b884220600d824cd Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: base64 PGRpdiBkaXI9Imx0ciI+SSBhbSBydW5uaW5nIGEgMTMuMi1SRUxFQVNFIEdFTkVSSUMga2VybmVs IGFuZCBzZWVpbmcgYSBwYXR0ZXJuIHdoZXJlLCBhZnRlciBhYm91dCAxMCBkYXlzIG9mIHVwdGlt ZSwgbXkgc3dhcCBiZWdpbnMgdG8gZmlsbCB1cC48YnI+PGJyPjxmb250IGZhY2U9Im1vbm9zcGFj ZSI+IyBzd2FwaW5mbyAtaDwvZm9udD48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSI+RGV2aWNl IMKgIMKgIMKgIMKgIMKgIMKgIMKgU2l6ZSDCoCDCoCBVc2VkIMKgIMKgQXZhaWwgQ2FwYWNpdHk8 YnI+L2Rldi9hZGEwcDMgwqAgwqAgwqAgwqAgwqA0OEcgwqAgwqAgMy42RyDCoCDCoCDCoDQ0RyDC oCDCoCA3JTxicj4vZGV2L2FkYTFwMyDCoCDCoCDCoCDCoCDCoDQ4RyDCoCDCoCAzLjZHIMKgIMKg IMKgNDRHIMKgIMKgIDclPGJyPi9kZXYvYWRhMnAzIMKgIMKgIMKgIMKgIMKgNDhHIMKgIMKgIDMu NkcgwqAgwqAgwqA0NEcgwqAgwqAgNyU8YnI+VG90YWwgwqAgwqAgwqAgwqAgwqAgwqAgwqAgMTQ0 RyDCoCDCoCDCoDExRyDCoCDCoCAxMzNHIMKgIMKgIDclPC9mb250PjwvZGl2PjxkaXY+PGJyPjwv ZGl2PjxkaXY+U28sIDExRyBvZiB0b3RhbCBzd2FwIHNwYWNlLsKgIFdoYXQmIzM5O3MgdXNpbmcg aXQ/PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UiPiMgc3lz dGF0IC1zd2FwPC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlIj7CoCDCoCDC oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCAvMCDCoCAvMSDCoCAvMiDCoCAvMyDCoCAvNCDCoCAvNSDC oCAvNiDCoCAvNyDCoCAvOCDCoCAvOSDCoCAvMTA8YnI+wqAgwqAgwqBMb2FkIEF2ZXJhZ2UgwqAg fHx8fHx8PGJyPjxicj5EZXZpY2UvUGF0aCDCoCDCoCDCoCBTaXplIMKgVXNlZCB8MCUgwqAvMTAg wqAvMjAgwqAvMzAgwqAvNDAgwqAvIDYwXCDCoDcwXCDCoDgwXCDCoDkwXCAxMDB8PGJyPmFkYTBw MyDCoCDCoCDCoCDCoCDCoCDCoCA0OEcgMzY2ME0gWFhYPGJyPmFkYTFwMyDCoCDCoCDCoCDCoCDC oCDCoCA0OEcgMzY2Nk0gWFhYPGJyPmFkYTJwMyDCoCDCoCDCoCDCoCDCoCDCoCA0OEcgMzY2NE0g WFhYPGJyPlRvdGFsIMKgIMKgIMKgIMKgIMKgIMKgIDE0NEcgwqAgMTFHIFhYWDxicj48YnI+UGlk IMKgIMKgVXNlcm5hbWUgwqAgQ29tbWFuZCDCoCDCoCBTd2FwL1RvdGFsIFBlci1Qcm9jZXNzIMKg IMKgUGVyLVN5c3RlbTxicj7CoDE0NzAzIHNjb3R0IMKgIMKgIMKgcHl0aG9uMy44IMKgIMKgNE0g LyAxNTRNIMKgMiUgwqAgwqAgwqAgwqAgwqAgwqAgwqAwJTxicj7CoCAyNDUxIHNjb3R0IMKgIMKg IMKgcmNsb25lIMKgIMKgIMKgIDRNIC8gOTM0TSDCoDAlIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8 YnI+wqAgMjQ1MiBzY290dCDCoCDCoCDCoHJjbG9uZSDCoCDCoCDCoCAzTSAvIMKgIDFHIMKgMCUg wqAgwqAgwqAgwqAgwqAgwqAgwqAwJTxicj7CoDczODI3IHNjb3R0IMKgIMKgIMKgYmFzaCDCoCDC oCDCoCDCoCAxTSAvIMKgMTdNIMKgNiUgwqAgwqAgwqAgwqAgwqAgwqAgwqAwJTxicj7CoDM5NDE2 IHNjb3R0IMKgIMKgIMKgdG11eCDCoCDCoCDCoCA5NjhLIC8gwqA1NE0gwqAxJSDCoCDCoCDCoCDC oCDCoCDCoCDCoDAlPGJyPsKgNDE2NjEgc2NvdHQgwqAgwqAgwqBiYXNoIMKgIMKgIMKgIDgyOEsg LyDCoDE3TSDCoDQlIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8YnI+wqAxNTcyNyBzY290dCDCoCDC oCDCoGJhc2ggwqAgwqAgwqAgODA4SyAvIMKgMTdNIMKgNCUgwqAgwqAgwqAgwqAgwqAgwqAgwqAw JTxicj7CoDM5NDIwIHNjb3R0IMKgIMKgIMKgYmFzaCDCoCDCoCDCoCA4MDRLIC8gwqAxN00gwqA0 JSDCoCDCoCDCoCDCoCDCoCDCoCDCoDAlPGJyPsKgIDI0NTUgc2NvdHQgwqAgwqAgwqBiYXNoIMKg IMKgIMKgIDU0NEsgLyDCoDE1TSDCoDMlIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8YnI+wqAzOTM2 NyBzY290dCDCoCDCoCDCoHRtdXggwqAgwqAgwqAgNTEySyAvIMKgMTVNIMKgMyUgwqAgwqAgwqAg wqAgwqAgwqAgwqAwJTxicj7CoCAyNDQ3IHNjb3R0IMKgIMKgIMKgYmFzaCDCoCDCoCDCoCAzNzZL IC8gwqAxNU0gwqAyJSDCoCDCoCDCoCDCoCDCoCDCoCDCoDAlPGJyPsKgIDI0NTAgc2NvdHQgwqAg wqAgwqBiYXNoIMKgIMKgIMKgIDM2NEsgLyDCoDE1TSDCoDIlIMKgIMKgIMKgIMKgIMKgIMKgIMKg MCU8YnI+wqAgMjQ1MyBzY290dCDCoCDCoCDCoGJhc2ggwqAgwqAgwqAgMzI0SyAvIMKgMTVNIMKg MiUgwqAgwqAgwqAgwqAgwqAgwqAgwqAwJTxicj7CoCAyNDU0IHNjb3R0IMKgIMKgIMKgYmFzaCDC oCDCoCDCoCAzMTZLIC8gwqAxNU0gwqAyJSDCoCDCoCDCoCDCoCDCoCDCoCDCoDAlPGJyPsKgIDI0 NDUgc2NvdHQgwqAgwqAgwqBiYXNoIMKgIMKgIMKgIDMxMksgLyDCoDE1TSDCoDIlIMKgIMKgIMKg IMKgIMKgIMKgIMKgMCU8YnI+wqA0NDkzNyBzY290dCDCoCDCoCDCoGJhc2ggwqAgwqAgwqAgMzA0 SyAvIMKgMTdNIMKgMSUgwqAgwqAgwqAgwqAgwqAgwqAgwqAwJTxicj7CoCAyNDU4IHNjb3R0IMKg IMKgIMKgYmFzaCDCoCDCoCDCoCDCoDcySyAvIMKgMTVNIMKgMCUgwqAgwqAgwqAgwqAgwqAgwqAg wqAwJTxicj48L2ZvbnQ+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5BdCBsZWFzdCB0aGV5IGFn cmVlIGFib3V0IGl0IGJlaW5nIDExRy7CoCBJcyB0aGlzIGtlcm5lbCBtZW1vcnkgYmVpbmcgcGFn ZWQgb3V0IHRvIHN3YXA/wqAgVGhlIG1hY2hpbmUgaGFzIDEyOEcgb2YgcGh5c2ljYWwgbWVtb3J5 IGFuZCBpc24mIzM5O3QgdW5kZXIgdmVyeSBoZWF2eSBsb2FkIGF0IHRoZSBtb21lbnQuwqDCoDwv ZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+SSBzdXNwZWN0IHRoaXMgaXMgYSBidWcgaW4gc29tZSBr ZXJuZWwgbW9kdWxlLi4uIHBvc3NpYmx5IHdpcmVndWFyZMKgYmVjYXVzZSBJIHJ1biB3aXJlZ3Vh cmQgaW4gYSB2bmV0IGphaWwgYW5kIGRpZG4mIzM5O3Qgb2JzZXJ2ZSB0aGlzIHByb2JsZW0gdW50 aWwgc2V0dGluZyB0aGF0IHVwLsKgIEJ1dCBJIGRvbiYjMzk7dCBoYXZlIGFueSBoYXJkIGV2aWRl bmNlLjwvZGl2PjxkaXY+PGJyPkkmIzM5O3ZlIHRyaWVkIHRvIG1pdGlnYXRlIHRoaXMgdmlhIHN3 YXBvZmYgLWEuwqAgVGhpcyB3b3JrcyBvbmNlIGJ1dCB0aGUgbmV4dDxicj5kYXkgc3dhcCB3aWxs IGJlIGJhY2ssIGV2ZW4gZnVsbGVyLsKgIEkmIzM5O3ZlIGJlZW4gZG9pbmcgcmVndWxhciByZWJv b3RzIHRvIGZpeCB0aGlzIGJ1dCB3b3VsZCBsaWtlIHRvIGdldCB0byB0aGUgYm90dG9tIG9mIGl0 LsKgIElmIGxlZnQgYWxvbmUsIHN3YXAgd2lsbDxicj5maWxsIHVwIGFuZCB0aGUgbWFjaGluZSB3 aWxsIGdldCBpbnRvIGEgJnF1b3Q7bm90IHF1aXRlIGh1bmcmcXVvdDsgYnV0IHVudXNhYmxlIGFu ZCB1c2VsZXNzIHN0YXRlLjxicj48YnI+QW0gSSBvZmYtYmFzZSB3aXRoIG15IHN1c3BpY2lvbiB0 aGF0IHRoaXMgaXMga2VybmVsIG1vZGUgbWVtb3J5PyBDYW4gc29tZW9uZSB0ZWFjaCBtZSBob3cg dG8gZGlhZ25vc2UgdGhlIHN0YXR1cyBvZiBrZXJuZWwgbW9kZSBtZW1vcnkgaGVhcD88YnI+PGJy PlRoeCw8YnI+U2NvdHQ8L2Rpdj48ZGl2Pjxicj48L2Rpdj48L2Rpdj4NCg== --000000000000b884220600d824cd-- From nobody Wed Jul 19 18:34:14 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R5kxZ6qk7z4n1Xc; Wed, 19 Jul 2023 18:34:26 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [66.165.241.226]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.nomadlogic.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R5kxX51JYz3l5x; Wed, 19 Jul 2023 18:34:24 +0000 (UTC) (envelope-from pete@nomadlogic.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=nomadlogic.org header.s=04242021 header.b=fzgHwV+n; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 66.165.241.226 as permitted sender) smtp.mailfrom=pete@nomadlogic.org; dmarc=pass (policy=quarantine) header.from=nomadlogic.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nomadlogic.org; s=04242021; t=1689791655; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/jaz3BLndlyFn14v8H7acTnMkOVnYk7eonTBMJn+58Y=; b=fzgHwV+niTzTTt2+GwvfU1SEmAvGKez4CSEVE5LuQNJsWbrKAm0TnzeBBqtvoqdACMWSf6 LvcK7VnnLA+G/3ZUNa0Zm9+6Nf4qF3Uk9cALOkF6/W6xQcFRT1F3h9MyBXrnI2JkDmj2rJ 7OQrKvSYpLH7gcpIa3NEvx+DEoOc9Lo= Received: from [192.168.1.160] (cpe-24-24-168-214.socal.res.rr.com [24.24.168.214]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id 173c2ee4 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 19 Jul 2023 18:34:14 +0000 (UTC) Message-ID: Date: Wed, 19 Jul 2023 11:34:14 -0700 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: Swap filling up, usermode process swap usage doesn't explain Content-Language: en-US To: Scott Gasch , freebsd-questions , freebsd-hackers@freebsd.org References: From: Pete Wright In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.95 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; NEURAL_HAM_SHORT(-0.95)[-0.951]; DMARC_POLICY_ALLOW(-0.50)[nomadlogic.org,quarantine]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[nomadlogic.org:s=04242021]; MIME_GOOD(-0.10)[text/plain]; ASN(0.00)[asn:29802, ipnet:66.165.240.0/22, country:US]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-questions@freebsd.org,freebsd-hackers@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[nomadlogic.org:+]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TAGGED_RCPT(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4R5kxX51JYz3l5x X-Spamd-Bar: --- On 7/19/23 07:49, Scott Gasch wrote: > I am running a 13.2-RELEASE GENERIC kernel and seeing a pattern where, > after about 10 days of uptime, my swap begins to fill up. > > > At least they agree about it being 11G.  Is this kernel memory being > paged out to swap?  The machine has 128G of physical memory and isn't > under very heavy load at the moment. > Are you running ZFS by any chance? If so its possible it is trying to use as much memory as possible for the ARC. I've seen this on a few systems which lots of memory. One way to tell is to run "top" and look at the ARC stats: last pid: 71322; load averages: 1.02, 0.94, 0.87 up 8+18:38:34 11:31:26 376 processes: 1 running, 146 sleeping, 229 zombie CPU: 0.6% user, 0.0% nice, 6.5% system, 0.0% interrupt, 93.0% idle Mem: 3599M Active, 18G Inact, 4132M Laundry, 4272M Wired, 892M Free ARC: 1749M Total, 651M MFU, 239M MRU, 1864K Anon, 13M Header, 844M Other 216M Compressed, 758M Uncompressed, 3.52:1 Ratio On a few of my larger memory systems I cap the ARC by setting this sysctl knob (this is like 45G on my system): vfs.zfs.arc.max=45000000000 -pete -- Pete Wright pete@nomadlogic.org @nomadlogicLA From nobody Wed Jul 19 19:16:48 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R5ltY5T25z4ngXr; Wed, 19 Jul 2023 19:16:53 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R5ltY4fByz40BR; Wed, 19 Jul 2023 19:16:53 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1689794213; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Qm06ZT/rhsuKrwAZGIREi8mnxOYHC2xaRS+JdQviLOY=; b=Sf2Qz2o06OjdeT5N03XjirPivgW9+mvmMFufY5+fB650SXhlF5ovQ4JI0Ra+qHbnL0vGbT W3RCwHehhWH2o+C1OA7PvOmw0m7ikFbxSmB01b+FkJvjA9ilFiwnxCs4dLiL+EwU7ZQUAm 90DctrRib9r01RFIkDsSnXaJBKg8+t4raGDUSM+SJuCYH2+ZpBMkzQMQFLzjqX/uujdr0R 327KIKcF1WkL6S/9b2VRn9r6CUR5BM5R0WMaWWTb6LN/dj0nBsDVrAyxzT/t9wR8WW1jB2 doaRUx6D7JFqCH/0EXFLTw4cGbvG3VDtiIfrjfYGmpJNU1vZy1hqF768id7HnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1689794213; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Qm06ZT/rhsuKrwAZGIREi8mnxOYHC2xaRS+JdQviLOY=; b=s1T9KGrt+OXLE+1BmfxCt/5l0ivA3N+Lf3u90mgiFXvY/Cf058kOQWc68hmNYFQ6yx6YkA jEd8Vh3NF0gvJkEul5EvEexojJFGZTgebOev7jKjZjXdBwFPtfFBUK45Z7WeO/lttIHm1V i1krBS6yz770p2ZHbsaeY4Xw1aDxBnUx3AvDjVcdDkcKyMXpKsiFEp9LP9zHxsTJg8m+64 6IoxjmqsQ//MsarwCd94VYIKSYh/Kqzze9usjbk7HQ9L4a3MyhUsVi/KzdUwuOuWuHh/Ag x2e7WGCrfpWdcckVAiqtWpGBQ/nKhT3VXwTpt62vkCguBNAzy3lEeWWiUalLog== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1689794213; a=rsa-sha256; cv=none; b=LG+j/+8GtP4rVB5a0e5pL4Y4CMzNr+fxxyLNBqBascwcIF8FD1xmBga1SZZOcHDonc1Zz3 rQo7xSxiiIJlNniMzW5ZRunvpVhWtsUdJE0Yj3SmpswRFkZbAhoOWPTD6pt8A1opuGBuGT 3gnW1hIBKd7OAGuKLPbw4q09El6gVnuRt+gIFFF7gQf2mzbxDJRGw16jGa6qYSLQ2cWAW0 53Y7/zX70sCmVDU49sulonwuY/xbf7ikGNUW5pLsChJGBw0oazYd7GACHGyfRqqsCTXZV+ KNnWnuNnF0xDDO0GtDJTOOKg6ip0g62/9Ky+PLNVIEDQv3xW9iv723qcjyS4IA== Received: from [IPV6:2001:470:1f1c:a0::2] (tunnel642390-pt.tunnel.tserv1.lon2.ipv6.he.net [IPv6:2001:470:1f1c:a0::2]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: grahamperrin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4R5ltY0PrLz16kG; Wed, 19 Jul 2023 19:16:52 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Message-ID: <1269698d-6639-b78d-5ed3-28e2e8bd4378@freebsd.org> Date: Wed, 19 Jul 2023 20:16:48 +0100 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: Swap filling up, usermode process swap usage doesn't explain To: FreeBSD questions , freebsd-hackers@freebsd.org References: Content-Language: en-US From: Graham Perrin Organization: FreeBSD In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------v0GLNcZqDmQJu98xrmss3PZx" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------v0GLNcZqDmQJu98xrmss3PZx Content-Type: multipart/mixed; boundary="------------Z0OV5IExqsjLYyC4m0QyFzlW"; protected-headers="v1" From: Graham Perrin To: FreeBSD questions , freebsd-hackers@freebsd.org Message-ID: <1269698d-6639-b78d-5ed3-28e2e8bd4378@freebsd.org> Subject: Re: Swap filling up, usermode process swap usage doesn't explain References: In-Reply-To: --------------Z0OV5IExqsjLYyC4m0QyFzlW Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 T24gMTkvMDcvMjAyMyAxOTozNCwgUGV0ZSBXcmlnaHQgd3JvdGU6DQo+IOKApg0KPg0KPiB2 ZnMuemZzLmFyYy5tYXg94oCmDQoNCg0KSWYgaXQncyB0cnVseSBuZWNlc3NhcnkgdG8gbWFr ZSBBUkMtcmVsYXRlZCBjaGFuZ2VzLCBwbGVhc2UgYWxzbyBiZSANCmF3YXJlIG9mOg0KDQp2 ZnMuemZzLmFyY19mcmVlX3RhcmdldA0KPGh0dHBzOi8vd2lraS5mcmVlYnNkLm9yZy9NeXRo cyN2ZnMuemZzLmFyY19mcmVlX3RhcmdldD4NCg0KdmZzLnpmcy5hcmMuc3lzX2ZyZWUNCjxo dHRwczovL3dpa2kuZnJlZWJzZC5vcmcvTXl0aHMjdmZzLnpmcy5hcmMuc3lzX2ZyZWU+DQoN Cg== --------------Z0OV5IExqsjLYyC4m0QyFzlW-- --------------v0GLNcZqDmQJu98xrmss3PZx Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsFAmS4NqEFAwAAAAAACgkQt2dIb0oY1Au6 3w//YGCl/cVZ4A1rbVtz0nvjUWK/i7p2rwO6namaHfnZwPIzE+ZELSYUn5pbRQgMZqlfFu3yoXRi 2e0lXVDjEoD4S/rF8hApz2rbEDKAcSjsG1uHDqzm7eewb+e1WpPWSRkMekvKBy5FMHg2iI8B4Tf4 ZpF5OOMFYB633d4LLgcv0liEoWLryeVVI2jpYfsAIC+ULW6fny2iy8tYeF32Kn/BF6Cl7xb8fmpd b2JYcCJuc0U0K+KiDeIn3cfnDUJAGvyKyugzWlLS11uz47frWR/FPH1JyqrEhd8jVT9I+r/iviN9 aL8xfySELF822K1FyXVFhHRIrKnCjE5QbLypnlCDSV19DBgJjS83KsXJILvuRSXlVrOrJ0MOaCX4 mYBIgpyPvYtlupsQKSt8P5u/zurbac4G5GEIi2sXcxwq62PRmofyONyYWkrMMDZcZayKG+hhVmXU OVWXVUgHZQPU3o1Y6qWjXYL7fTx//muXvuP9RrDq+taxO4fuJua/9btP0EgWkmuTCzB6trMxyVxP gex2zN/VnRDUieJuOqeWR31rnnrJjsUKsiYv+fVMjCagYZTrNdRcvMCWHvTMrtK71b8CKoqMAQUf stuQ0omFS7FS4tJKtLTeU2YiLC/DMwJ2XQuLLCnnZuWXnOJeXXz4DwjKlaZ7mcmG6TdR9zJ1eARS 0IU= =1DIe -----END PGP SIGNATURE----- --------------v0GLNcZqDmQJu98xrmss3PZx-- From nobody Wed Jul 19 22:11:30 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R5qmZ3ndmz4mxwm; Wed, 19 Jul 2023 22:11:58 +0000 (UTC) (envelope-from scott.gasch@gmail.com) Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R5qmY6bSLz3tjw; Wed, 19 Jul 2023 22:11:57 +0000 (UTC) (envelope-from scott.gasch@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x102c.google.com with SMTP id 98e67ed59e1d1-26586e824e7so44792a91.3; Wed, 19 Jul 2023 15:11:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689804716; x=1692396716; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=PhDbvCfOUCmF25V5xf674+P9I+zo/hOrmqkPpvpR9g0=; b=fLFVXEwL6LlggX/44TOL2HmXSSy9acrg/J/mGaDbrNqWsrazMYpjz8sKKWWmBd6q6E mJgHFfhuoTdEn0gEQ8lBzxlFXxg6Fz4mNH7LqMM+OelAcESoTZ57UMs5HcQPSwahgbSl elUYX4ywTE2GYQ4zWEJYlEW18op0JV8WZL233+KL5zrqNZGovKYlZSHOJQznhUf5yUYt g3UVyGkep6icQV6b9089tgx+MnfZxwywEkAfpVpmNV8mesLfzQz8JAtycx7X+MBI1sfA cPONpnA6rYau12tQ3KTTHcUwBCfg4fk9CbY+pH+rtt3KcOVEJV+Y4v2z9UdPDGYyTcu0 S1lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689804716; x=1692396716; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=PhDbvCfOUCmF25V5xf674+P9I+zo/hOrmqkPpvpR9g0=; b=QGmtevLi6Z0lhuf8w0alBYu37x1+wothjw9LIV7YM/RWD7HEA+ukTuJ9F4dE6MipAS +v2MDGs0qEXnWotXmZ7T8WStJUpum6Su7enI6mNGGL+yp5XZeS8q7/VXd2/beow5uKWJ wYMiu4f7Muk6IKbkIkdKkhGTZJ7DtGW4AlOGjGbp7c6q8DH0n3zjtlv7KmbwUZpFBfHP g4iqVzvY53VQ1IXHYtRAAIcT+7ZFH98csh0thJjKqj3/LhZTQZrh1pVv6ZUaSehSy4M6 1lTF2zgJa26IxZy8I5qTq3zqcbnKolZHkPfJuiINk9o56TZS0mElEa/4K1zc3tSynrqA c/qQ== X-Gm-Message-State: ABy/qLY/OlT2YGOwlFTdeHLKwX9+OoTCAv+sGC6fb60c/ZfgPLHuzmaf eUhBTYByss13Gk5Z/dpOv1tbhdWZWKsl+6yvhxEGqalfgXg= X-Google-Smtp-Source: APBJJlEorU3/q8U7SmRFvB+0RYXi2QoEKT6DB3cSWxIJ+cNbHo98NG1xdErXqgHau71TysDT3fJJIPm3RzEsG4LyigE= X-Received: by 2002:a17:90b:50b:b0:263:1af0:5541 with SMTP id r11-20020a17090b050b00b002631af05541mr2921306pjz.32.1689804716167; Wed, 19 Jul 2023 15:11:56 -0700 (PDT) List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Scott Gasch Date: Wed, 19 Jul 2023 15:11:30 -0700 Message-ID: Subject: Re: Swap filling up, usermode process swap usage doesn't explain To: Pete Wright Cc: freebsd-questions , freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="000000000000d8dc6f0600de5111" X-Rspamd-Queue-Id: 4R5qmY6bSLz3tjw X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated --000000000000d8dc6f0600de5111 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Yes, I'm using ZFS. Here's what top says: last pid: 88926; load averages: 1.20, 0.96, 0.87 up 5+17:48:34 15:09:58 274 processes: 1 running, 272 sleeping, 1 zombie CPU: 1.8% user, 0.0% nice, 0.5% system, 0.0% interrupt, 97.8% idle Mem: 1844M Active, 7777M Inact, 77G Laundry, 35G Wired, 750M Buf, 3367M Fre= e ARC: 24G Total, 2878M MFU, 18G MRU, 21M Anon, 119M Header, 2622M Other 18G Compressed, 25G Uncompressed, 1.33:1 Ratio Swap: 144G Total, 11G Used, 133G Free, 7% Inuse If I leave this alone it will grow to consume all available swap space. I'll try your fix with the sysctl knob and see what happens... I hope this is it, I've been fighting this for a while now. Thx, Scott On Wed, Jul 19, 2023 at 11:34=E2=80=AFAM Pete Wright = wrote: > > > On 7/19/23 07:49, Scott Gasch wrote: > > I am running a 13.2-RELEASE GENERIC kernel and seeing a pattern where, > > after about 10 days of uptime, my swap begins to fill up. > > > > > > > At least they agree about it being 11G. Is this kernel memory being > > paged out to swap? The machine has 128G of physical memory and isn't > > under very heavy load at the moment. > > > > Are you running ZFS by any chance? If so its possible it is trying to > use as much memory as possible for the ARC. I've seen this on a few > systems which lots of memory. One way to tell is to run "top" and look > at the ARC stats: > > last pid: 71322; load averages: 1.02, 0.94, 0.87 up > 8+18:38:34 11:31:26 > 376 processes: 1 running, 146 sleeping, 229 zombie > CPU: 0.6% user, 0.0% nice, 6.5% system, 0.0% interrupt, 93.0% idle > Mem: 3599M Active, 18G Inact, 4132M Laundry, 4272M Wired, 892M Free > ARC: 1749M Total, 651M MFU, 239M MRU, 1864K Anon, 13M Header, 844M Other > 216M Compressed, 758M Uncompressed, 3.52:1 Ratio > > > > On a few of my larger memory systems I cap the ARC by setting this > sysctl knob (this is like 45G on my system): > vfs.zfs.arc.max=3D45000000000 > > > -pete > > > -- > Pete Wright > pete@nomadlogic.org > @nomadlogicLA > --000000000000d8dc6f0600de5111 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Yes, I'm using ZFS.=C2=A0 Here's what top says:
last pid: 88926; =C2=A0load averages: =C2=A01.20, =C2=A00.= 96, =C2=A00.87 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0up 5+17:48:34 =C2=A015:09:58
274 processes: 1 ru= nning, 272 sleeping, 1 zombie
CPU: =C2=A01.8% user, =C2=A00.0% nice, =C2= =A00.5% system, =C2=A00.0% interrupt, 97.8% idle
Mem: 1844M Active, 7777= M Inact, 77G Laundry, 35G Wired, 750M Buf, 3367M Free
ARC: 24G Total, 28= 78M MFU, 18G MRU, 21M Anon, 119M Header, 2622M Other
=C2=A0 =C2=A0 =C2= =A018G Compressed, 25G Uncompressed, 1.33:1 Ratio
Swap: 144G Total, 11G = Used, 133G Free, 7% Inuse

If I leave this alon= e it will grow to consume all available swap space.=C2=A0 I'll try your= fix with the sysctl knob and see what happens...=C2=A0 I hope this is it, = I've been fighting this for a while now.

Thx,<= /div>
Scott


On Wed, Jul 19, 2023 at 11:34=E2=80=AFA= M Pete Wright <pete@nomadlogic.or= g> wrote:


On 7/19/23 07:49, Scott Gasch wrote:
> I am running a 13.2-RELEASE GENERIC kernel and seeing a pattern where,=
> after about 10 days of uptime, my swap begins to fill up.
>
<snip>
>
> At least they agree about it being 11G.=C2=A0 Is this kernel memory be= ing
> paged out to swap?=C2=A0 The machine has 128G of physical memory and i= sn't
> under very heavy load at the moment.
>

Are you running ZFS by any chance?=C2=A0 If so its possible it is trying to=
use as much memory as possible for the ARC.=C2=A0 I've seen this on a f= ew
systems which lots of memory.=C2=A0 One way to tell is to run "top&quo= t; and look
at the ARC stats:

last pid: 71322;=C2=A0 load averages:=C2=A0 1.02,=C2=A0 0.94,=C2=A0 0.87=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 up
8+18:38:34=C2=A0 11:31:26
376 processes: 1 running, 146 sleeping, 229 zombie
CPU:=C2=A0 0.6% user,=C2=A0 0.0% nice,=C2=A0 6.5% system,=C2=A0 0.0% interr= upt, 93.0% idle
Mem: 3599M Active, 18G Inact, 4132M Laundry, 4272M Wired, 892M Free
ARC: 1749M Total, 651M MFU, 239M MRU, 1864K Anon, 13M Header, 844M Other =C2=A0 =C2=A0 =C2=A0 216M Compressed, 758M Uncompressed, 3.52:1 Ratio



On a few of my larger memory systems I cap the ARC by setting this
sysctl knob (this is like 45G on my system):
vfs.zfs.arc.max=3D45000000000


-pete


--
Pete Wright
pete@nomadlogic.or= g
@nomadlogicLA
--000000000000d8dc6f0600de5111-- From nobody Wed Jul 19 22:40:03 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R5rP86Cj8z4nRnd; Wed, 19 Jul 2023 22:40:12 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [66.165.241.226]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.nomadlogic.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R5rP74JmJz43LQ; Wed, 19 Jul 2023 22:40:11 +0000 (UTC) (envelope-from pete@nomadlogic.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=nomadlogic.org header.s=04242021 header.b=XrTA77F3; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 66.165.241.226 as permitted sender) smtp.mailfrom=pete@nomadlogic.org; dmarc=pass (policy=quarantine) header.from=nomadlogic.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nomadlogic.org; s=04242021; t=1689806406; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vh6MAnC7/Z6hlsM1K+9HU/Ddw8LBzlCd4eRE3fJW7KI=; b=XrTA77F3adEIv1ItX3XH3FqBhbVJ/I8TKBul9iJCuFYzfDCqsCzQkEF6XcOR/pTAl2aT08 7Ksl7KW9Agtzvhq+pRPsBEhUCvh+XqxX6nJHSGFyHhugSJ+fv4YhvXijFd8X5DW73V+CBI a6rTlqe5Adyllr/i4r4yx8zoZ9mr36Q= Received: from [192.168.1.160] (cpe-24-24-168-214.socal.res.rr.com [24.24.168.214]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id 916497ef (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 19 Jul 2023 22:40:03 +0000 (UTC) Message-ID: <88e0d702-cb86-b922-a5ce-82fe32e66a40@nomadlogic.org> Date: Wed, 19 Jul 2023 15:40:03 -0700 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: Swap filling up, usermode process swap usage doesn't explain Content-Language: en-US To: Scott Gasch Cc: freebsd-questions , freebsd-hackers@freebsd.org References: From: Pete Wright In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.94 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.94)[-0.940]; DMARC_POLICY_ALLOW(-0.50)[nomadlogic.org,quarantine]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[nomadlogic.org:s=04242021]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org,freebsd-questions@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; ASN(0.00)[asn:29802, ipnet:66.165.240.0/22, country:US]; MIME_TRACE(0.00)[0:+]; DKIM_TRACE(0.00)[nomadlogic.org:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; BLOCKLISTDE_FAIL(0.00)[24.24.168.214:server fail,66.165.241.226:server fail]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TAGGED_RCPT(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4R5rP74JmJz43LQ X-Spamd-Bar: --- On 7/19/23 15:11, Scott Gasch wrote: > Yes, I'm using ZFS.  Here's what top says: > > last pid: 88926;  load averages:  1.20,  0.96,  0.87 >    up 5+17:48:34  15:09:58 > 274 processes: 1 running, 272 sleeping, 1 zombie > CPU:  1.8% user,  0.0% nice,  0.5% system,  0.0% interrupt, 97.8% idle > Mem: 1844M Active, 7777M Inact, 77G Laundry, 35G Wired, 750M Buf, 3367M Free > ARC: 24G Total, 2878M MFU, 18G MRU, 21M Anon, 119M Header, 2622M Other >      18G Compressed, 25G Uncompressed, 1.33:1 Ratio > Swap: 144G Total, 11G Used, 133G Free, 7% Inuse > > If I leave this alone it will grow to consume all available swap space. > I'll try your fix with the sysctl knob and see what happens...  I hope > this is it, I've been fighting this for a while now. worth a shot, but 24G of ARC isn't that bad, especially if you are doing quite a bit of disk i/o. i'm more interested in the 77G of Laundry memory, that seems like quite a bit. but i don't know your workload so not sure... -p -- Pete Wright pete@nomadlogic.org @nomadlogicLA From nobody Wed Jul 19 22:50:44 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R5rdr3sgTz4nXHC for ; Wed, 19 Jul 2023 22:51:12 +0000 (UTC) (envelope-from scott.gasch@gmail.com) Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R5rdr1d3Nz46B7 for ; Wed, 19 Jul 2023 22:51:12 +0000 (UTC) (envelope-from scott.gasch@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-io1-xd2a.google.com with SMTP id ca18e2360f4ac-78374596182so7725139f.0 for ; Wed, 19 Jul 2023 15:51:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689807071; x=1692399071; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=IK0jkmtbuRzyB++NEwn58JgGDL2KY7F6GbpYLelrsy0=; b=Gaiuz9oQUh8Pb4V07lVfALmcom6JOZ0DOrmB6t7C247s8hflgGfF4lelppoBgo2Yla 0D+fYYkg/TM3Hlmc3h30Ac+I763uRFR7yMP04RoakLYtokgD06GPW9/0Qy8KW5UFusSr QTLPH5NMP5jSk+W9Q4F+drb2UlD0ZNrCsXVKAX+6T83LoDXgCDpkdauLbdgZzcaz8one wTp9OqMO+uUy+LMKHf/+//KYvfb32uy7uTVw7CKryJiF4F57z31xGRoR8tBgTVLH26o2 CsIitQiDFmrNUnKh0IkrAjp3ks1vZgxvf/oAGLQgTUb8G4ljozKrq3NDgftZ5iddg42I 5nfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689807071; x=1692399071; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=IK0jkmtbuRzyB++NEwn58JgGDL2KY7F6GbpYLelrsy0=; b=Bg5HvbTV5FIWUUNrkbgar1v5SpvnnTJvq9d7kkLeXqc/p/r9QQ79h+NfNyNNI57syN pfx6DS9C7J25q/5qtkW9ESmn4u7tf60DIfp5MaVsiuourrjKeu/AhcuuO/I6IF7BrKdG QXempeP7/pMrmJnSWasuAyUR9mxy/uwlsO1T6Cm23k/X4eNGrVPnG7FDVkjX/yST+T2U GEUWa8yOerZci7uwizUO44LKCj0H/TFOXBwYXhkPN66BZiHJ016VT3FucOm2aKrxrcbc 4fo0TTD/o1gc4opPIEshMOTa0V/GbgCKOtibahjAm09MvdHKyACZtQNbe9M8SZum8EZo etwA== X-Gm-Message-State: ABy/qLb6/odwhiOo8fYVWG2Iv8iRM5VcpaFoOKb7ftDtxA7redKZhl8/ okFsBlKfsw5q910PH/a+qybjZaixhTsREijfNsgLjnIM X-Google-Smtp-Source: APBJJlHz2Tt+srhAdvsJPxmZ0fE8pLAc+LbjBi6Cb/R/kX35fWg0I+fEObztUYTL8kthW4vNDwoloP+1AJ/Ihc/+fWM= X-Received: by 2002:a05:6e02:1526:b0:345:aee1:eae2 with SMTP id i6-20020a056e02152600b00345aee1eae2mr7661326ilu.20.1689807070980; Wed, 19 Jul 2023 15:51:10 -0700 (PDT) List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 References: <88e0d702-cb86-b922-a5ce-82fe32e66a40@nomadlogic.org> In-Reply-To: <88e0d702-cb86-b922-a5ce-82fe32e66a40@nomadlogic.org> From: Scott Gasch Date: Wed, 19 Jul 2023 15:50:44 -0700 Message-ID: Subject: Re: Swap filling up, usermode process swap usage doesn't explain To: Pete Wright Cc: freebsd-questions Content-Type: multipart/alternative; boundary="0000000000003471af0600dedee9" X-Rspamd-Queue-Id: 4R5rdr1d3Nz46B7 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated --0000000000003471af0600dedee9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable -cc hackers The laundry is abnormally high and doesn't seem to go down. My understanding of it is that it's pages that are available for reuse and have to be zeroed by some kernel thread that does that job(?). Typically this filling up swap with no clear usermode culprit happens after the machine has been running for ~10 days and I eventually reboot. I can say for sure laundry is nowhere near this full after a reboot for several days. On Wed, Jul 19, 2023 at 3:40=E2=80=AFPM Pete Wright w= rote: > > > On 7/19/23 15:11, Scott Gasch wrote: > > Yes, I'm using ZFS. Here's what top says: > > > > last pid: 88926; load averages: 1.20, 0.96, 0.87 > > up 5+17:48:34 15:09:58 > > 274 processes: 1 running, 272 sleeping, 1 zombie > > CPU: 1.8% user, 0.0% nice, 0.5% system, 0.0% interrupt, 97.8% idle > > Mem: 1844M Active, 7777M Inact, 77G Laundry, 35G Wired, 750M Buf, 3367M > Free > > ARC: 24G Total, 2878M MFU, 18G MRU, 21M Anon, 119M Header, 2622M Other > > 18G Compressed, 25G Uncompressed, 1.33:1 Ratio > > Swap: 144G Total, 11G Used, 133G Free, 7% Inuse > > > > If I leave this alone it will grow to consume all available swap space. > > I'll try your fix with the sysctl knob and see what happens... I hope > > this is it, I've been fighting this for a while now. > > worth a shot, but 24G of ARC isn't that bad, especially if you are doing > quite a bit of disk i/o. > > i'm more interested in the 77G of Laundry memory, that seems like quite > a bit. but i don't know your workload so not sure... > > -p > > -- > Pete Wright > pete@nomadlogic.org > @nomadlogicLA > --0000000000003471af0600dedee9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
-cc hackers

The laundry is abnorm= ally high and doesn't seem to go down.=C2=A0 My understanding of it is = that it's pages that are available for reuse and have to be zeroed by s= ome kernel thread that does that job(?).

Typically this = filling up swap with no clear usermode=C2=A0culprit happens after the machi= ne has been running for ~10 days and I eventually reboot.=C2=A0 I can say f= or sure laundry is nowhere near this full after a reboot for several days.<= /div>




On Wed, Jul 19= , 2023 at 3:40=E2=80=AFPM Pete Wright <pete@nomadlogic.org> wrote:


On 7/19/23 15:11, Scott Gasch wrote:
> Yes, I'm using ZFS.=C2=A0 Here's what top says:
>
> last pid: 88926; =C2=A0load averages: =C2=A01.20, =C2=A00.96, =C2=A00.= 87=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0
>=C2=A0 =C2=A0 =C2=A0up 5+17:48:34 =C2=A015:09:58
> 274 processes: 1 running, 272 sleeping, 1 zombie
> CPU: =C2=A01.8% user, =C2=A00.0% nice, =C2=A00.5% system, =C2=A00.0% i= nterrupt, 97.8% idle
> Mem: 1844M Active, 7777M Inact, 77G Laundry, 35G Wired, 750M Buf, 3367= M Free
> ARC: 24G Total, 2878M MFU, 18G MRU, 21M Anon, 119M Header, 2622M Other=
>=C2=A0 =C2=A0 =C2=A0 =C2=A018G Compressed, 25G Uncompressed, 1.33:1 Rat= io
> Swap: 144G Total, 11G Used, 133G Free, 7% Inuse
>
> If I leave this alone it will grow to consume all available swap space= .=C2=A0
> I'll try your fix with the sysctl knob and see what happens...=C2= =A0 I hope
> this is it, I've been fighting this for a while now.

worth a shot, but 24G of ARC isn't that bad, especially if you are doin= g
quite a bit of disk i/o.

i'm more interested in the 77G of Laundry memory, that seems like quite=
a bit.=C2=A0 but i don't know your workload so not sure...

-p

--
Pete Wright
pete@nomadlogic.or= g
@nomadlogicLA
--0000000000003471af0600dedee9-- From nobody Wed Jul 19 23:15:05 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R5s9y1Vq8z4njBy for ; Wed, 19 Jul 2023 23:15:34 +0000 (UTC) (envelope-from scott.gasch@gmail.com) Received: from mail-il1-x136.google.com (mail-il1-x136.google.com [IPv6:2607:f8b0:4864:20::136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R5s9x01k4z4FPC for ; Wed, 19 Jul 2023 23:15:33 +0000 (UTC) (envelope-from scott.gasch@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=H8Q87Zvn; spf=pass (mx1.freebsd.org: domain of scott.gasch@gmail.com designates 2607:f8b0:4864:20::136 as permitted sender) smtp.mailfrom=scott.gasch@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-il1-x136.google.com with SMTP id e9e14a558f8ab-34574eb05f4so1516625ab.0 for ; Wed, 19 Jul 2023 16:15:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689808531; x=1692400531; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=dcJJB/Ptnw0ow414DuFKPpwvaO0V8WSgfAf2KKEETtw=; b=H8Q87ZvnxkGf0DlvNLzPm+6t0R0g1ZgMgHcd5lYxIy0UlW+VAxjDKZJdBnTCXz8eh6 +GbQYlmq5/W88VvH1lso8e9m+VAviR8D4z9nwFDis60LDDrDI0oJMqi6+D1JE9FtWV3R Z3e7kBH3z6US2EY7tzs7XLv7FV9lT26+J/wT5Zl5kI18owpsVHThUXSRelJODlf4mOeT eP15Cb2x2QPK4IE4qTe+XfPCGTk459OuRfYiXgYye8sj8je/HVfTHUfmjlF9R742d5Zf Gx+5Q7A5inzSe40nl3VdOtZRQm2PLBAkzi1k55Oz3t5WPS/PFxD/sHiorc9001AJGC7L imag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689808531; x=1692400531; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dcJJB/Ptnw0ow414DuFKPpwvaO0V8WSgfAf2KKEETtw=; b=JylHSDwpqhLaLqDCJWhfuEqDBjE/VM0udOidn2v73n9sUm8lBwyskHDo7JVOSFj2U+ HrUYabMlKq5rKmdxe3kMgay2JdkOcJ1eBXpAoLMGwvATGW7ww7bZsGqz4kiPkLCYJsmv 5CQnTySsjNwAzv00JR9eqCSfuYhf4PHe62JJ3tL7dQn05wzkjVddbxNtIMPg4Ja4t7MG hkhlFeneF8lHRw99i+PCzjMqCax9n3YT6ybxDFDBy9bQwSU1DCDKDc1Bo1J59KSWLwRP Vzy2Il32hISn/G3wbTYkTow9FWyk0uKvi1O+fwMzBNTyrm4AeQYh0tKHFEc3A1jR3hfz 9BhQ== X-Gm-Message-State: ABy/qLZYbI9wz/qa2z62tyqD6xg776ltZG0d4SqWxLnv+O9dm1twrxSt KwgJbEg7Kadlurtml2nckKJuOuFXu3cf0jMNOoXZcoff X-Google-Smtp-Source: APBJJlF3xsN2Ke8ELGeK978dnFR9I+Vsooj1l1vbOhugD5sP7VFCbdc+Xwu7GNgrBih/q6lBt0L12vWLS+XQFSlcdNo= X-Received: by 2002:a05:6e02:1e06:b0:346:624e:29eb with SMTP id g6-20020a056e021e0600b00346624e29ebmr518642ila.6.1689808531145; Wed, 19 Jul 2023 16:15:31 -0700 (PDT) List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Scott Gasch Date: Wed, 19 Jul 2023 16:15:05 -0700 Message-ID: Subject: Re: Swap filling up, usermode process swap usage doesn't explain To: freebsd-questions Content-Type: multipart/alternative; boundary="0000000000003cc79d0600df35a4" X-Spamd-Result: default: False [-0.42 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_SPAM_MEDIUM(0.83)[0.826]; NEURAL_SPAM_SHORT(0.65)[0.654]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_BASE64_TEXT(0.10)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; BLOCKLISTDE_FAIL(0.00)[2607:f8b0:4864:20::136:server fail]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::136:from]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MLMMJ_DEST(0.00)[freebsd-questions@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4R5s9x01k4z4FPC X-Spamd-Bar: / --0000000000003cc79d0600df35a4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Replying to my own post with more info... I tried stopping my wireguard jail and unloading the if_wg kmod and it did not affect the swap memory usage. Not sure if that lets wireguard off the hook or not though. If someone who understands kernel memory could chime in... it looks to me like the aggregate swap usage of usermode processes is nowhere near the total swap space used so I suspect something in kernel mode. Does this make sense or is there another explanation? Thx, Scott On Wed, Jul 19, 2023 at 7:49=E2=80=AFAM Scott Gasch = wrote: > I am running a 13.2-RELEASE GENERIC kernel and seeing a pattern where, > after about 10 days of uptime, my swap begins to fill up. > > # swapinfo -h > Device Size Used Avail Capacity > /dev/ada0p3 48G 3.6G 44G 7% > /dev/ada1p3 48G 3.6G 44G 7% > /dev/ada2p3 48G 3.6G 44G 7% > Total 144G 11G 133G 7% > > So, 11G of total swap space. What's using it? > > # systat -swap > /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10 > Load Average |||||| > > Device/Path Size Used |0% /10 /20 /30 /40 / 60\ 70\ 80\ 90= \ > 100| > ada0p3 48G 3660M XXX > ada1p3 48G 3666M XXX > ada2p3 48G 3664M XXX > Total 144G 11G XXX > > Pid Username Command Swap/Total Per-Process Per-System > 14703 scott python3.8 4M / 154M 2% 0% > 2451 scott rclone 4M / 934M 0% 0% > 2452 scott rclone 3M / 1G 0% 0% > 73827 scott bash 1M / 17M 6% 0% > 39416 scott tmux 968K / 54M 1% 0% > 41661 scott bash 828K / 17M 4% 0% > 15727 scott bash 808K / 17M 4% 0% > 39420 scott bash 804K / 17M 4% 0% > 2455 scott bash 544K / 15M 3% 0% > 39367 scott tmux 512K / 15M 3% 0% > 2447 scott bash 376K / 15M 2% 0% > 2450 scott bash 364K / 15M 2% 0% > 2453 scott bash 324K / 15M 2% 0% > 2454 scott bash 316K / 15M 2% 0% > 2445 scott bash 312K / 15M 2% 0% > 44937 scott bash 304K / 17M 1% 0% > 2458 scott bash 72K / 15M 0% 0% > > At least they agree about it being 11G. Is this kernel memory being page= d > out to swap? The machine has 128G of physical memory and isn't under ver= y > heavy load at the moment. > > I suspect this is a bug in some kernel module... possibly > wireguard because I run wireguard in a vnet jail and didn't observe this > problem until setting that up. But I don't have any hard evidence. > > I've tried to mitigate this via swapoff -a. This works once but the next > day swap will be back, even fuller. I've been doing regular reboots to > fix this but would like to get to the bottom of it. If left alone, swap > will > fill up and the machine will get into a "not quite hung" but unusable and > useless state. > > Am I off-base with my suspicion that this is kernel mode memory? Can > someone teach me how to diagnose the status of kernel mode memory heap? > > Thx, > Scott > > --0000000000003cc79d0600df35a4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: base64 PGRpdiBkaXI9Imx0ciI+UmVwbHlpbmcgdG8gbXkgb3duIHBvc3Qgd2l0aCBtb3JlIGluZm8uLi4g SSB0cmllZCBzdG9wcGluZyBteSB3aXJlZ3VhcmQgamFpbCBhbmQgdW5sb2FkaW5nIHRoZSBpZl93 ZyBrbW9kIGFuZCBpdCBkaWQgbm90IGFmZmVjdCB0aGUgc3dhcCBtZW1vcnkgdXNhZ2UuwqAgTm90 IHN1cmUgaWYgdGhhdCBsZXRzIHdpcmVndWFyZCBvZmYgdGhlIGhvb2sgb3Igbm90IHRob3VnaC48 ZGl2Pjxicj48L2Rpdj48ZGl2PklmIHNvbWVvbmUgd2hvIHVuZGVyc3RhbmRzIGtlcm5lbCBtZW1v cnkgY291bGQgY2hpbWUgaW4uLi4gaXQgbG9va3MgdG8gbWUgbGlrZSB0aGUgYWdncmVnYXRlIHN3 YXAgdXNhZ2Ugb2YgdXNlcm1vZGUgcHJvY2Vzc2VzIGlzIG5vd2hlcmUgbmVhciB0aGUgdG90YWwg c3dhcCBzcGFjZSB1c2VkIHNvIEkgc3VzcGVjdCBzb21ldGhpbmcgaW4ga2VybmVsIG1vZGUuwqAg RG9lcyB0aGlzIG1ha2Ugc2Vuc2Ugb3IgaXMgdGhlcmUgYW5vdGhlciBleHBsYW5hdGlvbj88L2Rp dj48ZGl2Pjxicj48L2Rpdj48ZGl2PlRoeCw8L2Rpdj48ZGl2PlNjb3R0PC9kaXY+PGRpdj48YnI+ PC9kaXY+PC9kaXY+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj48ZGl2IGRpcj0ibHRyIiBj bGFzcz0iZ21haWxfYXR0ciI+T24gV2VkLCBKdWwgMTksIDIwMjMgYXQgNzo0OeKAr0FNIFNjb3R0 IEdhc2NoICZsdDs8YSBocmVmPSJtYWlsdG86c2NvdHQuZ2FzY2hAZ21haWwuY29tIj5zY290dC5n YXNjaEBnbWFpbC5jb208L2E+Jmd0OyB3cm90ZTo8YnI+PC9kaXY+PGJsb2NrcXVvdGUgY2xhc3M9 ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0 OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiPjxkaXYgZGlyPSJs dHIiPkkgYW0gcnVubmluZyBhIDEzLjItUkVMRUFTRSBHRU5FUklDIGtlcm5lbCBhbmQgc2VlaW5n IGEgcGF0dGVybiB3aGVyZSwgYWZ0ZXIgYWJvdXQgMTAgZGF5cyBvZiB1cHRpbWUsIG15IHN3YXAg YmVnaW5zIHRvIGZpbGwgdXAuPGJyPjxicj48Zm9udCBmYWNlPSJtb25vc3BhY2UiPiMgc3dhcGlu Zm8gLWg8L2ZvbnQ+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UiPkRldmljZSDCoCDCoCDCoCDC oCDCoCDCoCDCoFNpemUgwqAgwqAgVXNlZCDCoCDCoEF2YWlsIENhcGFjaXR5PGJyPi9kZXYvYWRh MHAzIMKgIMKgIMKgIMKgIMKgNDhHIMKgIMKgIDMuNkcgwqAgwqAgwqA0NEcgwqAgwqAgNyU8YnI+ L2Rldi9hZGExcDMgwqAgwqAgwqAgwqAgwqA0OEcgwqAgwqAgMy42RyDCoCDCoCDCoDQ0RyDCoCDC oCA3JTxicj4vZGV2L2FkYTJwMyDCoCDCoCDCoCDCoCDCoDQ4RyDCoCDCoCAzLjZHIMKgIMKgIMKg NDRHIMKgIMKgIDclPGJyPlRvdGFsIMKgIMKgIMKgIMKgIMKgIMKgIMKgIDE0NEcgwqAgwqAgwqAx MUcgwqAgwqAgMTMzRyDCoCDCoCA3JTwvZm9udD48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PlNv LCAxMUcgb2YgdG90YWwgc3dhcCBzcGFjZS7CoCBXaGF0JiMzOTtzIHVzaW5nIGl0PzwvZGl2Pjxk aXY+PGJyPjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlIj4jIHN5c3RhdCAtc3dhcDwv Zm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSI+wqAgwqAgwqAgwqAgwqAgwqAg wqAgwqAgwqAgwqAgLzAgwqAgLzEgwqAgLzIgwqAgLzMgwqAgLzQgwqAgLzUgwqAgLzYgwqAgLzcg wqAgLzggwqAgLzkgwqAgLzEwPGJyPsKgIMKgIMKgTG9hZCBBdmVyYWdlIMKgIHx8fHx8fDxicj48 YnI+RGV2aWNlL1BhdGggwqAgwqAgwqAgU2l6ZSDCoFVzZWQgfDAlIMKgLzEwIMKgLzIwIMKgLzMw IMKgLzQwIMKgLyA2MFwgwqA3MFwgwqA4MFwgwqA5MFwgMTAwfDxicj5hZGEwcDMgwqAgwqAgwqAg wqAgwqAgwqAgNDhHIDM2NjBNIFhYWDxicj5hZGExcDMgwqAgwqAgwqAgwqAgwqAgwqAgNDhHIDM2 NjZNIFhYWDxicj5hZGEycDMgwqAgwqAgwqAgwqAgwqAgwqAgNDhHIDM2NjRNIFhYWDxicj5Ub3Rh bCDCoCDCoCDCoCDCoCDCoCDCoCAxNDRHIMKgIDExRyBYWFg8YnI+PGJyPlBpZCDCoCDCoFVzZXJu YW1lIMKgIENvbW1hbmQgwqAgwqAgU3dhcC9Ub3RhbCBQZXItUHJvY2VzcyDCoCDCoFBlci1TeXN0 ZW08YnI+wqAxNDcwMyBzY290dCDCoCDCoCDCoHB5dGhvbjMuOCDCoCDCoDRNIC8gMTU0TSDCoDIl IMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8YnI+wqAgMjQ1MSBzY290dCDCoCDCoCDCoHJjbG9uZSDC oCDCoCDCoCA0TSAvIDkzNE0gwqAwJSDCoCDCoCDCoCDCoCDCoCDCoCDCoDAlPGJyPsKgIDI0NTIg c2NvdHQgwqAgwqAgwqByY2xvbmUgwqAgwqAgwqAgM00gLyDCoCAxRyDCoDAlIMKgIMKgIMKgIMKg IMKgIMKgIMKgMCU8YnI+wqA3MzgyNyBzY290dCDCoCDCoCDCoGJhc2ggwqAgwqAgwqAgwqAgMU0g LyDCoDE3TSDCoDYlIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8YnI+wqAzOTQxNiBzY290dCDCoCDC oCDCoHRtdXggwqAgwqAgwqAgOTY4SyAvIMKgNTRNIMKgMSUgwqAgwqAgwqAgwqAgwqAgwqAgwqAw JTxicj7CoDQxNjYxIHNjb3R0IMKgIMKgIMKgYmFzaCDCoCDCoCDCoCA4MjhLIC8gwqAxN00gwqA0 JSDCoCDCoCDCoCDCoCDCoCDCoCDCoDAlPGJyPsKgMTU3Mjcgc2NvdHQgwqAgwqAgwqBiYXNoIMKg IMKgIMKgIDgwOEsgLyDCoDE3TSDCoDQlIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8YnI+wqAzOTQy MCBzY290dCDCoCDCoCDCoGJhc2ggwqAgwqAgwqAgODA0SyAvIMKgMTdNIMKgNCUgwqAgwqAgwqAg wqAgwqAgwqAgwqAwJTxicj7CoCAyNDU1IHNjb3R0IMKgIMKgIMKgYmFzaCDCoCDCoCDCoCA1NDRL IC8gwqAxNU0gwqAzJSDCoCDCoCDCoCDCoCDCoCDCoCDCoDAlPGJyPsKgMzkzNjcgc2NvdHQgwqAg wqAgwqB0bXV4IMKgIMKgIMKgIDUxMksgLyDCoDE1TSDCoDMlIMKgIMKgIMKgIMKgIMKgIMKgIMKg MCU8YnI+wqAgMjQ0NyBzY290dCDCoCDCoCDCoGJhc2ggwqAgwqAgwqAgMzc2SyAvIMKgMTVNIMKg MiUgwqAgwqAgwqAgwqAgwqAgwqAgwqAwJTxicj7CoCAyNDUwIHNjb3R0IMKgIMKgIMKgYmFzaCDC oCDCoCDCoCAzNjRLIC8gwqAxNU0gwqAyJSDCoCDCoCDCoCDCoCDCoCDCoCDCoDAlPGJyPsKgIDI0 NTMgc2NvdHQgwqAgwqAgwqBiYXNoIMKgIMKgIMKgIDMyNEsgLyDCoDE1TSDCoDIlIMKgIMKgIMKg IMKgIMKgIMKgIMKgMCU8YnI+wqAgMjQ1NCBzY290dCDCoCDCoCDCoGJhc2ggwqAgwqAgwqAgMzE2 SyAvIMKgMTVNIMKgMiUgwqAgwqAgwqAgwqAgwqAgwqAgwqAwJTxicj7CoCAyNDQ1IHNjb3R0IMKg IMKgIMKgYmFzaCDCoCDCoCDCoCAzMTJLIC8gwqAxNU0gwqAyJSDCoCDCoCDCoCDCoCDCoCDCoCDC oDAlPGJyPsKgNDQ5Mzcgc2NvdHQgwqAgwqAgwqBiYXNoIMKgIMKgIMKgIDMwNEsgLyDCoDE3TSDC oDElIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8YnI+wqAgMjQ1OCBzY290dCDCoCDCoCDCoGJhc2gg wqAgwqAgwqAgwqA3MksgLyDCoDE1TSDCoDAlIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8YnI+PC9m b250PjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+QXQgbGVhc3QgdGhleSBhZ3JlZSBhYm91dCBp dCBiZWluZyAxMUcuwqAgSXMgdGhpcyBrZXJuZWwgbWVtb3J5IGJlaW5nIHBhZ2VkIG91dCB0byBz d2FwP8KgIFRoZSBtYWNoaW5lIGhhcyAxMjhHIG9mIHBoeXNpY2FsIG1lbW9yeSBhbmQgaXNuJiMz OTt0IHVuZGVyIHZlcnkgaGVhdnkgbG9hZCBhdCB0aGUgbW9tZW50LsKgwqA8L2Rpdj48ZGl2Pjxi cj48L2Rpdj48ZGl2Pkkgc3VzcGVjdCB0aGlzIGlzIGEgYnVnIGluIHNvbWUga2VybmVsIG1vZHVs ZS4uLiBwb3NzaWJseSB3aXJlZ3VhcmTCoGJlY2F1c2UgSSBydW4gd2lyZWd1YXJkIGluIGEgdm5l dCBqYWlsIGFuZCBkaWRuJiMzOTt0IG9ic2VydmUgdGhpcyBwcm9ibGVtIHVudGlsIHNldHRpbmcg dGhhdCB1cC7CoCBCdXQgSSBkb24mIzM5O3QgaGF2ZSBhbnkgaGFyZCBldmlkZW5jZS48L2Rpdj48 ZGl2Pjxicj5JJiMzOTt2ZSB0cmllZCB0byBtaXRpZ2F0ZSB0aGlzIHZpYSBzd2Fwb2ZmIC1hLsKg IFRoaXMgd29ya3Mgb25jZSBidXQgdGhlIG5leHQ8YnI+ZGF5IHN3YXAgd2lsbCBiZSBiYWNrLCBl dmVuIGZ1bGxlci7CoCBJJiMzOTt2ZSBiZWVuIGRvaW5nIHJlZ3VsYXIgcmVib290cyB0byBmaXgg dGhpcyBidXQgd291bGQgbGlrZSB0byBnZXQgdG8gdGhlIGJvdHRvbSBvZiBpdC7CoCBJZiBsZWZ0 IGFsb25lLCBzd2FwIHdpbGw8YnI+ZmlsbCB1cCBhbmQgdGhlIG1hY2hpbmUgd2lsbCBnZXQgaW50 byBhICZxdW90O25vdCBxdWl0ZSBodW5nJnF1b3Q7IGJ1dCB1bnVzYWJsZSBhbmQgdXNlbGVzcyBz dGF0ZS48YnI+PGJyPkFtIEkgb2ZmLWJhc2Ugd2l0aCBteSBzdXNwaWNpb24gdGhhdCB0aGlzIGlz IGtlcm5lbCBtb2RlIG1lbW9yeT8gQ2FuIHNvbWVvbmUgdGVhY2ggbWUgaG93IHRvIGRpYWdub3Nl IHRoZSBzdGF0dXMgb2Yga2VybmVsIG1vZGUgbWVtb3J5IGhlYXA/PGJyPjxicj5UaHgsPGJyPlNj b3R0PC9kaXY+PGRpdj48YnI+PC9kaXY+PC9kaXY+DQo8L2Jsb2NrcXVvdGU+PC9kaXY+DQo= --0000000000003cc79d0600df35a4-- From nobody Thu Jul 20 02:34:15 2023 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R5xbH16rlz4mtKf for ; Thu, 20 Jul 2023 02:34:19 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R5xbH0Bhhz46c7 for ; Thu, 20 Jul 2023 02:34:19 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1689820459; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ovvMfNrDzDmGh9eNZgx/4yrUMBstIMGjgpJALi9OniE=; b=xhGihEyO8zZPMf3FbHzUKlEFkTVkC5EF3qFummK60UuV2tM6h20zBlbOZ20veLoQdoxqmf HivNnkC2Da/mWh5O7IND4VbizleGVc27jq6f0YuDoh3jYnkAIxvNwywvMP5TaKsGHpWmkd 0qhOZUuYPBaY1yi6RmZzKwlw2ZAif6EpzeIOuE81KPNY4noP6Kxr3h4D7kdabSt7X1Tzwq 1gU0sb/N4rkQSMAbliLd1BjV82sRYgNk1pQNSVnaeAZUYljFnna6db1KKgIXOlVaJ5tWML 9unfgxoWv85I0fBad9CB3kExX1z3W2X0ymx8ZA84/QQa7k/7/xH7OFIem76fgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1689820459; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ovvMfNrDzDmGh9eNZgx/4yrUMBstIMGjgpJALi9OniE=; b=i/GuUXo7wcxUWgYQOucdbkenrt4oypDrQWogxpbXDi8wovvuaMpiVJJZSg/uFg0ADQQ21/ tNcrzHBAYZ363xgifR7DcSFduWX2b7+4PuFhf7HH1uK/mx/Ded/tP6EwZKrJVMrFcVAQgQ iEmIYj3ZP1knyvTdL5g7P3dDGgT27qw6eySVNxYwhNTxx53etPl03ssvdFQ56CyjF43QAR 8zjYcOvMpGeNA3WmyB0oIKw+O44GTI3cWquE/Yh5/k48GZ2SXb4H6JZzBe+j21lAZhodht jmFdGiomMBks4UN+zcXZkfY/U4SMJntepdGbSGjJo/Qo/FmPJLvB2eEtE707wQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1689820459; a=rsa-sha256; cv=none; b=CWUIOs56xgzWHVAd+sj9uz1evrn0ZYSfWRFwbnMU4uWy8EsJUZqO2+u9ka44IZ2kBrUf7y zm18f5THu7Dhc625wxbaGHEX3LJPgjRksPgLTaTDWwBTo5MkaOa6BV0YIawR05h2/YOVhf qhbX+2qRIDzQnk+JHaR1EAdgz0WePYfJ0mLhz8ksbHbdY50EpHgu5FFgK7gQe4l5m5KdYY nITnHaidfgopOJcCb6c+ttPhHd04T4Ptz3jLODlICaOzG4ouK5q/lXBw6gD4jZFAaDtQup tFjHXA5WtjQG4AZAvQZrVOUpNYQY7vWOmQ2PHAnxx4T7/H+ykgUl3QSY6ropUw== Received: from [IPV6:2001:470:1f1c:a0::2] (tunnel642390-pt.tunnel.tserv1.lon2.ipv6.he.net [IPv6:2001:470:1f1c:a0::2]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: grahamperrin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4R5xbG56nRzGgT for ; Thu, 20 Jul 2023 02:34:18 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Message-ID: Date: Thu, 20 Jul 2023 03:34:15 +0100 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: Swap filling up, usermode process swap usage doesn't explain To: questions@freebsd.org References: <88e0d702-cb86-b922-a5ce-82fe32e66a40@nomadlogic.org> Content-Language: en-US From: Graham Perrin Organization: FreeBSD In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------m4Qahi0JUk259ZF6oVIdnxML" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------m4Qahi0JUk259ZF6oVIdnxML Content-Type: multipart/mixed; boundary="------------6DZsoTFsONai5pVBkP3J0zG2"; protected-headers="v1" From: Graham Perrin To: questions@freebsd.org Message-ID: Subject: Re: Swap filling up, usermode process swap usage doesn't explain References: <88e0d702-cb86-b922-a5ce-82fe32e66a40@nomadlogic.org> In-Reply-To: --------------6DZsoTFsONai5pVBkP3J0zG2 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 T24gMTkvMDcvMjAyMyAyMzo1MCwgU2NvdHQgR2FzY2ggd3JvdGU6DQoNCj4gLWNjIGhhY2tl cnMNCj4NCj4gVGhlIGxhdW5kcnkgaXMgYWJub3JtYWxseSBoaWdoIGFuZCBkb2Vzbid0IHNl ZW0gdG8gZ28gZG93bi4g4oCmDQoNCg0KSW4gYWRkaXRpb24gdG8gd2hhdCdzIHdpdGggZnJl ZWJzZC1oYWNrZXJzQA0KDQpTdWJzY3JpYmVycyB0byBmcmVlYnNkLXF1ZXN0aW9uc0AgbWF5 IGJlIHVuYXdhcmUgb2YgdGhlIG90aGVyIHJlY2VudCANCmRpc2N1c3Npb24gb2YgbGF1bmRy eSB3aXRoIGZyZWVic2QtY3VycmVudEAuIFRoaXMsIGFuZCBtb3JlIChmcm9tIDIwMTggDQph bmQgMjAyMSksIGNhbiBiZSBmb3VuZCB2aWEgDQo8aHR0cHM6Ly9mcmVlYnNkLm1hcmttYWls Lm9yZy9zZWFyY2gvP3E9aHVnZSthbW91bnQrbGF1bmRyeSttZW1vcnkrbm90K2JlaW5nK2Ns ZWFuZWQrdXA+Lg0KDQo= --------------6DZsoTFsONai5pVBkP3J0zG2-- --------------m4Qahi0JUk259ZF6oVIdnxML Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsFAmS4nScFAwAAAAAACgkQt2dIb0oY1AvL bQ/9G4uKUD1lh940dGxaHhu/TXiJf4amvRvw2kTXLVw9Dt94n+CCt6MSJDPDqOyOgY/x8rYXcyDP kFLL8LuWcuaUBG+SijWS8yq1IL7CczzKnImUsxlB/cfCvuxVFbTMOT6wBIztUVz9Oa1zoXzKPaD2 D9IDzial1U4FK9g3De4glhGrqM3VaHirD116I5NeqfhkQDGiKfBJDQqBLi3H5Icw46Lio8rBaT0W Boq9Ly7hjFrFPovgHMpCw6vTNT+d6J/m9CxNbZ6gwTY8hVBp2VuhevjmLeqjgc9LegulQRXe7k8Z wsQkCu3yTFhmkK1dTJCT5T6qH8lyVa70+cxiAnokHNsYhj84rYdet6H92yUhzL+/yWRhwTPO0lif JbnaTjW8D1GWajWnxC3f4qUkpUdGwJmTAzQb53gbnnN//K5jsaLflbaNgA1HXzJwlxPQ+UDN+yIo EkNQlCgwrQlviCAd/PMjJZ2XQCSaUhZWEZejUBnGp0R8yTOleOc9i8XWvFf17Vaqk9gdAk6gRj6O /VAin4bXsNOHFUSWxdgz6/5aDFR/2tohXdk/zpTWrB1Z5LPLjysJdX8FSvfCEdE2Qx0fL7eGEq1h ArDjl1AWdtUb3AWscNiL6AhcNLC4hMXgsgm2VQRWZjAm4pqxL4NG+bUJcxKuW7eRgQOAjLCAK/5j 1Mw= =yD9Z -----END PGP SIGNATURE----- --------------m4Qahi0JUk259ZF6oVIdnxML-- From nobody Thu Jul 20 05:53:20 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R62156vfCz4nvPR for ; Thu, 20 Jul 2023 05:53:29 +0000 (UTC) (envelope-from mlist@zwn24.nl) Received: from nb24.zwn24.nl (nb24.zwn24.nl [45.80.171.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "zwn24.nl", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R62150S5Hz3tDR for ; Thu, 20 Jul 2023 05:53:28 +0000 (UTC) (envelope-from mlist@zwn24.nl) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zwn24.nl header.s=sel1 header.b=vPjHewX7; spf=pass (mx1.freebsd.org: domain of mlist@zwn24.nl designates 45.80.171.6 as permitted sender) smtp.mailfrom=mlist@zwn24.nl; dmarc=pass (policy=reject) header.from=zwn24.nl Received: from nb24.zwn24.nl (nb24 [45.80.171.6]) by nb24.zwn24.nl (Postfix) with ESMTP id B743C1FCCAA for ; Thu, 20 Jul 2023 07:53:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=zwn24.nl; h=message-id :date:mime-version:subject:reply-to:references:to:from :in-reply-to:content-type:content-transfer-encoding; s=sel1; bh= pwgnXFwruZtzjNrCcyOfiVKov0U=; b=vPjHewX7uEg65x5eEYr8xiDYGhJLJepR qtprCjr0Ax6k8E5TQt/LsBr9+sPZWFqwDEX/6EcmCUtMIJX0ye/scDk0ywURgeyH FEqw0DwFwTqGiui9/nDf7JJwvfSh7yjnelF2SFBy+BFXBlpNeSo+pXm2qWqShGOU eDmMCwQ7sTJy5qeeIf0b3sNznjF9WmyuPq5H0Lc2XmUXnTWkOIlPHOXs4gfM+KoG SnGbStpZvAFEHoULWCyyiUj2/kP/JjaGwsDxvj1bAFcq1bgwAV4e9Ur/YmEdOZfe x5h9YgrPdgN2nGb2Sr2UQgEU8eeLMyVYdIUWkVyOxySjVPUIV4ymjA== DomainKey-Signature: a=rsa-sha1; c=nofws; d=zwn24.nl; h=message-id:date :mime-version:subject:reply-to:references:to:from:in-reply-to :content-type:content-transfer-encoding; q=dns; s=sel1; b=YPBuGa I0ty5h4xd7uILeFp1hJy2EAJ4e8a2mQActlEkyakh1EXCouDN24myiDtQichXovr Ds/WLNGbzDGPwlxmYrAh3YA78XH1a47dk5hn5hXAs0Zp2hLWEHruo2bxSB+F3Y0H pNnEAS29wBc0PtE4r9cYiYsjAR7LhvySZz8yS5LfAzoETjZtOFaYLYMcbMvYXfz1 2WTug3WPVyWh9Pg9eYE7eG8tXEXH75DRM42JlSdr4zA5YxzXK7X+GbTFp0rR1z43 W4yUNRz5GsNE/E3tGlDXwkkHTbsADuZSMwQ3XL+hzmm2otgP3B+m5uh7uENxzVNm hMd5Ng+pMir2UjYA== Received: from [10.10.11.151] (unknown [10.10.11.151]) by nb24.zwn24.nl (Postfix) with ESMTPSA id A041E1FCBFD for ; Thu, 20 Jul 2023 07:53:19 +0200 (CEST) Message-ID: Date: Thu, 20 Jul 2023 07:53:20 +0200 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Fwd: Swap filling up, usermode process swap usage doesn't explain Content-Language: nl Reply-To: mlist@zwn24.nl References: To: freebsd-questions@FreeBSD.org From: Jack Raats Organization: Jack Raats, Steenbergen, The Netherlands In-Reply-To: X-Forwarded-Message-Id: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.11 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.94)[-0.936]; DMARC_POLICY_ALLOW(-0.50)[zwn24.nl,reject]; NEURAL_HAM_SHORT(-0.28)[-0.279]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[zwn24.nl:s=sel1]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; MLMMJ_DEST(0.00)[freebsd-questions@FreeBSD.org]; HAS_REPLYTO(0.00)[mlist@zwn24.nl]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:206238, ipnet:45.80.168.0/22, country:NL]; DKIM_TRACE(0.00)[zwn24.nl:+]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; REPLYTO_ADDR_EQ_FROM(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4R62150S5Hz3tDR X-Spamd-Bar: --- Op 20-07-2023 om 00:40 schreef Pete Wright: > > > On 7/19/23 15:11, Scott Gasch wrote: >> Yes, I'm using ZFS.  Here's what top says: >> >> last pid: 88926;  load averages:  1.20,  0.96, >>  0.87                         up 5+17:48:34  15:09:58 >> 274 processes: 1 running, 272 sleeping, 1 zombie >> CPU:  1.8% user,  0.0% nice,  0.5% system,  0.0% interrupt, 97.8% idle >> Mem: 1844M Active, 7777M Inact, 77G Laundry, 35G Wired, 750M Buf, >> 3367M Free >> ARC: 24G Total, 2878M MFU, 18G MRU, 21M Anon, 119M Header, 2622M Other >>       18G Compressed, 25G Uncompressed, 1.33:1 Ratio >> Swap: 144G Total, 11G Used, 133G Free, 7% Inuse >> >> If I leave this alone it will grow to consume all available swap >> space.  I'll try your fix with the sysctl knob and see what >> happens...  I hope this is it, I've been fighting this for a while now. At my systems ZFS is very memory hungry. What is the status of the zpool? Gr., Jack From nobody Thu Jul 20 10:17:15 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R67sW4h1Wz4nZGP for ; Thu, 20 Jul 2023 10:17:19 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R67sW4GCYz4Ly9; Thu, 20 Jul 2023 10:17:19 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1689848239; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=rJq7x+fnHneTt1FPUcNoyR9cBWTrACCkaNzfty6Ze/o=; b=OfYWx9bXJsVOTcsvFUGrX7FjeYU2hGIYA5m33/UR7oIyqPMzaSXEz62KsLzIUjPBMOuWDO cBSImdE2kUDKEb9V6ZCujcL6CjwK7z2LtYJqw5ui5rsFWY2djY34+oPjgIIbMcSFhcu8RF NJfaJRCwjZzbTIg2eeVDI9U/dAxvFeckZFR4S8GydqyG8uythj2lrIUY08ZecYihF5y8Ld 2E8WZGPf8gP0+dYR3YA40kZtW5mFbgXN7t6JYpht1CNKxKt0CcCG+sM//fa3lIH/TINGjX mH/byoWKm17l9OOTzYeNhACY9BTXgZoavdji8amTmaxdC6wJ1HTzi95dTxq6HQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1689848239; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=rJq7x+fnHneTt1FPUcNoyR9cBWTrACCkaNzfty6Ze/o=; b=DZl/zdGZSxEtcU6xDGZLJm09waaqAf7na1IFUW6YAgOvEhZXzwLPNsAgw5+lZnsH5D6LJh tdYgx6+V5pVB+hR6RT6ofatSJdt9Do0r0xsR2bVGWtF4qgKmVLIZ5JDKgckR3tVPxfW7+l VAS6ej+UDELJBVxddYJFfD60I6sb6L9E+7seO3Qsj58IM1dRiUn0ilq16hi/lmQDZazcmy PBq29eY8S68A+jiwZ48q//3B2OHRRkEJjcqbDSdtQYZ6XbB91mUZo8ZW9vK/Ngtw9YVekw 7wGUaYBCaR++vskCFJJrlwIWxBDmgfGuQ/k5gaQIihsO16eC/hkbpiK4Vfg0jA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1689848239; a=rsa-sha256; cv=none; b=gqb/buyw0+KRZJ1giJ6DoQyTN9MHBM6W2wak+dcYCOygfoug1enZWON1QfCH/oLwL4Q6Cn DFm9+ovzrwUeZLypz502oAWgEvos6vqAs3qHL294n97YBqphLo31oqsLgS1kUp4MgOIs13 nlTO8KJhkdCf3c7V6PM66jcAqoNXSh8H8Y5zu/xG3rRHD1+y37heR6LManVhJipOqwEmNh 1y0rm6glIj5xZST/7yesRO5cupY+mnpqPAGJtT4GwAhS8uOxvS1y1t+i/0GMSlsCh8L1A2 172VudZWoTTi0Ff5nu4WZ0vBplNaVdraIb9HBRW2BZuRuldzgWHyt7EDJhHHbg== Received: from [192.168.1.10] (80-42-66-93.dynamic.dsl.as9105.com [80.42.66.93]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: grahamperrin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4R67sW1PqFzQhL; Thu, 20 Jul 2023 10:17:19 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Message-ID: <2137f9e4-07d0-1b89-1a48-daea622702a2@freebsd.org> Date: Thu, 20 Jul 2023 11:17:15 +0100 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: Fwd: Swap filling up, usermode process swap usage doesn't explain To: Jack Raats References: Content-Language: en-US Cc: FreeBSD questions From: Graham Perrin Organization: FreeBSD In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------wbvsKjVJZLzB1tdJu6h65U1h" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------wbvsKjVJZLzB1tdJu6h65U1h Content-Type: multipart/mixed; boundary="------------95Fl3x16KQVeGFhI9hDJtA7C"; protected-headers="v1" From: Graham Perrin To: Jack Raats Cc: FreeBSD questions Message-ID: <2137f9e4-07d0-1b89-1a48-daea622702a2@freebsd.org> Subject: Re: Fwd: Swap filling up, usermode process swap usage doesn't explain References: In-Reply-To: --------------95Fl3x16KQVeGFhI9hDJtA7C Content-Type: multipart/alternative; boundary="------------xr1kb37Qa3VH4ggofbemqZIF" --------------xr1kb37Qa3VH4ggofbemqZIF Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 T24gMjAvMDcvMjAyMyAwNjo1MywgSmFjayBSYWF0cyB3cm90ZToNCj4g4oCmDQo+DQo+IEF0 IG15IHN5c3RlbXMgWkZTIGlzIHZlcnkgbWVtb3J5IGh1bmdyeS4g4oCmDQoNCg0KV2hpY2gg dmVyc2lvbnMgb2YgRnJlZUJTRCwgZXhhY3RseSwgYXQgdGhvc2Ugc3lzdGVtcz8NCg0KdW5h bWUgLWFLVQ0KDQo= --------------xr1kb37Qa3VH4ggofbemqZIF Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On 20/07/2023 06:53, Jack Raats wrote:=
=E2=80=A6=

At my systems ZFS is very memory hungry. =E2=80=A6


Which versions of FreeBSD, exactly, at those systems?

uname -aKU

--------------xr1kb37Qa3VH4ggofbemqZIF-- --------------95Fl3x16KQVeGFhI9hDJtA7C-- --------------wbvsKjVJZLzB1tdJu6h65U1h Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsFAmS5CasFAwAAAAAACgkQt2dIb0oY1Aud 8g//YxwaAmNUxW4e8TLNrXFBfq2/PThseiRHA5PJkWMsE4i11x5MXVtfjcYp6lfnDnYypTGgJaSM Ao/UGrfaymdAp8K688jleyxvWhEQurgCIUPhul8PpRo5scHkY0R0qgM48fnyDfatqGXl26brOZzZ uTG+XaFSpQ/LLKfm/FWLLkJOZcU+PUEbLSFhMK8Ch7YDROHxElhnBtQXRckusElnoB+MMjIcQPYU UXuhzmbXQ2j/N9pzuTFPvLKYA+216ZN+nYd14YcMjQjd9BHQp3utzvjSCegK9ZYPeWVy2IIAgU4l vxjS0mOE1vP+6gCUvZwwzldv9dw/9cc4o/Q87hIJCz/vSGrvi8A/o2k9Zs5xGHKFuGG6VtESHugI KSOhnJGRb5As7evqzW96EWR1BUpG0mHwlGM9XaATsGzX25/1SGqHEIJzxGZDKP+02NSk6InWJKL3 WIN76X/bC9qXysA3sK6/BJckpZ3iq6B/LYERlQ6KBSaVNIuxtRuJwGfsXTeznw9qq1g/7gjb4frc JkKrmlAQ3+aYTtjDEDkWPDApl8I8edIklEvhZnw6265ctdkx0dQUc/1w8bHsvSLxBg4GVge6Y1/3 5tRyAZlsJVqiLniMrBt0CFhwY2onMscv9EtnwI37LRUKsb0Yb/sQuZKQPsXZZYNDr2tJlpvfAhx+ irE= =aDQd -----END PGP SIGNATURE----- --------------wbvsKjVJZLzB1tdJu6h65U1h-- From nobody Thu Jul 20 14:57:18 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R6G4j5jvWz4pFsq for ; Thu, 20 Jul 2023 14:57:25 +0000 (UTC) (envelope-from doug@safeport.com) Received: from fledge.watson.org (fledge.watson.org [147.160.157.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "A1-48603", Issuer "A1-48603" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R6G4j0JqQz3w5r for ; Thu, 20 Jul 2023 14:57:25 +0000 (UTC) (envelope-from doug@safeport.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=fail (mx1.freebsd.org: domain of doug@safeport.com does not designate 147.160.157.40 as permitted sender) smtp.mailfrom=doug@safeport.com; dmarc=none Received: from fledge.watson.org (doug@localhost [127.0.0.1]) by fledge.watson.org (8.16.1/8.16.1) with ESMTPS id 36KEvI7K007523 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Thu, 20 Jul 2023 14:57:18 GMT (envelope-from doug@safeport.com) Received: from localhost (doug@localhost) by fledge.watson.org (8.16.1/8.16.1/Submit) with ESMTP id 36KEvIxT007520 for ; Thu, 20 Jul 2023 14:57:18 GMT (envelope-from doug@safeport.com) X-Authentication-Warning: fledge.watson.org: doug owned process doing -bs Date: Thu, 20 Jul 2023 14:57:18 +0000 (UTC) From: doug@safeport.com Reply-To: doug@fledge.watson.org To: FreeBSD questions Subject: Re: Fwd: Swap filling up, usermode process swap usage doesn't explain In-Reply-To: <2137f9e4-07d0-1b89-1a48-daea622702a2@freebsd.org> Message-ID: <83cef040-8c5b-17cb-3d9a-f06d8369c7f3@safeport.com> References: <2137f9e4-07d0-1b89-1a48-daea622702a2@freebsd.org> List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Spamd-Result: default: False [-2.10 / 15.00]; R_SPF_FAIL(1.00)[-all]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MIME_GOOD(-0.10)[text/plain]; HAS_REPLYTO(0.00)[doug@fledge.watson.org]; FROM_NO_DN(0.00)[]; MLMMJ_DEST(0.00)[freebsd-questions@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:6405, ipnet:147.160.157.0/24, country:US]; TO_DN_ALL(0.00)[]; DMARC_NA(0.00)[safeport.com]; RCVD_COUNT_THREE(0.00)[3]; REPLYTO_DN_EQ_FROM_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; HAS_XAW(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4R6G4j0JqQz3w5r X-Spamd-Bar: -- On Thu, 20 Jul 2023, Graham Perrin wrote: > On 20/07/2023 06:53, Jack Raats wrote: > ? > > At my systems ZFS is very memory hungry. ? > > > Which versions of FreeBSD, exactly, at those systems? > > uname -aKU It does not really sound like your issue is similar to mine, but just in case ... The system in question runs several jails. I have this problem on an older system running FreeBSD 11.3 (can't update w/o being physically present) php 7.2, mysql 5.7 and apache 2.4. In my case some interaction between apache and mysql causes an increase in swap due to apache not freeing memory. For me when swap exceeds 50% swap space starts increasing very rapidly. I can recover swap space by restarting the various apache's This seems to be fixed in 12 and beyond. I have a 12 system that does not exhibit this behavior. From nobody Fri Jul 21 04:37:29 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R6cHV1tftz4p0bH for ; Fri, 21 Jul 2023 04:37:58 +0000 (UTC) (envelope-from scott.gasch@gmail.com) Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R6cHT3Jt2z3rts for ; Fri, 21 Jul 2023 04:37:57 +0000 (UTC) (envelope-from scott.gasch@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=YzFw1A3x; spf=pass (mx1.freebsd.org: domain of scott.gasch@gmail.com designates 2607:f8b0:4864:20::72b as permitted sender) smtp.mailfrom=scott.gasch@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-qk1-x72b.google.com with SMTP id af79cd13be357-76714caf466so153573085a.1 for ; Thu, 20 Jul 2023 21:37:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689914276; x=1690519076; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=hCQ1miFcJXhe9ML0bGJPSM0LtblXCNjH+dbh5dCx8EI=; b=YzFw1A3xXrGUKqOeEJ1fNXI8vcqJZb7E19PFZBrVp9eU6l26B36hAKrGXPwXr4P9Tt EEnJJsv0FqK06qdWoRSqvNIuUkfR9hMiehmtQLBgeU+a1g3W4oauTcTg7gYIpiN6XXd0 zOVx3mKEutuzEMdZ7x54rMZAW1xiB8ycDA925ZWcu7Au+z3ljz1FsoQ2go+N4CN6hZpS Lw8tLjfrXRxBxXOv49pYVe69ZHIuCnSnwTD9o9yEn98XhAezIGKNxIb7CQ1mrEyRJBRI anaJWbE/y+zvfCaJL9Ui8Wvnjh4bJ7rzor4coLr3S190aJPL7RbPbwVWjX+0vXX1Fx7Q 7VTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689914276; x=1690519076; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hCQ1miFcJXhe9ML0bGJPSM0LtblXCNjH+dbh5dCx8EI=; b=NQd2bQkTuA9z9j1fYCqcbeRIE4rXrgPWmDEF4wAIWYhYCZ5iQTLx+kWoMjpbsZzHbi JQdhD557fd8uCYCQImFzZTgfMoeiexqgIQLIQq+ftqnMvSDPYokiOawkbgLpPGrH7Uac AR9SzW/Sh52jAOwnBbR+FtS/YaUw+Rz6P73u15G7EzXKvHMTocCXa2VKDVWkPwXmF8/g C5rjcGo/eysa3xMD71lc+lJuphjkhPekm+K6rwWlsbGnYInufd5dMalpiV8CyCaZ9XH6 nWDYT/GbDUzHRxJBzLLCwkwbaOeXtYjAXSTpHnecA+GbM6d6IF+90cizBUx4d90DGsuS H5Hw== X-Gm-Message-State: ABy/qLb6mC+qyXfNpo3ttYIwvsTGbr+9XY03PE7zsynscO/xFVYtbnqD Slbl/Ae0UICWgBdzXDZdUPAf4+oobYONb8gQEgJDaUdn X-Google-Smtp-Source: APBJJlEIdmAgSXS+jBCf92TFMNZeHhrHeZxGsBRkJMSZp2dG4n23tSqMHkhLitKH1PFT3hezA0pltvZm+HilQmhgi80= X-Received: by 2002:a05:620a:2956:b0:767:2b4e:213 with SMTP id n22-20020a05620a295600b007672b4e0213mr1030535qkp.21.1689914276275; Thu, 20 Jul 2023 21:37:56 -0700 (PDT) List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Scott Gasch Date: Thu, 20 Jul 2023 21:37:29 -0700 Message-ID: Subject: Re: Swap filling up, usermode process swap usage doesn't explain To: freebsd-questions , Pete Wright Content-Type: multipart/alternative; boundary="000000000000236b100600f7d43f" X-Spamd-Result: default: False [-3.77 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.93)[-0.931]; NEURAL_HAM_SHORT(-0.84)[-0.843]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72b:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-questions@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4R6cHT3Jt2z3rts X-Spamd-Bar: --- --000000000000236b100600f7d43f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ok, I'm an idiot. I'm writing to confess and to maybe save someone else in the future. The issue was I mounted a tmpfs on /tmp and didn't specify an upper size limit. Invariably over time, /tmp would begin to fill up and my swap space would start to be used. Of course, I couldn't find any usermode process that was using the swap and I jumped to the conclusion that this had something to do with kernel memory. But really it was my own stupidity= . Thank you to Pete and others who tried to help. Scott On Wed, Jul 19, 2023 at 4:15=E2=80=AFPM Scott Gasch = wrote: > Replying to my own post with more info... I tried stopping my wireguard > jail and unloading the if_wg kmod and it did not affect the swap memory > usage. Not sure if that lets wireguard off the hook or not though. > > If someone who understands kernel memory could chime in... it looks to me > like the aggregate swap usage of usermode processes is nowhere near the > total swap space used so I suspect something in kernel mode. Does this > make sense or is there another explanation? > > Thx, > Scott > > > On Wed, Jul 19, 2023 at 7:49=E2=80=AFAM Scott Gasch wrote: > >> I am running a 13.2-RELEASE GENERIC kernel and seeing a pattern where, >> after about 10 days of uptime, my swap begins to fill up. >> >> # swapinfo -h >> Device Size Used Avail Capacity >> /dev/ada0p3 48G 3.6G 44G 7% >> /dev/ada1p3 48G 3.6G 44G 7% >> /dev/ada2p3 48G 3.6G 44G 7% >> Total 144G 11G 133G 7% >> >> So, 11G of total swap space. What's using it? >> >> # systat -swap >> /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /1= 0 >> Load Average |||||| >> >> Device/Path Size Used |0% /10 /20 /30 /40 / 60\ 70\ 80\ >> 90\ 100| >> ada0p3 48G 3660M XXX >> ada1p3 48G 3666M XXX >> ada2p3 48G 3664M XXX >> Total 144G 11G XXX >> >> Pid Username Command Swap/Total Per-Process Per-System >> 14703 scott python3.8 4M / 154M 2% 0% >> 2451 scott rclone 4M / 934M 0% 0% >> 2452 scott rclone 3M / 1G 0% 0% >> 73827 scott bash 1M / 17M 6% 0% >> 39416 scott tmux 968K / 54M 1% 0% >> 41661 scott bash 828K / 17M 4% 0% >> 15727 scott bash 808K / 17M 4% 0% >> 39420 scott bash 804K / 17M 4% 0% >> 2455 scott bash 544K / 15M 3% 0% >> 39367 scott tmux 512K / 15M 3% 0% >> 2447 scott bash 376K / 15M 2% 0% >> 2450 scott bash 364K / 15M 2% 0% >> 2453 scott bash 324K / 15M 2% 0% >> 2454 scott bash 316K / 15M 2% 0% >> 2445 scott bash 312K / 15M 2% 0% >> 44937 scott bash 304K / 17M 1% 0% >> 2458 scott bash 72K / 15M 0% 0% >> >> At least they agree about it being 11G. Is this kernel memory being >> paged out to swap? The machine has 128G of physical memory and isn't un= der >> very heavy load at the moment. >> >> I suspect this is a bug in some kernel module... possibly >> wireguard because I run wireguard in a vnet jail and didn't observe this >> problem until setting that up. But I don't have any hard evidence. >> >> I've tried to mitigate this via swapoff -a. This works once but the nex= t >> day swap will be back, even fuller. I've been doing regular reboots to >> fix this but would like to get to the bottom of it. If left alone, swap >> will >> fill up and the machine will get into a "not quite hung" but unusable an= d >> useless state. >> >> Am I off-base with my suspicion that this is kernel mode memory? Can >> someone teach me how to diagnose the status of kernel mode memory heap? >> >> Thx, >> Scott >> >> --000000000000236b100600f7d43f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Ok, I'm an idiot.=C2=A0 I'm writing to confess and= to maybe save someone else in the future.=C2=A0 The issue was I mounted a = tmpfs on /tmp and didn't specify an upper size limit.=C2=A0 Invariably = over time, /tmp would begin to fill up and my swap space would start to be = used.=C2=A0 Of course, I couldn't find any usermode process that was us= ing the swap and I jumped to the conclusion that this had something to do w= ith kernel memory.=C2=A0 But really it was my own stupidity.

=
Thank you to Pete and others who tried to help.

Scott

On Wed, Jul 19, 2023 at 4:15=E2=80=AFPM Scott Gasch <scott.gasch@gmail.com> wrote:<= br>
Replying to my own post with more info... I tried stopping my wireguard ja= il and unloading the if_wg kmod and it did not affect the swap memory usage= .=C2=A0 Not sure if that lets wireguard off the hook or not though.
If someone who understands kernel memory could chime in... it l= ooks to me like the aggregate swap usage of usermode processes is nowhere n= ear the total swap space used so I suspect something in kernel mode.=C2=A0 = Does this make sense or is there another explanation?

<= div>Thx,
Scott


On Wed, Jul 19, 2023 at 7:49= =E2=80=AFAM Scott Gasch <scott.gasch@gmail.com> wrote:
I am running a 13.2-RELEA= SE GENERIC kernel and seeing a pattern where, after about 10 days of uptime= , my swap begins to fill up.

# swapinfo -h<= /font>
Device =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0Size =C2=A0 =C2=A0 Used =C2=A0 =C2=A0Avail Capacity
/de= v/ada0p3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A048G =C2=A0 =C2=A0 3.6G =C2=A0 = =C2=A0 =C2=A044G =C2=A0 =C2=A0 7%
/dev/ada1p3 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A048G =C2=A0 =C2=A0 3.6G =C2=A0 =C2=A0 =C2=A044G =C2=A0 =C2=A0 7%/dev/ada2p3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A048G =C2=A0 =C2=A0 3.6G =C2= =A0 =C2=A0 =C2=A044G =C2=A0 =C2=A0 7%
Total =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 144G =C2=A0 =C2=A0 =C2=A011G =C2=A0 =C2=A0 133G =C2=A0= =C2=A0 7%

So, 11G of total swap space.=C2= =A0 What's using it?

= # systat -swap
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /0 =C2=A0 /1 =C2=A0 /2= =C2=A0 /3 =C2=A0 /4 =C2=A0 /5 =C2=A0 /6 =C2=A0 /7 =C2=A0 /8 =C2=A0 /9 =C2= =A0 /10
=C2=A0 =C2=A0 =C2=A0Load Average =C2=A0 ||||||

Device/Pat= h =C2=A0 =C2=A0 =C2=A0 Size =C2=A0Used |0% =C2=A0/10 =C2=A0/20 =C2=A0/30 = =C2=A0/40 =C2=A0/ 60\ =C2=A070\ =C2=A080\ =C2=A090\ 100|
ada0p3 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 48G 3660M XXX
ada1p3 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 48G 3666M XXX
ada2p3 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 48G 3664M XXX
Total =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 144G =C2=A0 11G XXX

Pid =C2=A0 =C2=A0Username =C2=A0 Comm= and =C2=A0 =C2=A0 Swap/Total Per-Process =C2=A0 =C2=A0Per-System
=C2=A01= 4703 scott =C2=A0 =C2=A0 =C2=A0python3.8 =C2=A0 =C2=A04M / 154M =C2=A02% = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%
=C2=A0 2451 scott =C2= =A0 =C2=A0 =C2=A0rclone =C2=A0 =C2=A0 =C2=A0 4M / 934M =C2=A00% =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%
=C2=A0 2452 scott =C2=A0 =C2=A0= =C2=A0rclone =C2=A0 =C2=A0 =C2=A0 3M / =C2=A0 1G =C2=A00% =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%
=C2=A073827 scott =C2=A0 =C2=A0 =C2= =A0bash =C2=A0 =C2=A0 =C2=A0 =C2=A0 1M / =C2=A017M =C2=A06% =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%
=C2=A039416 scott =C2=A0 =C2=A0 =C2= =A0tmux =C2=A0 =C2=A0 =C2=A0 968K / =C2=A054M =C2=A01% =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A00%
=C2=A041661 scott =C2=A0 =C2=A0 =C2=A0bas= h =C2=A0 =C2=A0 =C2=A0 828K / =C2=A017M =C2=A04% =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A00%
=C2=A015727 scott =C2=A0 =C2=A0 =C2=A0bash = =C2=A0 =C2=A0 =C2=A0 808K / =C2=A017M =C2=A04% =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A00%
=C2=A039420 scott =C2=A0 =C2=A0 =C2=A0bash =C2=A0= =C2=A0 =C2=A0 804K / =C2=A017M =C2=A04% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A00%
=C2=A0 2455 scott =C2=A0 =C2=A0 =C2=A0bash =C2=A0 =C2= =A0 =C2=A0 544K / =C2=A015M =C2=A03% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A00%
=C2=A039367 scott =C2=A0 =C2=A0 =C2=A0tmux =C2=A0 =C2=A0 = =C2=A0 512K / =C2=A015M =C2=A03% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A00%
=C2=A0 2447 scott =C2=A0 =C2=A0 =C2=A0bash =C2=A0 =C2=A0 =C2=A0= 376K / =C2=A015M =C2=A02% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= 0%
=C2=A0 2450 scott =C2=A0 =C2=A0 =C2=A0bash =C2=A0 =C2=A0 =C2=A0 364K = / =C2=A015M =C2=A02% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%
= =C2=A0 2453 scott =C2=A0 =C2=A0 =C2=A0bash =C2=A0 =C2=A0 =C2=A0 324K / =C2= =A015M =C2=A02% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%
=C2= =A0 2454 scott =C2=A0 =C2=A0 =C2=A0bash =C2=A0 =C2=A0 =C2=A0 316K / =C2=A01= 5M =C2=A02% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%
=C2=A0 24= 45 scott =C2=A0 =C2=A0 =C2=A0bash =C2=A0 =C2=A0 =C2=A0 312K / =C2=A015M =C2= =A02% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%
=C2=A044937 sco= tt =C2=A0 =C2=A0 =C2=A0bash =C2=A0 =C2=A0 =C2=A0 304K / =C2=A017M =C2=A01% = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%
=C2=A0 2458 scott =C2= =A0 =C2=A0 =C2=A0bash =C2=A0 =C2=A0 =C2=A0 =C2=A072K / =C2=A015M =C2=A00% = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%
At least they agree about it being 11G.=C2=A0 Is this kernel me= mory being paged out to swap?=C2=A0 The machine has 128G of physical memory= and isn't under very heavy load at the moment.=C2=A0=C2=A0
<= br>
I suspect this is a bug in some kernel module... possibly wir= eguard=C2=A0because I run wireguard in a vnet jail and didn't observe t= his problem until setting that up.=C2=A0 But I don't have any hard evid= ence.

I've tried to mitigate this via swapoff -a.=C2=A0 T= his works once but the next
day swap will be back, even fuller.=C2=A0 I&= #39;ve been doing regular reboots to fix this but would like to get to the = bottom of it.=C2=A0 If left alone, swap will
fill up and the machine wil= l get into a "not quite hung" but unusable and useless state.
=
Am I off-base with my suspicion that this is kernel mode memory? Can so= meone teach me how to diagnose the status of kernel mode memory heap?
Thx,
Scott

--000000000000236b100600f7d43f--