From nobody Wed Jul 19 22:50:44 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R5rdr3sgTz4nXHC for ; Wed, 19 Jul 2023 22:51:12 +0000 (UTC) (envelope-from scott.gasch@gmail.com) Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R5rdr1d3Nz46B7 for ; Wed, 19 Jul 2023 22:51:12 +0000 (UTC) (envelope-from scott.gasch@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-io1-xd2a.google.com with SMTP id ca18e2360f4ac-78374596182so7725139f.0 for ; Wed, 19 Jul 2023 15:51:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689807071; x=1692399071; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=IK0jkmtbuRzyB++NEwn58JgGDL2KY7F6GbpYLelrsy0=; b=Gaiuz9oQUh8Pb4V07lVfALmcom6JOZ0DOrmB6t7C247s8hflgGfF4lelppoBgo2Yla 0D+fYYkg/TM3Hlmc3h30Ac+I763uRFR7yMP04RoakLYtokgD06GPW9/0Qy8KW5UFusSr QTLPH5NMP5jSk+W9Q4F+drb2UlD0ZNrCsXVKAX+6T83LoDXgCDpkdauLbdgZzcaz8one wTp9OqMO+uUy+LMKHf/+//KYvfb32uy7uTVw7CKryJiF4F57z31xGRoR8tBgTVLH26o2 CsIitQiDFmrNUnKh0IkrAjp3ks1vZgxvf/oAGLQgTUb8G4ljozKrq3NDgftZ5iddg42I 5nfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689807071; x=1692399071; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=IK0jkmtbuRzyB++NEwn58JgGDL2KY7F6GbpYLelrsy0=; b=Bg5HvbTV5FIWUUNrkbgar1v5SpvnnTJvq9d7kkLeXqc/p/r9QQ79h+NfNyNNI57syN pfx6DS9C7J25q/5qtkW9ESmn4u7tf60DIfp5MaVsiuourrjKeu/AhcuuO/I6IF7BrKdG QXempeP7/pMrmJnSWasuAyUR9mxy/uwlsO1T6Cm23k/X4eNGrVPnG7FDVkjX/yST+T2U GEUWa8yOerZci7uwizUO44LKCj0H/TFOXBwYXhkPN66BZiHJ016VT3FucOm2aKrxrcbc 4fo0TTD/o1gc4opPIEshMOTa0V/GbgCKOtibahjAm09MvdHKyACZtQNbe9M8SZum8EZo etwA== X-Gm-Message-State: ABy/qLb6/odwhiOo8fYVWG2Iv8iRM5VcpaFoOKb7ftDtxA7redKZhl8/ okFsBlKfsw5q910PH/a+qybjZaixhTsREijfNsgLjnIM X-Google-Smtp-Source: APBJJlHz2Tt+srhAdvsJPxmZ0fE8pLAc+LbjBi6Cb/R/kX35fWg0I+fEObztUYTL8kthW4vNDwoloP+1AJ/Ihc/+fWM= X-Received: by 2002:a05:6e02:1526:b0:345:aee1:eae2 with SMTP id i6-20020a056e02152600b00345aee1eae2mr7661326ilu.20.1689807070980; Wed, 19 Jul 2023 15:51:10 -0700 (PDT) List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 References: <88e0d702-cb86-b922-a5ce-82fe32e66a40@nomadlogic.org> In-Reply-To: <88e0d702-cb86-b922-a5ce-82fe32e66a40@nomadlogic.org> From: Scott Gasch Date: Wed, 19 Jul 2023 15:50:44 -0700 Message-ID: Subject: Re: Swap filling up, usermode process swap usage doesn't explain To: Pete Wright Cc: freebsd-questions Content-Type: multipart/alternative; boundary="0000000000003471af0600dedee9" X-Rspamd-Queue-Id: 4R5rdr1d3Nz46B7 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated --0000000000003471af0600dedee9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable -cc hackers The laundry is abnormally high and doesn't seem to go down. My understanding of it is that it's pages that are available for reuse and have to be zeroed by some kernel thread that does that job(?). Typically this filling up swap with no clear usermode culprit happens after the machine has been running for ~10 days and I eventually reboot. I can say for sure laundry is nowhere near this full after a reboot for several days. On Wed, Jul 19, 2023 at 3:40=E2=80=AFPM Pete Wright w= rote: > > > On 7/19/23 15:11, Scott Gasch wrote: > > Yes, I'm using ZFS. Here's what top says: > > > > last pid: 88926; load averages: 1.20, 0.96, 0.87 > > up 5+17:48:34 15:09:58 > > 274 processes: 1 running, 272 sleeping, 1 zombie > > CPU: 1.8% user, 0.0% nice, 0.5% system, 0.0% interrupt, 97.8% idle > > Mem: 1844M Active, 7777M Inact, 77G Laundry, 35G Wired, 750M Buf, 3367M > Free > > ARC: 24G Total, 2878M MFU, 18G MRU, 21M Anon, 119M Header, 2622M Other > > 18G Compressed, 25G Uncompressed, 1.33:1 Ratio > > Swap: 144G Total, 11G Used, 133G Free, 7% Inuse > > > > If I leave this alone it will grow to consume all available swap space. > > I'll try your fix with the sysctl knob and see what happens... I hope > > this is it, I've been fighting this for a while now. > > worth a shot, but 24G of ARC isn't that bad, especially if you are doing > quite a bit of disk i/o. > > i'm more interested in the 77G of Laundry memory, that seems like quite > a bit. but i don't know your workload so not sure... > > -p > > -- > Pete Wright > pete@nomadlogic.org > @nomadlogicLA > --0000000000003471af0600dedee9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
-cc hackers

The laundry is abnorm= ally high and doesn't seem to go down.=C2=A0 My understanding of it is = that it's pages that are available for reuse and have to be zeroed by s= ome kernel thread that does that job(?).

Typically this = filling up swap with no clear usermode=C2=A0culprit happens after the machi= ne has been running for ~10 days and I eventually reboot.=C2=A0 I can say f= or sure laundry is nowhere near this full after a reboot for several days.<= /div>




On Wed, Jul 19= , 2023 at 3:40=E2=80=AFPM Pete Wright <pete@nomadlogic.org> wrote:


On 7/19/23 15:11, Scott Gasch wrote:
> Yes, I'm using ZFS.=C2=A0 Here's what top says:
>
> last pid: 88926; =C2=A0load averages: =C2=A01.20, =C2=A00.96, =C2=A00.= 87=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0
>=C2=A0 =C2=A0 =C2=A0up 5+17:48:34 =C2=A015:09:58
> 274 processes: 1 running, 272 sleeping, 1 zombie
> CPU: =C2=A01.8% user, =C2=A00.0% nice, =C2=A00.5% system, =C2=A00.0% i= nterrupt, 97.8% idle
> Mem: 1844M Active, 7777M Inact, 77G Laundry, 35G Wired, 750M Buf, 3367= M Free
> ARC: 24G Total, 2878M MFU, 18G MRU, 21M Anon, 119M Header, 2622M Other=
>=C2=A0 =C2=A0 =C2=A0 =C2=A018G Compressed, 25G Uncompressed, 1.33:1 Rat= io
> Swap: 144G Total, 11G Used, 133G Free, 7% Inuse
>
> If I leave this alone it will grow to consume all available swap space= .=C2=A0
> I'll try your fix with the sysctl knob and see what happens...=C2= =A0 I hope
> this is it, I've been fighting this for a while now.

worth a shot, but 24G of ARC isn't that bad, especially if you are doin= g
quite a bit of disk i/o.

i'm more interested in the 77G of Laundry memory, that seems like quite=
a bit.=C2=A0 but i don't know your workload so not sure...

-p

--
Pete Wright
pete@nomadlogic.or= g
@nomadlogicLA
--0000000000003471af0600dedee9--