Date: Wed, 19 Jul 2023 16:15:05 -0700 From: Scott Gasch <scott.gasch@gmail.com> To: freebsd-questions <freebsd-questions@freebsd.org> Subject: Re: Swap filling up, usermode process swap usage doesn't explain Message-ID: <CABYAQkT%2BaFUsKvk0yQjD1dO9HuaYSRAVs_LNBscnUSJ_eU4vmg@mail.gmail.com> In-Reply-To: <CABYAQkQftAfRXpdSJnqH2Hi=uD-dOiGWdFU8u1XqfeZNBUA35w@mail.gmail.com> References: <CABYAQkQftAfRXpdSJnqH2Hi=uD-dOiGWdFU8u1XqfeZNBUA35w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000003cc79d0600df35a4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Replying to my own post with more info... I tried stopping my wireguard jail and unloading the if_wg kmod and it did not affect the swap memory usage. Not sure if that lets wireguard off the hook or not though. If someone who understands kernel memory could chime in... it looks to me like the aggregate swap usage of usermode processes is nowhere near the total swap space used so I suspect something in kernel mode. Does this make sense or is there another explanation? Thx, Scott On Wed, Jul 19, 2023 at 7:49=E2=80=AFAM Scott Gasch <scott.gasch@gmail.com>= wrote: > I am running a 13.2-RELEASE GENERIC kernel and seeing a pattern where, > after about 10 days of uptime, my swap begins to fill up. > > # swapinfo -h > Device Size Used Avail Capacity > /dev/ada0p3 48G 3.6G 44G 7% > /dev/ada1p3 48G 3.6G 44G 7% > /dev/ada2p3 48G 3.6G 44G 7% > Total 144G 11G 133G 7% > > So, 11G of total swap space. What's using it? > > # systat -swap > /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10 > Load Average |||||| > > Device/Path Size Used |0% /10 /20 /30 /40 / 60\ 70\ 80\ 90= \ > 100| > ada0p3 48G 3660M XXX > ada1p3 48G 3666M XXX > ada2p3 48G 3664M XXX > Total 144G 11G XXX > > Pid Username Command Swap/Total Per-Process Per-System > 14703 scott python3.8 4M / 154M 2% 0% > 2451 scott rclone 4M / 934M 0% 0% > 2452 scott rclone 3M / 1G 0% 0% > 73827 scott bash 1M / 17M 6% 0% > 39416 scott tmux 968K / 54M 1% 0% > 41661 scott bash 828K / 17M 4% 0% > 15727 scott bash 808K / 17M 4% 0% > 39420 scott bash 804K / 17M 4% 0% > 2455 scott bash 544K / 15M 3% 0% > 39367 scott tmux 512K / 15M 3% 0% > 2447 scott bash 376K / 15M 2% 0% > 2450 scott bash 364K / 15M 2% 0% > 2453 scott bash 324K / 15M 2% 0% > 2454 scott bash 316K / 15M 2% 0% > 2445 scott bash 312K / 15M 2% 0% > 44937 scott bash 304K / 17M 1% 0% > 2458 scott bash 72K / 15M 0% 0% > > At least they agree about it being 11G. Is this kernel memory being page= d > out to swap? The machine has 128G of physical memory and isn't under ver= y > heavy load at the moment. > > I suspect this is a bug in some kernel module... possibly > wireguard because I run wireguard in a vnet jail and didn't observe this > problem until setting that up. But I don't have any hard evidence. > > I've tried to mitigate this via swapoff -a. This works once but the next > day swap will be back, even fuller. I've been doing regular reboots to > fix this but would like to get to the bottom of it. If left alone, swap > will > fill up and the machine will get into a "not quite hung" but unusable and > useless state. > > Am I off-base with my suspicion that this is kernel mode memory? Can > someone teach me how to diagnose the status of kernel mode memory heap? > > Thx, > Scott > > --0000000000003cc79d0600df35a4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: base64 PGRpdiBkaXI9Imx0ciI+UmVwbHlpbmcgdG8gbXkgb3duIHBvc3Qgd2l0aCBtb3JlIGluZm8uLi4g SSB0cmllZCBzdG9wcGluZyBteSB3aXJlZ3VhcmQgamFpbCBhbmQgdW5sb2FkaW5nIHRoZSBpZl93 ZyBrbW9kIGFuZCBpdCBkaWQgbm90IGFmZmVjdCB0aGUgc3dhcCBtZW1vcnkgdXNhZ2UuwqAgTm90 IHN1cmUgaWYgdGhhdCBsZXRzIHdpcmVndWFyZCBvZmYgdGhlIGhvb2sgb3Igbm90IHRob3VnaC48 ZGl2Pjxicj48L2Rpdj48ZGl2PklmIHNvbWVvbmUgd2hvIHVuZGVyc3RhbmRzIGtlcm5lbCBtZW1v cnkgY291bGQgY2hpbWUgaW4uLi4gaXQgbG9va3MgdG8gbWUgbGlrZSB0aGUgYWdncmVnYXRlIHN3 YXAgdXNhZ2Ugb2YgdXNlcm1vZGUgcHJvY2Vzc2VzIGlzIG5vd2hlcmUgbmVhciB0aGUgdG90YWwg c3dhcCBzcGFjZSB1c2VkIHNvIEkgc3VzcGVjdCBzb21ldGhpbmcgaW4ga2VybmVsIG1vZGUuwqAg RG9lcyB0aGlzIG1ha2Ugc2Vuc2Ugb3IgaXMgdGhlcmUgYW5vdGhlciBleHBsYW5hdGlvbj88L2Rp dj48ZGl2Pjxicj48L2Rpdj48ZGl2PlRoeCw8L2Rpdj48ZGl2PlNjb3R0PC9kaXY+PGRpdj48YnI+ PC9kaXY+PC9kaXY+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj48ZGl2IGRpcj0ibHRyIiBj bGFzcz0iZ21haWxfYXR0ciI+T24gV2VkLCBKdWwgMTksIDIwMjMgYXQgNzo0OeKAr0FNIFNjb3R0 IEdhc2NoICZsdDs8YSBocmVmPSJtYWlsdG86c2NvdHQuZ2FzY2hAZ21haWwuY29tIj5zY290dC5n YXNjaEBnbWFpbC5jb208L2E+Jmd0OyB3cm90ZTo8YnI+PC9kaXY+PGJsb2NrcXVvdGUgY2xhc3M9 ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0 OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiPjxkaXYgZGlyPSJs dHIiPkkgYW0gcnVubmluZyBhIDEzLjItUkVMRUFTRSBHRU5FUklDIGtlcm5lbCBhbmQgc2VlaW5n IGEgcGF0dGVybiB3aGVyZSwgYWZ0ZXIgYWJvdXQgMTAgZGF5cyBvZiB1cHRpbWUsIG15IHN3YXAg YmVnaW5zIHRvIGZpbGwgdXAuPGJyPjxicj48Zm9udCBmYWNlPSJtb25vc3BhY2UiPiMgc3dhcGlu Zm8gLWg8L2ZvbnQ+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UiPkRldmljZSDCoCDCoCDCoCDC oCDCoCDCoCDCoFNpemUgwqAgwqAgVXNlZCDCoCDCoEF2YWlsIENhcGFjaXR5PGJyPi9kZXYvYWRh MHAzIMKgIMKgIMKgIMKgIMKgNDhHIMKgIMKgIDMuNkcgwqAgwqAgwqA0NEcgwqAgwqAgNyU8YnI+ L2Rldi9hZGExcDMgwqAgwqAgwqAgwqAgwqA0OEcgwqAgwqAgMy42RyDCoCDCoCDCoDQ0RyDCoCDC oCA3JTxicj4vZGV2L2FkYTJwMyDCoCDCoCDCoCDCoCDCoDQ4RyDCoCDCoCAzLjZHIMKgIMKgIMKg NDRHIMKgIMKgIDclPGJyPlRvdGFsIMKgIMKgIMKgIMKgIMKgIMKgIMKgIDE0NEcgwqAgwqAgwqAx MUcgwqAgwqAgMTMzRyDCoCDCoCA3JTwvZm9udD48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PlNv LCAxMUcgb2YgdG90YWwgc3dhcCBzcGFjZS7CoCBXaGF0JiMzOTtzIHVzaW5nIGl0PzwvZGl2Pjxk aXY+PGJyPjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlIj4jIHN5c3RhdCAtc3dhcDwv Zm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSI+wqAgwqAgwqAgwqAgwqAgwqAg wqAgwqAgwqAgwqAgLzAgwqAgLzEgwqAgLzIgwqAgLzMgwqAgLzQgwqAgLzUgwqAgLzYgwqAgLzcg wqAgLzggwqAgLzkgwqAgLzEwPGJyPsKgIMKgIMKgTG9hZCBBdmVyYWdlIMKgIHx8fHx8fDxicj48 YnI+RGV2aWNlL1BhdGggwqAgwqAgwqAgU2l6ZSDCoFVzZWQgfDAlIMKgLzEwIMKgLzIwIMKgLzMw IMKgLzQwIMKgLyA2MFwgwqA3MFwgwqA4MFwgwqA5MFwgMTAwfDxicj5hZGEwcDMgwqAgwqAgwqAg wqAgwqAgwqAgNDhHIDM2NjBNIFhYWDxicj5hZGExcDMgwqAgwqAgwqAgwqAgwqAgwqAgNDhHIDM2 NjZNIFhYWDxicj5hZGEycDMgwqAgwqAgwqAgwqAgwqAgwqAgNDhHIDM2NjRNIFhYWDxicj5Ub3Rh bCDCoCDCoCDCoCDCoCDCoCDCoCAxNDRHIMKgIDExRyBYWFg8YnI+PGJyPlBpZCDCoCDCoFVzZXJu YW1lIMKgIENvbW1hbmQgwqAgwqAgU3dhcC9Ub3RhbCBQZXItUHJvY2VzcyDCoCDCoFBlci1TeXN0 ZW08YnI+wqAxNDcwMyBzY290dCDCoCDCoCDCoHB5dGhvbjMuOCDCoCDCoDRNIC8gMTU0TSDCoDIl IMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8YnI+wqAgMjQ1MSBzY290dCDCoCDCoCDCoHJjbG9uZSDC oCDCoCDCoCA0TSAvIDkzNE0gwqAwJSDCoCDCoCDCoCDCoCDCoCDCoCDCoDAlPGJyPsKgIDI0NTIg c2NvdHQgwqAgwqAgwqByY2xvbmUgwqAgwqAgwqAgM00gLyDCoCAxRyDCoDAlIMKgIMKgIMKgIMKg IMKgIMKgIMKgMCU8YnI+wqA3MzgyNyBzY290dCDCoCDCoCDCoGJhc2ggwqAgwqAgwqAgwqAgMU0g LyDCoDE3TSDCoDYlIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8YnI+wqAzOTQxNiBzY290dCDCoCDC oCDCoHRtdXggwqAgwqAgwqAgOTY4SyAvIMKgNTRNIMKgMSUgwqAgwqAgwqAgwqAgwqAgwqAgwqAw JTxicj7CoDQxNjYxIHNjb3R0IMKgIMKgIMKgYmFzaCDCoCDCoCDCoCA4MjhLIC8gwqAxN00gwqA0 JSDCoCDCoCDCoCDCoCDCoCDCoCDCoDAlPGJyPsKgMTU3Mjcgc2NvdHQgwqAgwqAgwqBiYXNoIMKg IMKgIMKgIDgwOEsgLyDCoDE3TSDCoDQlIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8YnI+wqAzOTQy MCBzY290dCDCoCDCoCDCoGJhc2ggwqAgwqAgwqAgODA0SyAvIMKgMTdNIMKgNCUgwqAgwqAgwqAg wqAgwqAgwqAgwqAwJTxicj7CoCAyNDU1IHNjb3R0IMKgIMKgIMKgYmFzaCDCoCDCoCDCoCA1NDRL IC8gwqAxNU0gwqAzJSDCoCDCoCDCoCDCoCDCoCDCoCDCoDAlPGJyPsKgMzkzNjcgc2NvdHQgwqAg wqAgwqB0bXV4IMKgIMKgIMKgIDUxMksgLyDCoDE1TSDCoDMlIMKgIMKgIMKgIMKgIMKgIMKgIMKg MCU8YnI+wqAgMjQ0NyBzY290dCDCoCDCoCDCoGJhc2ggwqAgwqAgwqAgMzc2SyAvIMKgMTVNIMKg MiUgwqAgwqAgwqAgwqAgwqAgwqAgwqAwJTxicj7CoCAyNDUwIHNjb3R0IMKgIMKgIMKgYmFzaCDC oCDCoCDCoCAzNjRLIC8gwqAxNU0gwqAyJSDCoCDCoCDCoCDCoCDCoCDCoCDCoDAlPGJyPsKgIDI0 NTMgc2NvdHQgwqAgwqAgwqBiYXNoIMKgIMKgIMKgIDMyNEsgLyDCoDE1TSDCoDIlIMKgIMKgIMKg IMKgIMKgIMKgIMKgMCU8YnI+wqAgMjQ1NCBzY290dCDCoCDCoCDCoGJhc2ggwqAgwqAgwqAgMzE2 SyAvIMKgMTVNIMKgMiUgwqAgwqAgwqAgwqAgwqAgwqAgwqAwJTxicj7CoCAyNDQ1IHNjb3R0IMKg IMKgIMKgYmFzaCDCoCDCoCDCoCAzMTJLIC8gwqAxNU0gwqAyJSDCoCDCoCDCoCDCoCDCoCDCoCDC oDAlPGJyPsKgNDQ5Mzcgc2NvdHQgwqAgwqAgwqBiYXNoIMKgIMKgIMKgIDMwNEsgLyDCoDE3TSDC oDElIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8YnI+wqAgMjQ1OCBzY290dCDCoCDCoCDCoGJhc2gg wqAgwqAgwqAgwqA3MksgLyDCoDE1TSDCoDAlIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8YnI+PC9m b250PjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+QXQgbGVhc3QgdGhleSBhZ3JlZSBhYm91dCBp dCBiZWluZyAxMUcuwqAgSXMgdGhpcyBrZXJuZWwgbWVtb3J5IGJlaW5nIHBhZ2VkIG91dCB0byBz d2FwP8KgIFRoZSBtYWNoaW5lIGhhcyAxMjhHIG9mIHBoeXNpY2FsIG1lbW9yeSBhbmQgaXNuJiMz OTt0IHVuZGVyIHZlcnkgaGVhdnkgbG9hZCBhdCB0aGUgbW9tZW50LsKgwqA8L2Rpdj48ZGl2Pjxi cj48L2Rpdj48ZGl2Pkkgc3VzcGVjdCB0aGlzIGlzIGEgYnVnIGluIHNvbWUga2VybmVsIG1vZHVs ZS4uLiBwb3NzaWJseSB3aXJlZ3VhcmTCoGJlY2F1c2UgSSBydW4gd2lyZWd1YXJkIGluIGEgdm5l dCBqYWlsIGFuZCBkaWRuJiMzOTt0IG9ic2VydmUgdGhpcyBwcm9ibGVtIHVudGlsIHNldHRpbmcg dGhhdCB1cC7CoCBCdXQgSSBkb24mIzM5O3QgaGF2ZSBhbnkgaGFyZCBldmlkZW5jZS48L2Rpdj48 ZGl2Pjxicj5JJiMzOTt2ZSB0cmllZCB0byBtaXRpZ2F0ZSB0aGlzIHZpYSBzd2Fwb2ZmIC1hLsKg IFRoaXMgd29ya3Mgb25jZSBidXQgdGhlIG5leHQ8YnI+ZGF5IHN3YXAgd2lsbCBiZSBiYWNrLCBl dmVuIGZ1bGxlci7CoCBJJiMzOTt2ZSBiZWVuIGRvaW5nIHJlZ3VsYXIgcmVib290cyB0byBmaXgg dGhpcyBidXQgd291bGQgbGlrZSB0byBnZXQgdG8gdGhlIGJvdHRvbSBvZiBpdC7CoCBJZiBsZWZ0 IGFsb25lLCBzd2FwIHdpbGw8YnI+ZmlsbCB1cCBhbmQgdGhlIG1hY2hpbmUgd2lsbCBnZXQgaW50 byBhICZxdW90O25vdCBxdWl0ZSBodW5nJnF1b3Q7IGJ1dCB1bnVzYWJsZSBhbmQgdXNlbGVzcyBz dGF0ZS48YnI+PGJyPkFtIEkgb2ZmLWJhc2Ugd2l0aCBteSBzdXNwaWNpb24gdGhhdCB0aGlzIGlz IGtlcm5lbCBtb2RlIG1lbW9yeT8gQ2FuIHNvbWVvbmUgdGVhY2ggbWUgaG93IHRvIGRpYWdub3Nl IHRoZSBzdGF0dXMgb2Yga2VybmVsIG1vZGUgbWVtb3J5IGhlYXA/PGJyPjxicj5UaHgsPGJyPlNj b3R0PC9kaXY+PGRpdj48YnI+PC9kaXY+PC9kaXY+DQo8L2Jsb2NrcXVvdGU+PC9kaXY+DQo= --0000000000003cc79d0600df35a4--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABYAQkT%2BaFUsKvk0yQjD1dO9HuaYSRAVs_LNBscnUSJ_eU4vmg>