Date: Wed, 19 Jul 2023 07:49:23 -0700 From: Scott Gasch <scott.gasch@gmail.com> To: freebsd-questions <freebsd-questions@freebsd.org>, freebsd-hackers@freebsd.org Subject: Swap filling up, usermode process swap usage doesn't explain Message-ID: <CABYAQkQftAfRXpdSJnqH2Hi=uD-dOiGWdFU8u1XqfeZNBUA35w@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
--000000000000b884220600d824cd
Content-Type: text/plain; charset="UTF-8"
I am running a 13.2-RELEASE GENERIC kernel and seeing a pattern where,
after about 10 days of uptime, my swap begins to fill up.
# swapinfo -h
Device Size Used Avail Capacity
/dev/ada0p3 48G 3.6G 44G 7%
/dev/ada1p3 48G 3.6G 44G 7%
/dev/ada2p3 48G 3.6G 44G 7%
Total 144G 11G 133G 7%
So, 11G of total swap space. What's using it?
# systat -swap
/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
Load Average ||||||
Device/Path Size Used |0% /10 /20 /30 /40 / 60\ 70\ 80\ 90\
100|
ada0p3 48G 3660M XXX
ada1p3 48G 3666M XXX
ada2p3 48G 3664M XXX
Total 144G 11G XXX
Pid Username Command Swap/Total Per-Process Per-System
14703 scott python3.8 4M / 154M 2% 0%
2451 scott rclone 4M / 934M 0% 0%
2452 scott rclone 3M / 1G 0% 0%
73827 scott bash 1M / 17M 6% 0%
39416 scott tmux 968K / 54M 1% 0%
41661 scott bash 828K / 17M 4% 0%
15727 scott bash 808K / 17M 4% 0%
39420 scott bash 804K / 17M 4% 0%
2455 scott bash 544K / 15M 3% 0%
39367 scott tmux 512K / 15M 3% 0%
2447 scott bash 376K / 15M 2% 0%
2450 scott bash 364K / 15M 2% 0%
2453 scott bash 324K / 15M 2% 0%
2454 scott bash 316K / 15M 2% 0%
2445 scott bash 312K / 15M 2% 0%
44937 scott bash 304K / 17M 1% 0%
2458 scott bash 72K / 15M 0% 0%
At least they agree about it being 11G. Is this kernel memory being paged
out to swap? The machine has 128G of physical memory and isn't under very
heavy load at the moment.
I suspect this is a bug in some kernel module... possibly wireguard because
I run wireguard in a vnet jail and didn't observe this problem until
setting that up. But I don't have any hard evidence.
I've tried to mitigate this via swapoff -a. This works once but the next
day swap will be back, even fuller. I've been doing regular reboots to fix
this but would like to get to the bottom of it. If left alone, swap will
fill up and the machine will get into a "not quite hung" but unusable and
useless state.
Am I off-base with my suspicion that this is kernel mode memory? Can
someone teach me how to diagnose the status of kernel mode memory heap?
Thx,
Scott
--000000000000b884220600d824cd
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: base64
PGRpdiBkaXI9Imx0ciI+SSBhbSBydW5uaW5nIGEgMTMuMi1SRUxFQVNFIEdFTkVSSUMga2VybmVs
IGFuZCBzZWVpbmcgYSBwYXR0ZXJuIHdoZXJlLCBhZnRlciBhYm91dCAxMCBkYXlzIG9mIHVwdGlt
ZSwgbXkgc3dhcCBiZWdpbnMgdG8gZmlsbCB1cC48YnI+PGJyPjxmb250IGZhY2U9Im1vbm9zcGFj
ZSI+IyBzd2FwaW5mbyAtaDwvZm9udD48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSI+RGV2aWNl
IMKgIMKgIMKgIMKgIMKgIMKgIMKgU2l6ZSDCoCDCoCBVc2VkIMKgIMKgQXZhaWwgQ2FwYWNpdHk8
YnI+L2Rldi9hZGEwcDMgwqAgwqAgwqAgwqAgwqA0OEcgwqAgwqAgMy42RyDCoCDCoCDCoDQ0RyDC
oCDCoCA3JTxicj4vZGV2L2FkYTFwMyDCoCDCoCDCoCDCoCDCoDQ4RyDCoCDCoCAzLjZHIMKgIMKg
IMKgNDRHIMKgIMKgIDclPGJyPi9kZXYvYWRhMnAzIMKgIMKgIMKgIMKgIMKgNDhHIMKgIMKgIDMu
NkcgwqAgwqAgwqA0NEcgwqAgwqAgNyU8YnI+VG90YWwgwqAgwqAgwqAgwqAgwqAgwqAgwqAgMTQ0
RyDCoCDCoCDCoDExRyDCoCDCoCAxMzNHIMKgIMKgIDclPC9mb250PjwvZGl2PjxkaXY+PGJyPjwv
ZGl2PjxkaXY+U28sIDExRyBvZiB0b3RhbCBzd2FwIHNwYWNlLsKgIFdoYXQmIzM5O3MgdXNpbmcg
aXQ/PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UiPiMgc3lz
dGF0IC1zd2FwPC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlIj7CoCDCoCDC
oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCAvMCDCoCAvMSDCoCAvMiDCoCAvMyDCoCAvNCDCoCAvNSDC
oCAvNiDCoCAvNyDCoCAvOCDCoCAvOSDCoCAvMTA8YnI+wqAgwqAgwqBMb2FkIEF2ZXJhZ2UgwqAg
fHx8fHx8PGJyPjxicj5EZXZpY2UvUGF0aCDCoCDCoCDCoCBTaXplIMKgVXNlZCB8MCUgwqAvMTAg
wqAvMjAgwqAvMzAgwqAvNDAgwqAvIDYwXCDCoDcwXCDCoDgwXCDCoDkwXCAxMDB8PGJyPmFkYTBw
MyDCoCDCoCDCoCDCoCDCoCDCoCA0OEcgMzY2ME0gWFhYPGJyPmFkYTFwMyDCoCDCoCDCoCDCoCDC
oCDCoCA0OEcgMzY2Nk0gWFhYPGJyPmFkYTJwMyDCoCDCoCDCoCDCoCDCoCDCoCA0OEcgMzY2NE0g
WFhYPGJyPlRvdGFsIMKgIMKgIMKgIMKgIMKgIMKgIDE0NEcgwqAgMTFHIFhYWDxicj48YnI+UGlk
IMKgIMKgVXNlcm5hbWUgwqAgQ29tbWFuZCDCoCDCoCBTd2FwL1RvdGFsIFBlci1Qcm9jZXNzIMKg
IMKgUGVyLVN5c3RlbTxicj7CoDE0NzAzIHNjb3R0IMKgIMKgIMKgcHl0aG9uMy44IMKgIMKgNE0g
LyAxNTRNIMKgMiUgwqAgwqAgwqAgwqAgwqAgwqAgwqAwJTxicj7CoCAyNDUxIHNjb3R0IMKgIMKg
IMKgcmNsb25lIMKgIMKgIMKgIDRNIC8gOTM0TSDCoDAlIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8
YnI+wqAgMjQ1MiBzY290dCDCoCDCoCDCoHJjbG9uZSDCoCDCoCDCoCAzTSAvIMKgIDFHIMKgMCUg
wqAgwqAgwqAgwqAgwqAgwqAgwqAwJTxicj7CoDczODI3IHNjb3R0IMKgIMKgIMKgYmFzaCDCoCDC
oCDCoCDCoCAxTSAvIMKgMTdNIMKgNiUgwqAgwqAgwqAgwqAgwqAgwqAgwqAwJTxicj7CoDM5NDE2
IHNjb3R0IMKgIMKgIMKgdG11eCDCoCDCoCDCoCA5NjhLIC8gwqA1NE0gwqAxJSDCoCDCoCDCoCDC
oCDCoCDCoCDCoDAlPGJyPsKgNDE2NjEgc2NvdHQgwqAgwqAgwqBiYXNoIMKgIMKgIMKgIDgyOEsg
LyDCoDE3TSDCoDQlIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8YnI+wqAxNTcyNyBzY290dCDCoCDC
oCDCoGJhc2ggwqAgwqAgwqAgODA4SyAvIMKgMTdNIMKgNCUgwqAgwqAgwqAgwqAgwqAgwqAgwqAw
JTxicj7CoDM5NDIwIHNjb3R0IMKgIMKgIMKgYmFzaCDCoCDCoCDCoCA4MDRLIC8gwqAxN00gwqA0
JSDCoCDCoCDCoCDCoCDCoCDCoCDCoDAlPGJyPsKgIDI0NTUgc2NvdHQgwqAgwqAgwqBiYXNoIMKg
IMKgIMKgIDU0NEsgLyDCoDE1TSDCoDMlIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCU8YnI+wqAzOTM2
NyBzY290dCDCoCDCoCDCoHRtdXggwqAgwqAgwqAgNTEySyAvIMKgMTVNIMKgMyUgwqAgwqAgwqAg
wqAgwqAgwqAgwqAwJTxicj7CoCAyNDQ3IHNjb3R0IMKgIMKgIMKgYmFzaCDCoCDCoCDCoCAzNzZL
IC8gwqAxNU0gwqAyJSDCoCDCoCDCoCDCoCDCoCDCoCDCoDAlPGJyPsKgIDI0NTAgc2NvdHQgwqAg
wqAgwqBiYXNoIMKgIMKgIMKgIDM2NEsgLyDCoDE1TSDCoDIlIMKgIMKgIMKgIMKgIMKgIMKgIMKg
MCU8YnI+wqAgMjQ1MyBzY290dCDCoCDCoCDCoGJhc2ggwqAgwqAgwqAgMzI0SyAvIMKgMTVNIMKg
MiUgwqAgwqAgwqAgwqAgwqAgwqAgwqAwJTxicj7CoCAyNDU0IHNjb3R0IMKgIMKgIMKgYmFzaCDC
oCDCoCDCoCAzMTZLIC8gwqAxNU0gwqAyJSDCoCDCoCDCoCDCoCDCoCDCoCDCoDAlPGJyPsKgIDI0
NDUgc2NvdHQgwqAgwqAgwqBiYXNoIMKgIMKgIMKgIDMxMksgLyDCoDE1TSDCoDIlIMKgIMKgIMKg
IMKgIMKgIMKgIMKgMCU8YnI+wqA0NDkzNyBzY290dCDCoCDCoCDCoGJhc2ggwqAgwqAgwqAgMzA0
SyAvIMKgMTdNIMKgMSUgwqAgwqAgwqAgwqAgwqAgwqAgwqAwJTxicj7CoCAyNDU4IHNjb3R0IMKg
IMKgIMKgYmFzaCDCoCDCoCDCoCDCoDcySyAvIMKgMTVNIMKgMCUgwqAgwqAgwqAgwqAgwqAgwqAg
wqAwJTxicj48L2ZvbnQ+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5BdCBsZWFzdCB0aGV5IGFn
cmVlIGFib3V0IGl0IGJlaW5nIDExRy7CoCBJcyB0aGlzIGtlcm5lbCBtZW1vcnkgYmVpbmcgcGFn
ZWQgb3V0IHRvIHN3YXA/wqAgVGhlIG1hY2hpbmUgaGFzIDEyOEcgb2YgcGh5c2ljYWwgbWVtb3J5
IGFuZCBpc24mIzM5O3QgdW5kZXIgdmVyeSBoZWF2eSBsb2FkIGF0IHRoZSBtb21lbnQuwqDCoDwv
ZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+SSBzdXNwZWN0IHRoaXMgaXMgYSBidWcgaW4gc29tZSBr
ZXJuZWwgbW9kdWxlLi4uIHBvc3NpYmx5IHdpcmVndWFyZMKgYmVjYXVzZSBJIHJ1biB3aXJlZ3Vh
cmQgaW4gYSB2bmV0IGphaWwgYW5kIGRpZG4mIzM5O3Qgb2JzZXJ2ZSB0aGlzIHByb2JsZW0gdW50
aWwgc2V0dGluZyB0aGF0IHVwLsKgIEJ1dCBJIGRvbiYjMzk7dCBoYXZlIGFueSBoYXJkIGV2aWRl
bmNlLjwvZGl2PjxkaXY+PGJyPkkmIzM5O3ZlIHRyaWVkIHRvIG1pdGlnYXRlIHRoaXMgdmlhIHN3
YXBvZmYgLWEuwqAgVGhpcyB3b3JrcyBvbmNlIGJ1dCB0aGUgbmV4dDxicj5kYXkgc3dhcCB3aWxs
IGJlIGJhY2ssIGV2ZW4gZnVsbGVyLsKgIEkmIzM5O3ZlIGJlZW4gZG9pbmcgcmVndWxhciByZWJv
b3RzIHRvIGZpeCB0aGlzIGJ1dCB3b3VsZCBsaWtlIHRvIGdldCB0byB0aGUgYm90dG9tIG9mIGl0
LsKgIElmIGxlZnQgYWxvbmUsIHN3YXAgd2lsbDxicj5maWxsIHVwIGFuZCB0aGUgbWFjaGluZSB3
aWxsIGdldCBpbnRvIGEgJnF1b3Q7bm90IHF1aXRlIGh1bmcmcXVvdDsgYnV0IHVudXNhYmxlIGFu
ZCB1c2VsZXNzIHN0YXRlLjxicj48YnI+QW0gSSBvZmYtYmFzZSB3aXRoIG15IHN1c3BpY2lvbiB0
aGF0IHRoaXMgaXMga2VybmVsIG1vZGUgbWVtb3J5PyBDYW4gc29tZW9uZSB0ZWFjaCBtZSBob3cg
dG8gZGlhZ25vc2UgdGhlIHN0YXR1cyBvZiBrZXJuZWwgbW9kZSBtZW1vcnkgaGVhcD88YnI+PGJy
PlRoeCw8YnI+U2NvdHQ8L2Rpdj48ZGl2Pjxicj48L2Rpdj48L2Rpdj4NCg==
--000000000000b884220600d824cd--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABYAQkQftAfRXpdSJnqH2Hi=uD-dOiGWdFU8u1XqfeZNBUA35w>