Date: Thu, 20 Jul 2023 21:37:29 -0700 From: Scott Gasch <scott.gasch@gmail.com> To: freebsd-questions <freebsd-questions@freebsd.org>, Pete Wright <pete@nomadlogic.org> Subject: Re: Swap filling up, usermode process swap usage doesn't explain Message-ID: <CABYAQkSpZshyfMrbC3ykfyFqZK4KPBPN4a5L8GQME3oZ8KuUnQ@mail.gmail.com> In-Reply-To: <CABYAQkT%2BaFUsKvk0yQjD1dO9HuaYSRAVs_LNBscnUSJ_eU4vmg@mail.gmail.com> References: <CABYAQkQftAfRXpdSJnqH2Hi=uD-dOiGWdFU8u1XqfeZNBUA35w@mail.gmail.com> <CABYAQkT%2BaFUsKvk0yQjD1dO9HuaYSRAVs_LNBscnUSJ_eU4vmg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000236b100600f7d43f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ok, I'm an idiot. I'm writing to confess and to maybe save someone else in the future. The issue was I mounted a tmpfs on /tmp and didn't specify an upper size limit. Invariably over time, /tmp would begin to fill up and my swap space would start to be used. Of course, I couldn't find any usermode process that was using the swap and I jumped to the conclusion that this had something to do with kernel memory. But really it was my own stupidity= . Thank you to Pete and others who tried to help. Scott On Wed, Jul 19, 2023 at 4:15=E2=80=AFPM Scott Gasch <scott.gasch@gmail.com>= wrote: > Replying to my own post with more info... I tried stopping my wireguard > jail and unloading the if_wg kmod and it did not affect the swap memory > usage. Not sure if that lets wireguard off the hook or not though. > > If someone who understands kernel memory could chime in... it looks to me > like the aggregate swap usage of usermode processes is nowhere near the > total swap space used so I suspect something in kernel mode. Does this > make sense or is there another explanation? > > Thx, > Scott > > > On Wed, Jul 19, 2023 at 7:49=E2=80=AFAM Scott Gasch <scott.gasch@gmail.co= m> wrote: > >> I am running a 13.2-RELEASE GENERIC kernel and seeing a pattern where, >> after about 10 days of uptime, my swap begins to fill up. >> >> # swapinfo -h >> Device Size Used Avail Capacity >> /dev/ada0p3 48G 3.6G 44G 7% >> /dev/ada1p3 48G 3.6G 44G 7% >> /dev/ada2p3 48G 3.6G 44G 7% >> Total 144G 11G 133G 7% >> >> So, 11G of total swap space. What's using it? >> >> # systat -swap >> /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /1= 0 >> Load Average |||||| >> >> Device/Path Size Used |0% /10 /20 /30 /40 / 60\ 70\ 80\ >> 90\ 100| >> ada0p3 48G 3660M XXX >> ada1p3 48G 3666M XXX >> ada2p3 48G 3664M XXX >> Total 144G 11G XXX >> >> Pid Username Command Swap/Total Per-Process Per-System >> 14703 scott python3.8 4M / 154M 2% 0% >> 2451 scott rclone 4M / 934M 0% 0% >> 2452 scott rclone 3M / 1G 0% 0% >> 73827 scott bash 1M / 17M 6% 0% >> 39416 scott tmux 968K / 54M 1% 0% >> 41661 scott bash 828K / 17M 4% 0% >> 15727 scott bash 808K / 17M 4% 0% >> 39420 scott bash 804K / 17M 4% 0% >> 2455 scott bash 544K / 15M 3% 0% >> 39367 scott tmux 512K / 15M 3% 0% >> 2447 scott bash 376K / 15M 2% 0% >> 2450 scott bash 364K / 15M 2% 0% >> 2453 scott bash 324K / 15M 2% 0% >> 2454 scott bash 316K / 15M 2% 0% >> 2445 scott bash 312K / 15M 2% 0% >> 44937 scott bash 304K / 17M 1% 0% >> 2458 scott bash 72K / 15M 0% 0% >> >> At least they agree about it being 11G. Is this kernel memory being >> paged out to swap? The machine has 128G of physical memory and isn't un= der >> very heavy load at the moment. >> >> I suspect this is a bug in some kernel module... possibly >> wireguard because I run wireguard in a vnet jail and didn't observe this >> problem until setting that up. But I don't have any hard evidence. >> >> I've tried to mitigate this via swapoff -a. This works once but the nex= t >> day swap will be back, even fuller. I've been doing regular reboots to >> fix this but would like to get to the bottom of it. If left alone, swap >> will >> fill up and the machine will get into a "not quite hung" but unusable an= d >> useless state. >> >> Am I off-base with my suspicion that this is kernel mode memory? Can >> someone teach me how to diagnose the status of kernel mode memory heap? >> >> Thx, >> Scott >> >> --000000000000236b100600f7d43f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Ok, I'm an idiot.=C2=A0 I'm writing to confess and= to maybe save someone else in the future.=C2=A0 The issue was I mounted a = tmpfs on /tmp and didn't specify an upper size limit.=C2=A0 Invariably = over time, /tmp would begin to fill up and my swap space would start to be = used.=C2=A0 Of course, I couldn't find any usermode process that was us= ing the swap and I jumped to the conclusion that this had something to do w= ith kernel memory.=C2=A0 But really it was my own stupidity.<div><br></div>= <div>Thank you to Pete and others who tried to help.</div><div><br></div><d= iv>Scott</div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class= =3D"gmail_attr">On Wed, Jul 19, 2023 at 4:15=E2=80=AFPM Scott Gasch <<a = href=3D"mailto:scott.gasch@gmail.com">scott.gasch@gmail.com</a>> wrote:<= br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8e= x;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"= >Replying to my own post with more info... I tried stopping my wireguard ja= il and unloading the if_wg kmod and it did not affect the swap memory usage= .=C2=A0 Not sure if that lets wireguard off the hook or not though.<div><br= ></div><div>If someone who understands kernel memory could chime in... it l= ooks to me like the aggregate swap usage of usermode processes is nowhere n= ear the total swap space used so I suspect something in kernel mode.=C2=A0 = Does this make sense or is there another explanation?</div><div><br></div><= div>Thx,</div><div>Scott</div><div><br></div></div><br><div class=3D"gmail_= quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Jul 19, 2023 at 7:49= =E2=80=AFAM Scott Gasch <<a href=3D"mailto:scott.gasch@gmail.com" target= =3D"_blank">scott.gasch@gmail.com</a>> wrote:<br></div><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg= b(204,204,204);padding-left:1ex"><div dir=3D"ltr">I am running a 13.2-RELEA= SE GENERIC kernel and seeing a pattern where, after about 10 days of uptime= , my swap begins to fill up.<br><br><font face=3D"monospace"># swapinfo -h<= /font><div><font face=3D"monospace">Device =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0Size =C2=A0 =C2=A0 Used =C2=A0 =C2=A0Avail Capacity<br>/de= v/ada0p3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A048G =C2=A0 =C2=A0 3.6G =C2=A0 = =C2=A0 =C2=A044G =C2=A0 =C2=A0 7%<br>/dev/ada1p3 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A048G =C2=A0 =C2=A0 3.6G =C2=A0 =C2=A0 =C2=A044G =C2=A0 =C2=A0 7%<b= r>/dev/ada2p3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A048G =C2=A0 =C2=A0 3.6G =C2= =A0 =C2=A0 =C2=A044G =C2=A0 =C2=A0 7%<br>Total =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 144G =C2=A0 =C2=A0 =C2=A011G =C2=A0 =C2=A0 133G =C2=A0= =C2=A0 7%</font></div><div><br></div><div>So, 11G of total swap space.=C2= =A0 What's using it?</div><div><br></div><div><font face=3D"monospace">= # systat -swap</font></div><div><font face=3D"monospace">=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /0 =C2=A0 /1 =C2=A0 /2= =C2=A0 /3 =C2=A0 /4 =C2=A0 /5 =C2=A0 /6 =C2=A0 /7 =C2=A0 /8 =C2=A0 /9 =C2= =A0 /10<br>=C2=A0 =C2=A0 =C2=A0Load Average =C2=A0 ||||||<br><br>Device/Pat= h =C2=A0 =C2=A0 =C2=A0 Size =C2=A0Used |0% =C2=A0/10 =C2=A0/20 =C2=A0/30 = =C2=A0/40 =C2=A0/ 60\ =C2=A070\ =C2=A080\ =C2=A090\ 100|<br>ada0p3 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 48G 3660M XXX<br>ada1p3 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 48G 3666M XXX<br>ada2p3 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 48G 3664M XXX<br>Total =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 144G =C2=A0 11G XXX<br><br>Pid =C2=A0 =C2=A0Username =C2=A0 Comm= and =C2=A0 =C2=A0 Swap/Total Per-Process =C2=A0 =C2=A0Per-System<br>=C2=A01= 4703 scott =C2=A0 =C2=A0 =C2=A0python3.8 =C2=A0 =C2=A04M / 154M =C2=A02% = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%<br>=C2=A0 2451 scott =C2= =A0 =C2=A0 =C2=A0rclone =C2=A0 =C2=A0 =C2=A0 4M / 934M =C2=A00% =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%<br>=C2=A0 2452 scott =C2=A0 =C2=A0= =C2=A0rclone =C2=A0 =C2=A0 =C2=A0 3M / =C2=A0 1G =C2=A00% =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%<br>=C2=A073827 scott =C2=A0 =C2=A0 =C2= =A0bash =C2=A0 =C2=A0 =C2=A0 =C2=A0 1M / =C2=A017M =C2=A06% =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%<br>=C2=A039416 scott =C2=A0 =C2=A0 =C2= =A0tmux =C2=A0 =C2=A0 =C2=A0 968K / =C2=A054M =C2=A01% =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A00%<br>=C2=A041661 scott =C2=A0 =C2=A0 =C2=A0bas= h =C2=A0 =C2=A0 =C2=A0 828K / =C2=A017M =C2=A04% =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A00%<br>=C2=A015727 scott =C2=A0 =C2=A0 =C2=A0bash = =C2=A0 =C2=A0 =C2=A0 808K / =C2=A017M =C2=A04% =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A00%<br>=C2=A039420 scott =C2=A0 =C2=A0 =C2=A0bash =C2=A0= =C2=A0 =C2=A0 804K / =C2=A017M =C2=A04% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A00%<br>=C2=A0 2455 scott =C2=A0 =C2=A0 =C2=A0bash =C2=A0 =C2= =A0 =C2=A0 544K / =C2=A015M =C2=A03% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A00%<br>=C2=A039367 scott =C2=A0 =C2=A0 =C2=A0tmux =C2=A0 =C2=A0 = =C2=A0 512K / =C2=A015M =C2=A03% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A00%<br>=C2=A0 2447 scott =C2=A0 =C2=A0 =C2=A0bash =C2=A0 =C2=A0 =C2=A0= 376K / =C2=A015M =C2=A02% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= 0%<br>=C2=A0 2450 scott =C2=A0 =C2=A0 =C2=A0bash =C2=A0 =C2=A0 =C2=A0 364K = / =C2=A015M =C2=A02% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%<br>= =C2=A0 2453 scott =C2=A0 =C2=A0 =C2=A0bash =C2=A0 =C2=A0 =C2=A0 324K / =C2= =A015M =C2=A02% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%<br>=C2= =A0 2454 scott =C2=A0 =C2=A0 =C2=A0bash =C2=A0 =C2=A0 =C2=A0 316K / =C2=A01= 5M =C2=A02% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%<br>=C2=A0 24= 45 scott =C2=A0 =C2=A0 =C2=A0bash =C2=A0 =C2=A0 =C2=A0 312K / =C2=A015M =C2= =A02% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%<br>=C2=A044937 sco= tt =C2=A0 =C2=A0 =C2=A0bash =C2=A0 =C2=A0 =C2=A0 304K / =C2=A017M =C2=A01% = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%<br>=C2=A0 2458 scott =C2= =A0 =C2=A0 =C2=A0bash =C2=A0 =C2=A0 =C2=A0 =C2=A072K / =C2=A015M =C2=A00% = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00%<br></font></div><div><br= ></div><div>At least they agree about it being 11G.=C2=A0 Is this kernel me= mory being paged out to swap?=C2=A0 The machine has 128G of physical memory= and isn't under very heavy load at the moment.=C2=A0=C2=A0</div><div><= br></div><div>I suspect this is a bug in some kernel module... possibly wir= eguard=C2=A0because I run wireguard in a vnet jail and didn't observe t= his problem until setting that up.=C2=A0 But I don't have any hard evid= ence.</div><div><br>I've tried to mitigate this via swapoff -a.=C2=A0 T= his works once but the next<br>day swap will be back, even fuller.=C2=A0 I&= #39;ve been doing regular reboots to fix this but would like to get to the = bottom of it.=C2=A0 If left alone, swap will<br>fill up and the machine wil= l get into a "not quite hung" but unusable and useless state.<br>= <br>Am I off-base with my suspicion that this is kernel mode memory? Can so= meone teach me how to diagnose the status of kernel mode memory heap?<br><b= r>Thx,<br>Scott</div><div><br></div></div> </blockquote></div> </blockquote></div> --000000000000236b100600f7d43f--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABYAQkSpZshyfMrbC3ykfyFqZK4KPBPN4a5L8GQME3oZ8KuUnQ>