Date: Fri, 5 Jul 2024 10:56:17 +0300 From: Odhiambo Washington <odhiambo@gmail.com> To: questions <questions@freebsd.org> Subject: Server became inaccessible because it ran out of swap space Message-ID: <CAAdA2WPSngEy4Dr4Yt8B7CHboHbxaYBaCpK2VZ%2BppB4fWYUX2g@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
--000000000000249ff1061c7b68b8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I have a server with 64GB RAM, 2CPUs each with 16 cores. I have also configured 13GB or swap space. ``` root@gw:/usr/local/bhyve-vms/scripts # swapinfo Device 1K-blocks Used Avail Capacity /dev/ada0p3 3163136 703316 2459820 22% /dev/md0.eli 10485760 709352 9776408 7% Total 13648896 1412668 12236228 10% root@gw:/usr/local/bhyve-vms/scripts # ``` A number of times it has become inaccessible until I do a hard reboot and this has been caused by what I believe is running out of swap. Below is what I have obtained from /var/log/messages after I rebooted. How do I identify the culprit? Arrest the situation? ``` Jul 5 06:50:56 gw kernel: failed Jul 5 06:52:11 gw kernel: failed Jul 5 06:52:11 gw kernel: out of swap space Jul 5 06:52:11 gw kernel: failed Jul 5 06:52:11 gw kernel: failed Jul 5 06:52:12 gw kernel: failed Jul 5 06:52:12 gw kernel: failed Jul 5 06:54:06 gw kernel: out of swap space Jul 5 06:54:06 gw kernel: failed Jul 5 07:16:30 gw kernel: pid 4076 (bhyve), jid 0, uid 0, was killed: failed to reclaim memory Jul 5 07:16:30 gw kernel: pid 4076 (bhyve), jid 0, uid 0, was killed: failed to reclaim memory Jul 5 07:16:30 gw kernel: tap4: link state changed to DOWN Jul 5 07:16:30 gw kernel: out of swap space Jul 5 07:16:30 gw kernel: failed Jul 5 07:16:30 gw kernel: failed Jul 5 07:16:30 gw kernel: failed Jul 5 07:16:30 gw kernel: pid 20849 (bhyve), jid 0, uid 0, was killed: failed to reclaim memory Jul 5 07:16:30 gw kernel: pid 20849 (bhyve), jid 0, uid 0, was killed: failed to reclaim memory Jul 5 07:16:30 gw kernel: tap5: link state changed to DOWN Jul 5 07:16:30 gw kernel: failed Jul 5 07:16:30 gw kernel: failed Jul 5 07:16:30 gw kernel: sonewconn: pcb 0xfffff8002866d100 (local:/var/run/wsgi.38620.0.1.sock): Listen queue overflow: 151 already in queue awaiting acceptance (1 occurrences), euid 0, rgid 0, jail 0 Jul 5 07:16:30 gw kernel: pid 3591 (bhyve), jid 0, uid 0, was killed: failed to reclaim memory Jul 5 07:16:30 gw kernel: pid 3591 (bhyve), jid 0, uid 0, was killed: failed to reclaim memory Jul 5 07:16:30 gw kernel: tap3: link state changed to DOWN Jul 5 07:16:30 gw kernel: failed Jul 5 07:16:30 gw kernel: out of swap space Jul 5 07:16:30 gw kernel: failed Jul 5 07:16:31 gw kernel: failed Jul 5 07:16:31 gw kernel: failed Jul 5 07:16:32 gw kernel: out of swap space Jul 5 07:16:33 gw kernel: out of swap space Jul 5 07:16:33 gw kernel: failed Jul 5 07:16:33 gw kernel: failed Jul 5 07:16:34 gw kernel: out of swap space Jul 5 07:16:34 gw kernel: failed Jul 5 07:16:36 gw kernel: failed Jul 5 07:16:36 gw kernel: failed Jul 5 07:16:36 gw kernel: failed Jul 5 07:16:36 gw kernel: failed Jul 5 07:16:36 gw kernel: failed Jul 5 07:16:37 gw kernel: failed Jul 5 07:16:37 gw kernel: failed Jul 5 07:16:37 gw kernel: failed Jul 5 07:16:37 gw kernel: failed Jul 5 07:16:37 gw kernel: failed Jul 5 07:16:37 gw kernel: failed Jul 5 07:16:37 gw kernel: failed Jul 5 07:16:38 gw kernel: failed ``` --=20 Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' =C2=AF\_(=E3=83=84)_/=C2=AF :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html] --000000000000249ff1061c7b68b8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">I have a server with 64GB RAM, 2CPUs each with 16 cores. I= have also configured 13GB or swap space.<div><br></div><div>```</div><div>= root@gw:/usr/local/bhyve-vms/scripts # swapinfo<br>Device =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A01K-blocks =C2=A0 =C2=A0 Used =C2=A0 =C2=A0Avail Capacity<b= r>/dev/ada0p3 =C2=A0 =C2=A0 =C2=A0 3163136 =C2=A0 703316 =C2=A02459820 =C2= =A0 =C2=A022%<br>/dev/md0.eli =C2=A0 =C2=A0 10485760 =C2=A0 709352 =C2=A097= 76408 =C2=A0 =C2=A0 7%<br>Total =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A013= 648896 =C2=A01412668 12236228 =C2=A0 =C2=A010%<br>root@gw:/usr/local/bhyve-= vms/scripts #<br></div><div>```</div><div><br></div><div><div>A number of t= imes it has become inaccessible until I do a hard reboot and this has been = caused by what I believe is running out of swap.</div><div><br></div><div>B= elow is what I have obtained from /var/log/messages after I rebooted.</div>= <div><br></div><div>How do I identify the culprit? Arrest the situation?</d= iv><div><br></div><div><br></div><div>```</div><div>Jul =C2=A05 06:50:56 gw= kernel: failed<br>Jul =C2=A05 06:52:11 gw kernel: failed<br>Jul =C2=A05 06= :52:11 gw kernel: out of swap space<br>Jul =C2=A05 06:52:11 gw kernel: fail= ed<br>Jul =C2=A05 06:52:11 gw kernel: failed<br>Jul =C2=A05 06:52:12 gw ker= nel: failed<br>Jul =C2=A05 06:52:12 gw kernel: failed<br>Jul =C2=A05 06:54:= 06 gw kernel: out of swap space<br>Jul =C2=A05 06:54:06 gw kernel: failed<b= r>Jul =C2=A05 07:16:30 gw kernel: pid 4076 (bhyve), jid 0, uid 0, was kille= d: failed to reclaim memory<br>Jul =C2=A05 07:16:30 gw kernel: pid 4076 (bh= yve), jid 0, uid 0, was killed: failed to reclaim memory<br>Jul =C2=A05 07:= 16:30 gw kernel: tap4: link state changed to DOWN<br>Jul =C2=A05 07:16:30 g= w kernel: out of swap space<br>Jul =C2=A05 07:16:30 gw kernel: failed<br>Ju= l =C2=A05 07:16:30 gw kernel: failed<br>Jul =C2=A05 07:16:30 gw kernel: fai= led<br>Jul =C2=A05 07:16:30 gw kernel: pid 20849 (bhyve), jid 0, uid 0, was= killed: failed to reclaim memory<br>Jul =C2=A05 07:16:30 gw kernel: pid 20= 849 (bhyve), jid 0, uid 0, was killed: failed to reclaim memory<br>Jul =C2= =A05 07:16:30 gw kernel: tap5: link state changed to DOWN<br>Jul =C2=A05 07= :16:30 gw kernel: failed<br>Jul =C2=A05 07:16:30 gw kernel: failed<br>Jul = =C2=A05 07:16:30 gw kernel: sonewconn: pcb 0xfffff8002866d100 (local:/var/r= un/wsgi.38620.0.1.sock): Listen queue overflow: 151 already in queue awaiti= ng acceptance (1 occurrences), euid 0, rgid 0, jail 0<br>Jul =C2=A05 07:16:= 30 gw kernel: pid 3591 (bhyve), jid 0, uid 0, was killed: failed to reclaim= memory<br>Jul =C2=A05 07:16:30 gw kernel: pid 3591 (bhyve), jid 0, uid 0, = was killed: failed to reclaim memory<br>Jul =C2=A05 07:16:30 gw kernel: tap= 3: link state changed to DOWN<br>Jul =C2=A05 07:16:30 gw kernel: failed<br>= Jul =C2=A05 07:16:30 gw kernel: out of swap space<br>Jul =C2=A05 07:16:30 g= w kernel: failed<br>Jul =C2=A05 07:16:31 gw kernel: failed<br>Jul =C2=A05 0= 7:16:31 gw kernel: failed<br>Jul =C2=A05 07:16:32 gw kernel: out of swap sp= ace<br>Jul =C2=A05 07:16:33 gw kernel: out of swap space<br>Jul =C2=A05 07:= 16:33 gw kernel: failed<br>Jul =C2=A05 07:16:33 gw kernel: failed<br>Jul = =C2=A05 07:16:34 gw kernel: out of swap space<br>Jul =C2=A05 07:16:34 gw ke= rnel: failed<br>Jul =C2=A05 07:16:36 gw kernel: failed<br>Jul =C2=A05 07:16= :36 gw kernel: failed<br>Jul =C2=A05 07:16:36 gw kernel: failed<br>Jul =C2= =A05 07:16:36 gw kernel: failed<br>Jul =C2=A05 07:16:36 gw kernel: failed<b= r>Jul =C2=A05 07:16:37 gw kernel: failed<br>Jul =C2=A05 07:16:37 gw kernel:= failed<br>Jul =C2=A05 07:16:37 gw kernel: failed<br>Jul =C2=A05 07:16:37 g= w kernel: failed<br>Jul =C2=A05 07:16:37 gw kernel: failed<br>Jul =C2=A05 0= 7:16:37 gw kernel: failed<br>Jul =C2=A05 07:16:37 gw kernel: failed<br>Jul = =C2=A05 07:16:38 gw kernel: failed</div><div>```</div><div><br><div><div><b= r></div><span class=3D"gmail_signature_prefix">-- </span><br><div dir=3D"lt= r" class=3D"gmail_signature" data-smartmail=3D"gmail_signature"><div dir=3D= "ltr"><div dir=3D"ltr"><div>Best regards,<br>Odhiambo WASHINGTON,<br>Nairob= i,KE<br>+254 7 3200 0004/+254 7 2274 3223</div><div><span style=3D"color:rg= b(34,34,34)">=C2=A0In=C2=A0</span><span style=3D"color:rgb(34,34,34)">an In= ternet failure case, the #1 suspect is a constant: DNS.</span><br>"<sp= an style=3D"font-size:12.8px">Oh, the cruft.</span><span style=3D"font-size= :12.8px">",=C2=A0</span><span style=3D"font-size:12.8px">egrep -v '= ;^$|^.*#'=C2=A0</span><span style=3D"background-color:rgb(34,34,34);col= or:rgb(238,238,238);font-family:"Lucida Console",Consolas,"C= ourier New",monospace;font-size:13.6px">=C2=AF\_(=E3=83=84)_/=C2=AF</s= pan><span style=3D"font-size:12.8px">=C2=A0:-)</span></div><div><span style= =3D"font-size:12.8px">[How to ask smart questions:=C2=A0</span><span style= =3D"font-size:12.8px"><a href=3D"http://www.catb.org/~esr/faqs/smart-questi= ons.html" target=3D"_blank">http://www.catb.org/~esr/faqs/smart-questions.h= tml</a>]</span></div></div></div></div></div></div></div></div> --000000000000249ff1061c7b68b8--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAdA2WPSngEy4Dr4Yt8B7CHboHbxaYBaCpK2VZ%2BppB4fWYUX2g>