Date: Wed, 24 Feb 2021 10:34:23 -0700 From: Alan Somers <asomers@freebsd.org> To: Olivier Certner <olivier.freebsd@free.fr> Cc: Konstantin Belousov <kostikbel@gmail.com>, FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: The out-of-swap killer makes poor choices Message-ID: <CAOtMX2iYr4NDYE0xHSa_w1hA5XQ2m9cA28NzPoGbfzAKKox9aQ@mail.gmail.com> In-Reply-To: <1984125.0OzZcVfBr4@ravel> References: <CAOtMX2jYmrK7ftx62_NEfNCWS7O=giHKL1p9kXCqq1t5E1arxA@mail.gmail.com> <CAOtMX2i3Njo=KBP=99_G0%2BKuSa00CVgNvacmzhTaoZUYEhwPPA@mail.gmail.com> <YDYyQ1V/hEAGV%2ByJ@kib.kiev.ua> <1984125.0OzZcVfBr4@ravel>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Feb 24, 2021 at 5:02 AM Olivier Certner <olivier.freebsd@free.fr> wrote: > Hi, > > > > Ok, I'll abandon this idea. > > I hope you don't abandon the idea of improving the OOM killer in the long > term > if you feel that something is wrong. > > > I explained the reasoning for the current design, even if it actually > > evolved this way, instead being written as a whole with the stated goal. > > I do not object against adding something that would help to get it more > > fit with different goals as well, but the current idea of making the > > system survive should be kept. > > So true. The main goal (system survival) does not prevent (not so) > secondary > ones. > > I'm sorry not to have any technical contribution to propose, but instead I > have some testimony that may interest you, although old. > > 2 to 3 years ago, I stumbled against production problems on servers doing > heavy computations. Only a few processes (2 generally) were doing them, > and > most of the time consumed less than 1/4 of the available RAM (2 GiB). > Apart > from that, no other process was allocating any significant amount of > memory. > Only some base default daemons (syslogd, cron) and sshd were running. > Occasionally, very big jobs would come, and one or more of these processes > would start eating up all available memory, until FreeBSD decided that it > was > time to take action. > > Sometimes it would decide to kill one of these processes, but more often > than > not sshd or cron were killed instead, although they were consuming > ridiculous > amounts of memory. I tried tweaks via vm.pageout_oom_seq (I think I set it > to > 120, as Mark did) and vm.pfault_oom_attempts, without much change. > > In the end, I decided to use 'protect', via rc.conf's '*_oomprotect="YES"' > facility, to workaround this problem and save me some headaches. > > At some point, some of these machines had swap configured (separate AWS > disks), but later I removed swap entirely. What I report occurred for the > latter configuration, but IIRC I observed similar behavior in the former. > > I have not had this use case since then, so I can't say if this has been > fixed > (by commits such as r353734/d307bdcc2c473858) or not. > > -- > Olivier Certner > There's another silly problem that I didn't mention in my original post. The old rule of thumb is that the swap partition's size should be twice as large as the amount of RAM. However, that's no longer possible in many cases. The kernel imposes a hard limit of 64 GiB (on amd64 at least) on the usable size of any swap partition, and many servers now have far more than 64 GiB of RAM. So the advice needs to change with the times. I don't know what the best size would be for a modern server, but I would guess that it must be at least several times the RSS of your largest process, and also at least one tenth of RAM (for use as a dump device with compressed core dumps). -Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2iYr4NDYE0xHSa_w1hA5XQ2m9cA28NzPoGbfzAKKox9aQ>