Date: Sat, 16 Feb 2013 12:25:25 -0600 From: Alan Cox <alan.l.cox@gmail.com> To: Eugene Grosbein <eugen@grosbein.pp.ru> Cc: stable@freebsd.org Subject: Re: i386: vm.pmap kernel local race condition Message-ID: <CAJUyCcOHFgGqzvm43y-dojvsXdNydYmfd9rwHMkdjqDyvjJuKQ@mail.gmail.com> In-Reply-To: <511CECCC.60400@grosbein.pp.ru> References: <511CECCC.60400@grosbein.pp.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Feb 14, 2013 at 7:55 AM, Eugene Grosbein <eugen@grosbein.pp.ru>wrote: > Hi! > > I've got FreeBSD 8.3-STABLE/i386 server that can be reliably panicked > using just 'squid -k rotatelog' command. It seems the system suffers > from the problem described here: > > http://cxsecurity.com/issue/WLB-2010090156 > > I could not find any FreeBSD Security Advisory containing a fix. > > My server has 4G physical RAM (about 3.2G available) and runs > squid (about 110M VSS) with 500 ntlm_auth subprocesses. > Lesser number of ntlm_auth sometimes results in squid crash > as it sometimes has several hundreds requests per second to authorize > and is intolerant to exhaustion of free ntlm_auth. > > "squid -k rotatelog" at midnight results in crash: > > Feb 14 00:03:00 irl savecore: reboot after panic: get_pv_entry: increase > vm.pmap.shpgperproc > Feb 14 00:03:00 irl savecore: writing core to vmcore.1 > > Btw, I have coredump. > > vm.pmap.shpgperproc has default value (200) here, as well as m.v_free_min, > vm.v_free_reserved, and vm.v_free_target and KVA_PAGES. > > These crashes are pretty regular > > # last|fgrep reboot > reboot ~ Thu Feb 14 00:03 > reboot ~ Wed Feb 13 19:08 > reboot ~ Wed Feb 13 10:40 > reboot ~ Wed Feb 13 00:04 > reboot ~ Tue Feb 12 00:09 > reboot ~ Mon Feb 11 00:03 > reboot ~ Sun Feb 10 00:03 > reboot ~ Thu Feb 7 00:03 > reboot ~ Wed Feb 6 10:52 > reboot ~ Sun Feb 3 00:03 > reboot ~ Sat Feb 2 00:03 > > May this be considered as security problem? > Can it be fixed without switch to amd64? > I have only remote access to this production server, no serial console. > > Regardless of what that web site says, this is not really a race condition. Instead, you're exhausting a resource in the kernel because of the characteristics of your workload. The kernel tries to handle this gracefully, but in extreme cases, the kernel can't keep up with the demand. Have you simply tried doing as the panic message suggests, i.e., increase vm.pmap.shpgperproc? Alternatively, you can increase vm.pmap.pv_entry_max to more directly accomplish the same. That said, if possible, you should do as Adrian suggests and change your Squid configuration to not use 500 helper processes. That will allow a lot more of your machine's physical memory to go to caching data rather bookkeeping data structures in the kernel. Regards, Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJUyCcOHFgGqzvm43y-dojvsXdNydYmfd9rwHMkdjqDyvjJuKQ>