From owner-freebsd-stable@FreeBSD.ORG Fri Aug 1 18:07:51 2008 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 778661065674 for ; Fri, 1 Aug 2008 18:07:51 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id EFA038FC0C; Fri, 1 Aug 2008 18:07:49 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <489350F4.5070009@FreeBSD.org> Date: Fri, 01 Aug 2008 20:07:48 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.16 (Macintosh/20080707) MIME-Version: 1.0 To: Royce Williams References: <488638DA.7010005@alaska.net> <20080723053404.GA46617@eos.sc1.parodius.com> <4886D1E9.1090905@alaska.net> <48933557.8010904@alaska.net> In-Reply-To: <48933557.8010904@alaska.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org Subject: Re: 6.3-RELEASE-p3 recurring panics on multiple SM PDSMi+ X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Aug 2008 18:07:51 -0000 Royce Williams wrote: > Royce Williams wrote, on 7/22/2008 10:38 PM: >> Jeremy Chadwick wrote, on 7/22/2008 9:34 PM: >>> On Tue, Jul 22, 2008 at 11:45:30AM -0800, Royce Williams wrote: >>>> We have 10 SuperMicro PDSMi+ 5015M-MTs that are panic'ing every few >>>> days. This started shortly after upgrade from 6.2-RELEASE to >>>> 6.3-RELEASE with freebsd-update. >>> We use the same hardware (board and chassis), and have no such problems >>> running both RELENG_6 and RELENG_7. >>> >>> I don't think your issue is specific to the board or chassis. Kris's >>> explanation makes a lot more sense. :-) >> Jeremy/Kris/Clifton - >> >> Looks like we have consensus. :-) Thanks, all of you, for your >> helpful insight. >> >> I've bumped vm.kmem_size up to 400M on half of the affected boxes, >> leaving the other half as a control group. I'll report back once I >> have something to report. > > After having bumped up to 400M, a few boxes panic'd again yesterday. > I caught a core, and it is "kmem_map too small", just as Kris > suspected: > > Jul 31 15:38:05 [redacted] savecore: reboot after panic: kmem_malloc(4096): kmem_map too small: 419430400 total allocated > > > The docs state that 400M should be plenty for systems up to 6G, but > Kris said earlier in this thread that it's better to say 'increase > until the pain stops'. :-) Accordingly, I have some some follow-up > questions; hopefully, this will be useful to others. > > - What is a reasonable increment? (I'm trying 448M next). > > - What are the practical and hard maximums? > > - I suspect that it's worth trying to make kmem 'as big as I need, but > no bigger', so that non-kernel memory is also maximized? > > - In a larger sense, if 400M is probably big enough for 6G systems, > and these are 4G systems, should I be suspicious that 400M isn't > cutting it? In other words, is there a point at which should I be > looking for obvious places where the kernel is eating too much memory > and reduce them, rather than feeding it more? > > For example, I recall now that a network guy in my group did some > sysctl tuning relating to networking on these systems, and I see > from man tuning(7) that a number of these tweaks (obviously) can > cause increased kernel consumption. > > $ egrep -v '^#|^$' /etc/sysctl.conf | sort > kern.corefile=/var/cores/%U/%N-%P.core > kern.ipc.maxsockbuf=8388608 > kern.ipc.maxsockets=32768 > kern.ipc.nmbclusters=65535 > kern.ipc.somaxconn=4096 > kern.maxfiles=262144 > kern.maxfilesperproc=65535 > net.inet.ip.portrange.first=8192 > net.inet.ip.portrange.hifirst=8192 > net.inet.ip.portrange.hilast=65535 > net.inet.ip.portrange.last=65535 > net.inet.ipf.fr_tcpclosed=60 > net.inet.ipf.fr_tcpclosewait=120 > net.inet.ipf.fr_tcphalfclosed=300 > net.inet.ipf.fr_udptimeout=120 > net.inet.tcp.delayed_ack=0 > net.inet.tcp.inflight.enable=0 > net.inet.tcp.msl=10000 > net.inet.tcp.mssdflt=1460 > net.inet.tcp.recvspace=65536 > net.inet.tcp.rfc1323=1 > net.inet.tcp.sendspace=65536 > net.inet.udp.maxdgram=57344 > net.inet.udp.recvspace=65535 > vfs.nfs.iodmax=32 > vfs.nfs.iodmin=8 > > > My apologies for not including this sooner. I didn't think of it > until yesterday, primarily because it had been fine under 6.2. In > retrospect, that was bad reasoning. > > Royce > The statement that "400MB should be enough for up to 6GB" is completely bogus. The amount of memory your kernel needs is a function of the work you give it to do. On i386 the kernel only has 1GB of address space though you can increase it by tuning KVA_PAGES at the expense of less memory available to user processes (everything comes out of 4GB address space). So that is the upper bound although other things need to fit in there too. On amd64 the situation is more complicated but on older versions than 8.0 there is 2GB for the kernel address space and in practise a limit of about 1500MB of kmem. It is possible that you are hitting a memory leak but I would just increase kmem further and see if it persists. Looking at vmstat -m etc may help to figure out if something is leaking over time. Kris