From owner-freebsd-stable@FreeBSD.ORG Wed Nov 29 07:33:58 2006 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5BE9C16A407 for ; Wed, 29 Nov 2006 07:33:58 +0000 (UTC) (envelope-from yar@comp.chem.msu.su) Received: from comp.chem.msu.su (comp.chem.msu.su [158.250.32.97]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0842D43E00 for ; Wed, 29 Nov 2006 07:31:44 +0000 (GMT) (envelope-from yar@comp.chem.msu.su) Received: from comp.chem.msu.su (localhost [127.0.0.1]) by comp.chem.msu.su (8.13.4/8.13.3) with ESMTP id kAT5QTGZ020303; Wed, 29 Nov 2006 08:26:29 +0300 (MSK) (envelope-from yar@comp.chem.msu.su) Received: (from yar@localhost) by comp.chem.msu.su (8.13.4/8.13.3/Submit) id kAT5QPnp020302; Wed, 29 Nov 2006 08:26:25 +0300 (MSK) (envelope-from yar) Date: Wed, 29 Nov 2006 08:26:25 +0300 From: Yar Tikhiy To: Joerg Pernfuss Message-ID: <20061129052625.GG96853@comp.chem.msu.su> References: <20061128195452.05e38596@loki.starkstrom.lan> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20061128195452.05e38596@loki.starkstrom.lan> User-Agent: Mutt/1.5.9i Cc: stable@freebsd.org Subject: Re: kernel statistics weirdness on RELENG_6{_1,_2} / 64bit? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Nov 2006 07:33:58 -0000 On Tue, Nov 28, 2006 at 07:54:52PM +0100, Joerg Pernfuss wrote: > Hi, > > a while ago I noticed something odd on my 64bit FreeBSD 6 systems - two > amd64, one alpha. I can't make head or tails of it, so I post it in hope > that someone can. > > Box #1: > elessar@snake: ~% uname -srm > FreeBSD 6.1-RELEASE-p7 amd64 > elessar@snake: ~% grep kern.ipc /boot/loader.conf > kern.ipc.nsfbufs="8192" > kern.ipc.nmbclusters="49152" > elessar@snake: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters > kern.ipc.nsfbufs: 0 > kern.ipc.nmbclusters: 32768 > elessar@snake: ~% netstat -m > 350/132/482/49152 mbuf clusters in use (current/cache/total/max) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 2864770 requests for I/O initiated by sendfile > > Box #2: > elessar@forseti: ~% uname -srm > FreeBSD 6.2-RC1 alpha > elessar@forseti: ~% grep kern.ipc /boot/loader.conf > kern.ipc.nsfbufs="4294967300" > elessar@forseti: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters > kern.ipc.nsfbufs: 0 > kern.ipc.nmbclusters: 1664 > elessar@forseti: ~% netstat -m > 64/80/144/1664 mbuf clusters in use (current/cache/total/max) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > > Box #3: > elessar@rivendell: ~% uname -srm > FreeBSD 6.1-RELEASE-p10 amd64 > elessar@rivendell: ~% grep kern.ipc /boot/loader.conf > kern.ipc.nsfbufs="8192" > kern.ipc.nmbclusters="49152" > elessar@rivendell: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters > kern.ipc.nsfbufs: 0 > kern.ipc.nmbclusters: 32768 > elessar@rivendell: ~% grep NSF /root/kernel/RIVENDELL > options NSFBUFS=8192 > elessar@rivendell: ~% netstat -m > 64/118/182/49152 mbuf clusters in use (current/cache/total/max) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 35 requests for I/O initiated by sendfile > > For comparison, Box #4: > elessar@web2: ~% uname -srm > FreeBSD 6.1-STABLE i386 > elessar@web2: ~% grep kern.ipc /boot/loader.conf > kern.ipc.nsfbufs="8192" > elessar@web2: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters > kern.ipc.nsfbufs: 8192 > kern.ipc.nmbclusters: 32768 > elessar@web2: ~% netstat -m > 104/272/376/32768 mbuf clusters in use (current/cache/total/max) > 1598/2635/8192 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 8938512 requests for I/O initiated by sendfile > > So, looking at the `netstat -m' output, setting kern.ipc.nmbclusters > in the loader.conf seems to work, but the sysctl reports the default > value that would be in place, if it wasn't overridden in the loader.conf. > > For the sendfile buffers, the stats are completely out of whack. > The sysctl reports 0 for every value (`netstat-m' just prints the sysctl > fields for these values if I understood the source right (unlikely). > But sendfile() works and the accounting on the number of sendfile calls > seems to work too. > So I believe the NSFBUFS kernel option and the loader.conf setting work, > but it is quite impossible to properly tune the buffers to the workload. > I did not bother trying the NMBCLUSTERS kernel option, it was removed > a while ago anyway. > > The loader.conf kern.ipc.nsfbufs value on box #2 is ridiculously high, > yes, I tried to get beyond the boundary of a u_int_32 to see if maybe > something good or bad happens. It did not. > > At this point, I wanted to include links to two mails on amd64@ I > found when I first searched for this oddity, both of which had no > reply. Instead I found a posting to net@ that occured in the meantime, > and a reply to it by Yar Tikhiy two days ago explaining the nsfbufs issue. > There are no such buffers on amd64, ia64 and alpha as they aren't needed. > > http://docs.freebsd.org/cgi/getmsg.cgi?fetch=26091+0+current/freebsd-net > > Ok, that leaves the question why `sysctl kern.ipc.nmbclusters' reports a > different value compared to the `netstat -m' output on box #1 and #3. netstat -m doesn't use the sysctl. It reads the zone limit via memstat_get_countlimit() from libmemstat(3) instead. Perhaps this can account for the difference observed. > Thanks for any pointers, > Joerg > > PS: documenting the nsfbufs issue on amd64/ia64/alpha might be a good > PS: idea. loader(8) seems a good place. Opinions? I've already documented the issue in sendfile(2) in CURRENT. MFC is due. A small note in loader(8) would be good, too -- thanks for the suggestion. -- Yar