Date: Wed, 29 Nov 2006 08:26:25 +0300
From: Yar Tikhiy <yar@comp.chem.msu.su>
To: Joerg Pernfuss <elessar@bsdforen.de>
Cc: stable@freebsd.org
Subject: Re: kernel statistics weirdness on RELENG_6{_1,_2} / 64bit?
Message-ID: <20061129052625.GG96853@comp.chem.msu.su>
In-Reply-To: <20061128195452.05e38596@loki.starkstrom.lan>
References: <20061128195452.05e38596@loki.starkstrom.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Nov 28, 2006 at 07:54:52PM +0100, Joerg Pernfuss wrote: > Hi, > > a while ago I noticed something odd on my 64bit FreeBSD 6 systems - two > amd64, one alpha. I can't make head or tails of it, so I post it in hope > that someone can. > > Box #1: > elessar@snake: ~% uname -srm > FreeBSD 6.1-RELEASE-p7 amd64 > elessar@snake: ~% grep kern.ipc /boot/loader.conf > kern.ipc.nsfbufs="8192" > kern.ipc.nmbclusters="49152" > elessar@snake: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters > kern.ipc.nsfbufs: 0 > kern.ipc.nmbclusters: 32768 > elessar@snake: ~% netstat -m > 350/132/482/49152 mbuf clusters in use (current/cache/total/max) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 2864770 requests for I/O initiated by sendfile > > Box #2: > elessar@forseti: ~% uname -srm > FreeBSD 6.2-RC1 alpha > elessar@forseti: ~% grep kern.ipc /boot/loader.conf > kern.ipc.nsfbufs="4294967300" > elessar@forseti: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters > kern.ipc.nsfbufs: 0 > kern.ipc.nmbclusters: 1664 > elessar@forseti: ~% netstat -m > 64/80/144/1664 mbuf clusters in use (current/cache/total/max) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > > Box #3: > elessar@rivendell: ~% uname -srm > FreeBSD 6.1-RELEASE-p10 amd64 > elessar@rivendell: ~% grep kern.ipc /boot/loader.conf > kern.ipc.nsfbufs="8192" > kern.ipc.nmbclusters="49152" > elessar@rivendell: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters > kern.ipc.nsfbufs: 0 > kern.ipc.nmbclusters: 32768 > elessar@rivendell: ~% grep NSF /root/kernel/RIVENDELL > options NSFBUFS=8192 > elessar@rivendell: ~% netstat -m > 64/118/182/49152 mbuf clusters in use (current/cache/total/max) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 35 requests for I/O initiated by sendfile > > For comparison, Box #4: > elessar@web2: ~% uname -srm > FreeBSD 6.1-STABLE i386 > elessar@web2: ~% grep kern.ipc /boot/loader.conf > kern.ipc.nsfbufs="8192" > elessar@web2: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters > kern.ipc.nsfbufs: 8192 > kern.ipc.nmbclusters: 32768 > elessar@web2: ~% netstat -m > 104/272/376/32768 mbuf clusters in use (current/cache/total/max) > 1598/2635/8192 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 8938512 requests for I/O initiated by sendfile > > So, looking at the `netstat -m' output, setting kern.ipc.nmbclusters > in the loader.conf seems to work, but the sysctl reports the default > value that would be in place, if it wasn't overridden in the loader.conf. > > For the sendfile buffers, the stats are completely out of whack. > The sysctl reports 0 for every value (`netstat-m' just prints the sysctl > fields for these values if I understood the source right (unlikely). > But sendfile() works and the accounting on the number of sendfile calls > seems to work too. > So I believe the NSFBUFS kernel option and the loader.conf setting work, > but it is quite impossible to properly tune the buffers to the workload. > I did not bother trying the NMBCLUSTERS kernel option, it was removed > a while ago anyway. > > The loader.conf kern.ipc.nsfbufs value on box #2 is ridiculously high, > yes, I tried to get beyond the boundary of a u_int_32 to see if maybe > something good or bad happens. It did not. > > At this point, I wanted to include links to two mails on amd64@ I > found when I first searched for this oddity, both of which had no > reply. Instead I found a posting to net@ that occured in the meantime, > and a reply to it by Yar Tikhiy two days ago explaining the nsfbufs issue. > There are no such buffers on amd64, ia64 and alpha as they aren't needed. > > http://docs.freebsd.org/cgi/getmsg.cgi?fetch=26091+0+current/freebsd-net > > Ok, that leaves the question why `sysctl kern.ipc.nmbclusters' reports a > different value compared to the `netstat -m' output on box #1 and #3. netstat -m doesn't use the sysctl. It reads the zone limit via memstat_get_countlimit() from libmemstat(3) instead. Perhaps this can account for the difference observed. > Thanks for any pointers, > Joerg > > PS: documenting the nsfbufs issue on amd64/ia64/alpha might be a good > PS: idea. loader(8) seems a good place. Opinions? I've already documented the issue in sendfile(2) in CURRENT. MFC is due. A small note in loader(8) would be good, too -- thanks for the suggestion. -- Yar
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20061129052625.GG96853>