From owner-freebsd-fs@FreeBSD.ORG Sun Feb 28 14:52:47 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C96A8106564A; Sun, 28 Feb 2010 14:52:47 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id 736698FC14; Sun, 28 Feb 2010 14:52:47 +0000 (UTC) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1NlkVg-00068M-Qk; Sun, 28 Feb 2010 16:52:44 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: "Robert N. M. Watson" In-reply-to: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> Comments: In-reply-to "Robert N. M. Watson" message dated "Sun, 28 Feb 2010 12:21:28 +0000." Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Date: Sun, 28 Feb 2010 16:52:44 +0200 From: Daniel Braniss Message-ID: Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?iso-8859-1?Q?Eirik_=D8verby?= , Jeremy Chadwick Subject: Re: mbuf leakage with nfs/udp (was: mbuf leakage with nfs/zfs) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 14:52:47 -0000 > > On Feb 28, 2010, at 12:11 PM, Daniel Braniss wrote: >=20 > >> I'm pulling in Robert Watson, who has some familiarity with the UDP > >> stack/code in FreeBSD. I'm not sure he'll be a sufficient source of= > >> knowledge for this specific issue since it appears (?) to be specifi= c =3Dto > >> NFS; Rick Macklem would be a better choice, but as reported, he's = =3DMIA. > >>=20 > >> Robert, are you aware of any changes or implementation issues which > >> might cause excessive (read: leaking) mbuf use under UDP-based NFS? = =3DDo > >> you know of a way folks could determine the source of the leak, =3De= ither > >> via DDB or while the system is live? > >=20 > > I have been runing some tests in a controlled environment. > > > > server and client are both 64bit Xeon/X5550 =40 2.67GHz with 16G= b of =3Dmemory > > FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads > > > > the client is runing latest 8.0 stable > > the load is created by runing 'make -j32 buildworld' and sleeping 150= =3Dsec. > > in between runs, this is the straight line you will see in the graphs= . > > Both the src and obj directories are NFS mounted from the server, =3D= regular UFS. > > > > when server is running 7.2-stable no leakage is seen. > > see ftp://ftp.cs.huji.ac.il/users/danny/freebsd/mbufs/=7Btcp,udp=7D-7= .2.ps > > when server is runing 8.0-stable > > see ftp://ftp.cs.huji.ac.il/users/danny/freebsd/mbufs/=7Btcp,udp=7D-8= .0.ps > > you can see that udp is leaking=21 > > > > cheers, > > danny > > ps: I think the subject should be changed again, removing zfs ... > > This type of problem (occurs with one client but not another) is almo= st =3Dalways the result of the access pattern of a particular client =3Dt= riggering a specific (and perhaps single) bug in error-handling. For =3De= xample, we might not be properly freeing the received request when =3Dgen= erating an EPERM in an edge case. The hard bit is identifying which =3Dit= is. If it's reproducible with UDP, then usually the process is: > > - Build a minimal test case to trigger the problem -- ideally with as= =3Dlittle complexity as possible. > - Run netstat -m at the beginning of the test and the end of the test o= n =3Dthe server to count the number of leaked mbufs > - Run wireshark throughout the test > - Walk the wireshark trace looking for some error that occurs at about = =3Dthe same or slightly lower number of times then the number of mbufs = =3Dleaked > - Iterate, narrowing the test case until it's either obvious exactly = =3Dwhat's going on, or you've identified a relatively constrained code pa= th =3Dand can just spot the bug by reading the code > > It's almost certainly one or a small number of very specific RPCs tha= t =3Dare triggering it -- maybe OpenBSD does an extra lookup, or stat, or= =3Dsomething, on a name that may not exist anymore, or does it sooner th= an =3Dthe other clients. Hard to say, other than to wave hands at the =3D= possibilities. > > And it may well be we're looking at two bugs: Danny may see one bug, = =3Dperhaps triggered by a race condition, but it may be different from th= e =3DOpenBSD client-triggered bug (to be clear: it's definitely a FreeBSD= =3Dbug, although we might only see it when an OpenBSD client is used =3D= because perhaps OpenBSD also has a bug or feature). > > Robert=3D well, I have further reduced the problem, it happens with NFS/UDP writes.= i'll try the wireshark road, but i'm very rusty with RPC, the other road = is to check the changes, my oldest is from late october (RC2) where it's happen= ing,=20 while Gerrit tried 8-pre from November and worked, so it will be fun trying to nail it down :-) cheers, danny