From owner-freebsd-fs@FreeBSD.ORG Sun Feb 28 12:21:32 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B02151065672; Sun, 28 Feb 2010 12:21:32 +0000 (UTC) (envelope-from rwatson@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 6E2758FC14; Sun, 28 Feb 2010 12:21:32 +0000 (UTC) Received: from [192.168.2.102] (host86-181-230-35.range86-181.btcentralplus.com [86.181.230.35]) by cyrus.watson.org (Postfix) with ESMTPSA id 9F06646B17; Sun, 28 Feb 2010 07:21:30 -0500 (EST) Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=us-ascii From: "Robert N. M. Watson" In-Reply-To: Date: Sun, 28 Feb 2010 12:21:28 +0000 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> To: Daniel Braniss X-Mailer: Apple Mail (2.1077) Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?iso-8859-1?Q?Eirik_=D8verby?= , Jeremy Chadwick Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 12:21:32 -0000 On Feb 28, 2010, at 12:11 PM, Daniel Braniss wrote: >> I'm pulling in Robert Watson, who has some familiarity with the UDP >> stack/code in FreeBSD. I'm not sure he'll be a sufficient source of >> knowledge for this specific issue since it appears (?) to be specific = to >> NFS; Rick Macklem would be a better choice, but as reported, he's = MIA. >>=20 >> Robert, are you aware of any changes or implementation issues which >> might cause excessive (read: leaking) mbuf use under UDP-based NFS? = Do >> you know of a way folks could determine the source of the leak, = either >> via DDB or while the system is live? >=20 > I have been runing some tests in a controlled environment. >=20 > server and client are both 64bit Xeon/X5550 @ 2.67GHz with 16Gb of = memory > FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads >=20 > the client is runing latest 8.0 stable > the load is created by runing 'make -j32 buildworld' and sleeping 150 = sec. > in between runs, this is the straight line you will see in the graphs. > Both the src and obj directories are NFS mounted from the server, = regular UFS. >=20 > when server is running 7.2-stable no leakage is seen. > see ftp://ftp.cs.huji.ac.il/users/danny/freebsd/mbufs/{tcp,udp}-7.2.ps > when server is runing 8.0-stable > see ftp://ftp.cs.huji.ac.il/users/danny/freebsd/mbufs/{tcp,udp}-8.0.ps > you can see that udp is leaking! >=20 > cheers, > danny > ps: I think the subject should be changed again, removing zfs ... This type of problem (occurs with one client but not another) is almost = always the result of the access pattern of a particular client = triggering a specific (and perhaps single) bug in error-handling. For = example, we might not be properly freeing the received request when = generating an EPERM in an edge case. The hard bit is identifying which = it is. If it's reproducible with UDP, then usually the process is: - Build a minimal test case to trigger the problem -- ideally with as = little complexity as possible. - Run netstat -m at the beginning of the test and the end of the test on = the server to count the number of leaked mbufs - Run wireshark throughout the test - Walk the wireshark trace looking for some error that occurs at about = the same or slightly lower number of times then the number of mbufs = leaked - Iterate, narrowing the test case until it's either obvious exactly = what's going on, or you've identified a relatively constrained code path = and can just spot the bug by reading the code It's almost certainly one or a small number of very specific RPCs that = are triggering it -- maybe OpenBSD does an extra lookup, or stat, or = something, on a name that may not exist anymore, or does it sooner than = the other clients. Hard to say, other than to wave hands at the = possibilities. And it may well be we're looking at two bugs: Danny may see one bug, = perhaps triggered by a race condition, but it may be different from the = OpenBSD client-triggered bug (to be clear: it's definitely a FreeBSD = bug, although we might only see it when an OpenBSD client is used = because perhaps OpenBSD also has a bug or feature). Robert=