From owner-freebsd-fs@FreeBSD.ORG Tue Mar 2 10:25:24 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA1A8106566B; Tue, 2 Mar 2010 10:25:24 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id 8B7B78FC08; Tue, 2 Mar 2010 10:25:24 +0000 (UTC) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1NmPHy-0009jy-Dj; Tue, 02 Mar 2010 12:25:18 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Rick Macklem In-reply-to: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> Comments: In-reply-to Rick Macklem message dated "Mon, 01 Mar 2010 17:21:02 -0500." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 02 Mar 2010 12:25:17 +0200 From: Daniel Braniss Message-ID: Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org, Jeremy Chadwick Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Mar 2010 10:25:25 -0000 > > > On Sat, 27 Feb 2010, Jeremy Chadwick wrote: > > >> I concur. > >> Everything in my network is now on TCP, and there is no mbuf leakage. > >> I just don't get over the 5500 mark, no matter what I throw at it. > >> > >> I do feel that TCP is not as well performing on a local net with Linux, > >> hence the choice for UDP. But TCP is workable as next best. > > > > NFS; Rick Macklem would be a better choice, but as reported, he's MIA. > > > > Not exactly MIA, but only able to read email from time to time at this > point. I don't know when I'll be able to do more than that. > > So, it does sound like it is UDP specific. Robert mentioned one scenario, > which was an infrequently executed code path that is being tickled and it > has a missing m_freem(). > > One thing someone could try is switching to the experimental nfs server > ("-e" on both mountd and nfsd) and see if the leak goes away. If it does > go away, it is almost certainly the above in the regular nfs server code. > runing with the experimental nfs server all is ok! (at least I can't see any mbuf leakage :-) so now that we can assume that the problem is in NFS/UDP writes via classic nfsserver, where to look? > If it doesn't go away, the problem is more likely in the krpc or the > generic udp code. (When I looked at svc_dg.c, I could only spot one > possible leak and you've already determined that patch doesn't help. > The other big difference when using udp on the FreeBSD8 krpc is the > reply cache code. I seem to recall it's an lru cache with a fixed upper > bound, but it might be broken and leaking. > > If you change the server to set sp_rcache = NULL in the initialization > function in sys/nfsserver/nfs_srvkrpc.c, I think that disables the replay > cache. You wouldn't want to run this way in production, but it would > determine if the leak is in it. > > Change the 3 lines in nfsrv_init() to: > nfsrv_pool->sp_rcache = NULL; > nfsrv_pool->sp_assign = NULL; > nfsrv_pool->sp_done = NULL; > > and I think the krpc replay cache will be disabled. > > Good luck with it and please report back if you get to try the above. > > I'll get back to committing etc one of these days, rick just keep sending insights/pointers and enjoy life danny