From owner-freebsd-fs@FreeBSD.ORG Mon Mar 1 22:08:58 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DCB6B106566B; Mon, 1 Mar 2010 22:08:57 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 59BB08FC0A; Mon, 1 Mar 2010 22:08:57 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAFjJi0uDaFvK/2dsb2JhbACbB3O9RYR7BIMX X-IronPort-AV: E=Sophos;i="4.49,562,1262581200"; d="scan'208";a="67378000" Received: from fraser.cs.uoguelph.ca ([131.104.91.202]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 01 Mar 2010 17:08:33 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by fraser.cs.uoguelph.ca (Postfix) with ESMTP id 53400109C327; Mon, 1 Mar 2010 17:08:56 -0500 (EST) X-Virus-Scanned: amavisd-new at fraser.cs.uoguelph.ca Received: from fraser.cs.uoguelph.ca ([127.0.0.1]) by localhost (fraser.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kW9fRQgs545g; Mon, 1 Mar 2010 17:08:55 -0500 (EST) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by fraser.cs.uoguelph.ca (Postfix) with ESMTP id A4B31109C34B; Mon, 1 Mar 2010 17:08:55 -0500 (EST) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o21ML2Z27999; Mon, 1 Mar 2010 17:21:02 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Mon, 1 Mar 2010 17:21:02 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Jeremy Chadwick In-Reply-To: <20100227220310.GA65110@icarus.home.lan> Message-ID: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 22:08:58 -0000 On Sat, 27 Feb 2010, Jeremy Chadwick wrote: >> I concur. >> Everything in my network is now on TCP, and there is no mbuf leakage. >> I just don't get over the 5500 mark, no matter what I throw at it. >> >> I do feel that TCP is not as well performing on a local net with Linux, >> hence the choice for UDP. But TCP is workable as next best. > > NFS; Rick Macklem would be a better choice, but as reported, he's MIA. > Not exactly MIA, but only able to read email from time to time at this point. I don't know when I'll be able to do more than that. So, it does sound like it is UDP specific. Robert mentioned one scenario, which was an infrequently executed code path that is being tickled and it has a missing m_freem(). One thing someone could try is switching to the experimental nfs server ("-e" on both mountd and nfsd) and see if the leak goes away. If it does go away, it is almost certainly the above in the regular nfs server code. If it doesn't go away, the problem is more likely in the krpc or the generic udp code. (When I looked at svc_dg.c, I could only spot one possible leak and you've already determined that patch doesn't help. The other big difference when using udp on the FreeBSD8 krpc is the reply cache code. I seem to recall it's an lru cache with a fixed upper bound, but it might be broken and leaking. If you change the server to set sp_rcache = NULL in the initialization function in sys/nfsserver/nfs_srvkrpc.c, I think that disables the replay cache. You wouldn't want to run this way in production, but it would determine if the leak is in it. Change the 3 lines in nfsrv_init() to: nfsrv_pool->sp_rcache = NULL; nfsrv_pool->sp_assign = NULL; nfsrv_pool->sp_done = NULL; and I think the krpc replay cache will be disabled. Good luck with it and please report back if you get to try the above. I'll get back to committing etc one of these days, rick