From owner-freebsd-fs@FreeBSD.ORG Wed Mar 3 00:40:34 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ECAF5106564A; Wed, 3 Mar 2010 00:40:33 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 651BF8FC14; Wed, 3 Mar 2010 00:40:33 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEADA/jUuDaFvK/2dsb2JhbACbDHO+PYR7BIMXix8 X-IronPort-AV: E=Sophos;i="4.49,570,1262581200"; d="scan'208";a="67549251" Received: from fraser.cs.uoguelph.ca ([131.104.91.202]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 02 Mar 2010 19:40:32 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by fraser.cs.uoguelph.ca (Postfix) with ESMTP id 3D3BF109C2BF; Tue, 2 Mar 2010 19:40:32 -0500 (EST) X-Virus-Scanned: amavisd-new at fraser.cs.uoguelph.ca Received: from fraser.cs.uoguelph.ca ([127.0.0.1]) by localhost (fraser.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id x0jwkDDWoWZl; Tue, 2 Mar 2010 19:40:31 -0500 (EST) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by fraser.cs.uoguelph.ca (Postfix) with ESMTP id AC603109C285; Tue, 2 Mar 2010 19:40:31 -0500 (EST) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o230qes05191; Tue, 2 Mar 2010 19:52:40 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Tue, 2 Mar 2010 19:52:40 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Daniel Braniss In-Reply-To: Message-ID: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org, Jeremy Chadwick Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Mar 2010 00:40:34 -0000 On Tue, 2 Mar 2010, Daniel Braniss wrote: > runing with the experimental nfs server all is ok! > (at least I can't see any mbuf leakage :-) > > so now that we can assume that the problem is in NFS/UDP writes via > classic nfsserver, where to look? > It might also be the krpc reply cache, since the experimental server isn't using it (nfsv4 requires a rather twisted reply cache and it was easier to just use that one for nfsv2,3 for the experimental server, as well). >> If it doesn't go away, the problem is more likely in the krpc or the >> generic udp code. (When I looked at svc_dg.c, I could only spot one >> possible leak and you've already determined that patch doesn't help. >> The other big difference when using udp on the FreeBSD8 krpc is the >> reply cache code. I seem to recall it's an lru cache with a fixed upper >> bound, but it might be broken and leaking. >> >> If you change the server to set sp_rcache = NULL in the initialization >> function in sys/nfsserver/nfs_srvkrpc.c, I think that disables the replay >> cache. You wouldn't want to run this way in production, but it would >> determine if the leak is in it. >> >> Change the 3 lines in nfsrv_init() to: >> nfsrv_pool->sp_rcache = NULL; >> nfsrv_pool->sp_assign = NULL; >> nfsrv_pool->sp_done = NULL; >> >> and I think the krpc replay cache will be disabled. >> If someone gets a chance to try the above (not in production mode:-), it will determine if the problem is in the reply cache or the nfs server's write code. >> Good luck with it and please report back if you get to try the above. >> Thanks for trying the experimental server. It is getting narrowed down, due to everyone's work on it. rick