From owner-freebsd-stable@FreeBSD.ORG Sun Apr 29 20:48:38 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5FDA4106564A; Sun, 29 Apr 2012 20:48:38 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id F143D8FC08; Sun, 29 Apr 2012 20:48:37 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEAC6onU+DaFvO/2dsb2JhbAA8CA4IhVKtOIIJAQEBAwEBAQEgKyALBQcPDgcDAgINGQIpAQkmBggCBQQBHASHZwULpn6SBIEviVoDhH6BGASTT4IvgRGPMYIvVYFA X-IronPort-AV: E=Sophos;i="4.75,501,1330923600"; d="scan'208";a="167145853" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 29 Apr 2012 16:48:36 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id EC3D3B40ED; Sun, 29 Apr 2012 16:48:36 -0400 (EDT) Date: Sun, 29 Apr 2012 16:48:36 -0400 (EDT) From: Rick Macklem To: Daniel Braniss Message-ID: <99719742.103996.1335732516952.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: killing@multiplay.co.uk, Hiroki Sato , freebsd-stable@FreeBSD.org, ob@e-Gitt.NET Subject: Re: 9-STABLE, ZFS, NFS, ggatec - suspected memory leak X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Apr 2012 20:48:38 -0000 Daniel Braniss wrote: > > Daniel Braniss wrote: > > > > Daniel Braniss wrote: > > > > > > ----Security_Multipart(Fri_Apr_27_13_35_56_2012_748)-- > > > > > > Content-Type: Text/Plain; charset=us-ascii > > > > > > Content-Transfer-Encoding: 7bit > > > > > > > > > > > > Rick Macklem wrote > > > > > > in > > > > > > <1527622626.3418715.1335445225510.JavaMail.root@erie.cs.uoguelph.ca>: > > > > > > > > > > > > rm> Steven Hartland wrote: > > > > > > rm> > ---- Original Message ----- > > > > > > rm> > From: "Rick Macklem" > > > > > > rm> > > At a glance, it looks to me like 8.x is affected. > > > > > > Note > > > > > > that > > > > > > the > > > > > > rm> > > bug only affects the new NFS server (the > > > > > > experimental > > > > > > one > > > > > > for 8.x) > > > > > > rm> > > when exporting ZFS volumes. (UFS exported volumes > > > > > > don't > > > > > > leak) > > > > > > rm> > > > > > > > > rm> > > If you are running a server that might be affected, > > > > > > just: > > > > > > rm> > > # vmstat -z | fgrep -i namei > > > > > > rm> > > on the server and see if the 3rd number shown is > > > > > > increasing. > > > > > > rm> > > > > > > > rm> > Many thanks Rick wasnt aware we had anything > > > > > > experimental > > > > > > enabled > > > > > > rm> > but I think that would be a yes looking at these > > > > > > number:- > > > > > > rm> > > > > > > > rm> > vmstat -z | fgrep -i namei > > > > > > rm> > NAMEI: 1024, 0, 1, 1483, 25285086096, 0 > > > > > > rm> > vmstat -z | fgrep -i namei > > > > > > rm> > NAMEI: 1024, 0, 0, 1484, 25285945725, 0 > > > > > > rm> > > > > > > > rm> ^ > > > > > > rm> I don't think so, since the 3rd number (USED) is 0 here. > > > > > > rm> If that # is increasing over time, you have the leak. > > > > > > You > > > > > > are > > > > > > rm> probably running the old (default in 8.x) NFS server. > > > > > > > > > > > > Just a report, I confirmed it affected 8.x servers running > > > > > > newnfs. > > > > > > > > > > > > Actually I have been suffered from memory starvation > > > > > > symptom on > > > > > > that > > > > > > server (24GB RAM) for a long time and watching vmstat -z > > > > > > periodically. It stopped working once a week. I > > > > > > investigated > > > > > > the > > > > > > vmstat log again and found the amount of NAMEI leak was > > > > > > 11,543,956 > > > > > > (about 11GB!) just before the locked-up. After applying the > > > > > > patch, > > > > > > the leak disappeared. Thank you for fixing it! > > > > > > > > > > > > -- Hiroki > > > > And thanks Hiroki for testing it on 8.x. > > > > > > > > > this is on 8.2-STABLE/amd64 from around August: > > > > > same here, this zfs+newnfs has been hanging every few months, > > > > > and > > > > > I > > > > > can see > > > > > now the leak, it's slowly increasing: > > > > > NAMEI: 1024, 0, 122975, 529, 15417248, 0 > > > > > NAMEI: 1024, 0, 122984, 520, 15421772, 0 > > > > > NAMEI: 1024, 0, 123002, 502, 15424743, 0 > > > > > NAMEI: 1024, 0, 123008, 496, 15425464, 0 > > > > > > > > > > cheers, > > > > > danny > > > > Maybe you could try the patch, too. > > > > > > > > It's at: > > > > http://people.freebsd.org/~rmacklem/namei-leak.patch > > > > > > > > I'll commit it to head soon with a 1 month MFC, so that > > > > hopefully > > > > Oliver will have a chance to try it on his production server > > > > before > > > > the MFC. > > > > > > > > Thanks everyone, for your help with this, rick > > > > > > I haven't applied the patch yet, but in the meanime I have been > > > running some > > > experiments on a zfs/nfs server running 8.3-STABLE, and don't see > > > any > > > leaks > > > what triggers the leak? > > > > > Fortunately Oliver isolated this. It should leak when you do a > > successful > > "rm" or "rmdir" while running the new/experimental server. > > > but that's what I did, I'm running the new/experimental nfs server > (or so I think :-), and did a huge rm -rf and nothing, nada, no leak. > To check the patch, I have to upgrade the production server, the one > with the > leak, > but I wanted to test it on a non production first. Anyways, ill patch > the > kernel > and try it on the leaking production server tomorrow. > Well, I think the patch should be harmless. You can check which server you are running by doing: # nfsstat -e -s - and see if the numbers are increasing if they're zero or not increasing, you are running the old (default on 8.x) server rick > danny > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org"