Date: Tue, 02 Mar 2010 12:25:17 +0200 From: Daniel Braniss <danny@cs.huji.ac.il> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen <wjw@digiware.nl>, =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= <ltning@anduin.net>, rwatson@freebsd.org, Jeremy Chadwick <freebsd@jdc.parodius.com> Subject: Re: mbuf leakage with nfs/zfs? Message-ID: <E1NmPHy-0009jy-Dj@kabab.cs.huji.ac.il> In-Reply-To: <Pine.GSO.4.63.1003011703100.26054@muncher.cs.uoguelph.ca> References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <E1Nl6VA-000557-D9@kabab.cs.huji.ac.il> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <BD8AC9F6-DF96-41F9-8E92-48A4E5606DC7@anduin.net> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> <Pine.GSO.4.63.1003011703100.26054@muncher.cs.uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
>
>
> On Sat, 27 Feb 2010, Jeremy Chadwick wrote:
>
> >> I concur.
> >> Everything in my network is now on TCP, and there is no mbuf leakage.
> >> I just don't get over the 5500 mark, no matter what I throw at it.
> >>
> >> I do feel that TCP is not as well performing on a local net with Linux,
> >> hence the choice for UDP. But TCP is workable as next best.
> >
> > NFS; Rick Macklem would be a better choice, but as reported, he's MIA.
> >
>
> Not exactly MIA, but only able to read email from time to time at this
> point. I don't know when I'll be able to do more than that.
>
> So, it does sound like it is UDP specific. Robert mentioned one scenario,
> which was an infrequently executed code path that is being tickled and it
> has a missing m_freem().
>
> One thing someone could try is switching to the experimental nfs server
> ("-e" on both mountd and nfsd) and see if the leak goes away. If it does
> go away, it is almost certainly the above in the regular nfs server code.
>
runing with the experimental nfs server all is ok!
(at least I can't see any mbuf leakage :-)
so now that we can assume that the problem is in NFS/UDP writes via
classic nfsserver, where to look?
> If it doesn't go away, the problem is more likely in the krpc or the
> generic udp code. (When I looked at svc_dg.c, I could only spot one
> possible leak and you've already determined that patch doesn't help.
> The other big difference when using udp on the FreeBSD8 krpc is the
> reply cache code. I seem to recall it's an lru cache with a fixed upper
> bound, but it might be broken and leaking.
>
> If you change the server to set sp_rcache = NULL in the initialization
> function in sys/nfsserver/nfs_srvkrpc.c, I think that disables the replay
> cache. You wouldn't want to run this way in production, but it would
> determine if the leak is in it.
>
> Change the 3 lines in nfsrv_init() to:
> nfsrv_pool->sp_rcache = NULL;
> nfsrv_pool->sp_assign = NULL;
> nfsrv_pool->sp_done = NULL;
>
> and I think the krpc replay cache will be disabled.
>
> Good luck with it and please report back if you get to try the above.
>
> I'll get back to committing etc one of these days, rick
just keep sending insights/pointers and enjoy life
danny
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E1NmPHy-0009jy-Dj>