From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 10:27:31 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 521F6106566C for ; Tue, 8 Feb 2011 10:27:31 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA11.westchester.pa.mail.comcast.net (qmta11.westchester.pa.mail.comcast.net [76.96.59.211]) by mx1.freebsd.org (Postfix) with ESMTP id F1E058FC16 for ; Tue, 8 Feb 2011 10:27:30 +0000 (UTC) Received: from omta08.westchester.pa.mail.comcast.net ([76.96.62.12]) by QMTA11.westchester.pa.mail.comcast.net with comcast id 5NSW1g0020Fqzac5BNTWqX; Tue, 08 Feb 2011 10:27:30 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta08.westchester.pa.mail.comcast.net with comcast id 5NTV1g00B2tehsa3UNTVCX; Tue, 08 Feb 2011 10:27:30 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id D225D9B422; Tue, 8 Feb 2011 02:27:27 -0800 (PST) Date: Tue, 8 Feb 2011 02:27:27 -0800 From: Jeremy Chadwick To: Bartosz Stec Message-ID: <20110208102727.GA8555@icarus.home.lan> References: <4D510BBB.1060708@kkip.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D510BBB.1060708@kkip.pl> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, pjd@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 10:27:31 -0000 On Tue, Feb 08, 2011 at 10:24:11AM +0100, Bartosz Stec wrote: > W dniu 2011-02-07 22:37, Emil Muratov pisze: > > > >>For the past few weeks, I noticed that the amount of memory > >>reported in top > >>(sum of active, inact, wired, cache buf and free) keeps > >>decreasing as the > >>uptime increases. I can't pinpoint to when I first noticed this, > >>as I have > >>updated the system a few times just in case this has been fixed. > > > >Yes, I have the same issue on my home file storage. My system is > >8.1 amd64, 2G ram, zfs on root raidz with 4x1,5T drives. > >After updating to stable a couple of days ago I noticed that the > >system leaks memory very fast. Checking here and there I found > >that the issue concerns sendfile (yep, again!). > > > >How to reproduce: > >Configure samba with aio and sendfile (mine is version 3.5.6) > > > >smb.conf > >[global] > >use sendfile=true > >aio read size = 16384 > > > >Download a couple of large samba shared files (8-10 gigs). > > > > > >While downloading files I can see that memory decreazes to nowhere > >very-very fast, several MBs per second! First it drains free mem, > >than active and inactive, than comes wired until the whole system > >commits suicide suffocating itself to the death. > >The only way to free memory is to reboot the system. I can't > >unload zfs module like PJD suggested to do, 'cause my root is on > >zfs :( > >I'll try to make a bootable flash and move root to the flash to > >try to unload module and what will happen. > > > >Everything was OK in stable before the new year, sendfile used to > >pump free and wired memory to inactive than slowly reclaiming it > >back. But it seems something was changed after NY holydays? > > I'm glad someone else finally picked that problem, so there's > appareantly no memory-eating ghost in my machine ;) > Here's my thread on stable list about this issue: > http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.html > > And in fact, PC reported in thread above is also SAMBA server with > aio/sendfile enabled and ZFS. > > I would be happy testing some patches if necessary, because until > now I need to monitor memory and reboot this server before it dies. The source and build date of your kernel will matter greatly here. I can't speak about the memory utilisation aspect, but I tend to disable sendfile everywhere possible when ZFS is in use on a system. The reason is based on something I and another user experienced back in October 2010 pertaining to sendfile() on ZFS locking up processes (making them unkillable). See here[1] for details; this problem has since been fixed[2] (look for commits around October). You'll also find some commits that went through in November pertaining to ZFS and sendfile. This is why I said the date of your kernel/sources matters. :-) The issue I referenced in [1] is not related to memory utilisation, but does indicate use of sendfile with ZFS may be a bad idea (by this I mean, there may be aspects of its implementation when mixed with ZFS that have been overlooked). Simple test: if you disable use of sendfile (but not AIO) in Samba, does the problem go away? Comparatively, Apache out-of-the-box defaults to mmap() and sendfile() being disabled. BSD ftpd, however, is known to use sendfile exclusively. [1]: http://lists.freebsd.org/pipermail/freebsd-fs/2010-October/thread.html#9729 [2]: http://www.freshbsd.org/?branch=RELENG_8&project=freebsd&committer=&module=src&q=sendfile -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |