From owner-freebsd-fs@FreeBSD.ORG Thu Feb 17 01:10:40 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 54AA21065673 for ; Thu, 17 Feb 2011 01:10:40 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta09.westchester.pa.mail.comcast.net (qmta09.westchester.pa.mail.comcast.net [76.96.62.96]) by mx1.freebsd.org (Postfix) with ESMTP id 00C0E8FC1A for ; Thu, 17 Feb 2011 01:10:35 +0000 (UTC) Received: from omta10.westchester.pa.mail.comcast.net ([76.96.62.28]) by qmta09.westchester.pa.mail.comcast.net with comcast id 8p5S1g0040cZkys59pAcgo; Thu, 17 Feb 2011 01:10:36 +0000 Received: from koitsu.dyndns.org ([98.248.33.18]) by omta10.westchester.pa.mail.comcast.net with comcast id 8pAZ1g0140PUQVN3WpAamX; Thu, 17 Feb 2011 01:10:36 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 4F6059B422; Wed, 16 Feb 2011 17:10:32 -0800 (PST) Date: Wed, 16 Feb 2011 17:10:32 -0800 From: Jeremy Chadwick To: Ivan Voras Message-ID: <20110217011032.GA15027@icarus.home.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: zfs directory listing X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Feb 2011 01:10:40 -0000 On Thu, Feb 17, 2011 at 01:55:46AM +0100, Ivan Voras wrote: > On 16/02/2011 23:52, Andrew Thompson wrote: > >Hi, > > > > > >I have a zfs file system on 8.1-RELEASE amd64 which as a large number > >of files in /var/spool/mqueue. loader.conf has vfs.zfs.arc_max=2G for > >an 8G box. > > >mqueue has a link count of 522824. I can not list the contents of this > >directory, when I do the number of read IOPS sits> 100 and it will > >never complete > > Your problem is probably not FreeBSD-specific and possibly not even > ZFS-specific. That is a fairly large number of files in a directory > for any file system; The rule of thumb is usually to start sharding > as soon as the number of files gets even two orders of magnitude > lower than what you have there. > > As others said, try "cd /var/spool/mqueue && find ." - the find > utility just reads the directory, it doesn't try to gather other > metadata which "ls" uses and is usually one of the rare utilities > which can work with gigantic directories (usually in the form "find > . -delete" :) ). > > The reason for this is that filenames and file metadata are separate > objects on the drives and the drives need to seek between them to > get both if they are not cached. Simple version, to the OP: your mail server has an immense number of queued mails in it. You need to find out why and put an end to it. I also recommend you stop sendmail, "rm -r mqueue ; chown root:daemon mqueue ; chmod 755 mqueue", then start sendmail again. I'll also point out that things like ls -l and related methods take a significantly longer amount of time than doing something like "echo *" given the number of stat() calls that have to be made. Regarding the focus on ZFS with regards to this situation: you may want to look at vfs.numvnodes (sysctl) and if it's reaching kern.maxvnodes, increase kern.maxvnodes (via sysctl.conf). Be aware more vnodes means more memory usage, and this is separate/unrelated to/from ARC memory. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |