From owner-freebsd-fs@FreeBSD.ORG Fri Feb 22 02:47:26 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 0A192F3C for ; Fri, 22 Feb 2013 02:47:26 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id C3601EBB for ; Fri, 22 Feb 2013 02:47:25 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id r1M2lHQu010153; Thu, 21 Feb 2013 20:47:17 -0600 (CST) Date: Thu, 21 Feb 2013 20:47:17 -0600 (CST) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Kevin Day Subject: Re: Improving ZFS performance for large directories In-Reply-To: <19E0C908-79F1-43F8-899C-6B60F998D4A5@dragondata.com> Message-ID: References: <19DB8F4A-6788-44F6-9A2C-E01DEA01BED9@dragondata.com> <20130201192416.GA76461@server.rulingia.com> <19E0C908-79F1-43F8-899C-6B60F998D4A5@dragondata.com> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Thu, 21 Feb 2013 20:47:17 -0600 (CST) Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Feb 2013 02:47:26 -0000 On Tue, 19 Feb 2013, Kevin Day wrote: > Sorry for the late followup, I've been doing some testing with an L2ARC device. > >>> Doing it twice back-to-back makes a bit of difference but it's still slow either way. >> >> ZFS can very conservative about caching data and twice might not be enough. >> I suggest you try 8-10 times, or until the time stops reducing. > > Timing doing an "ls" in large directories 20 times, the first is the slowest, then all subsequent listings are roughly the same. There doesn't appear to be any gain after 20 repetitions You might consider that the bottleneck might be in 'ls' or something outside of zfs. Make sure that you are doing 'ls -f' or else you are just measuring its sorting performance. On a Solaris 10 system in a zfs directory with a million files: % time ls -f |wc -l 1000002 /bin/ls -F -f 0.76s user 0.93s system 89% cpu 1.897 total wc -l 0.08s user 0.02s system 5% cpu 1.697 total % time ls |wc -l 1000000 /bin/ls -F 4.32s user 8.10s system 97% cpu 12.682 total wc -l 0.08s user 0.02s system 0% cpu 12.432 total Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/