From owner-freebsd-fs@FreeBSD.ORG Wed Jan 30 10:20:05 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9BC97CB6 for ; Wed, 30 Jan 2013 10:20:05 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.78]) by mx1.freebsd.org (Postfix) with ESMTP id 3739CD31 for ; Wed, 30 Jan 2013 10:20:04 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.69) (envelope-from ) id 1U0Ulu-0001bp-Hx for freebsd-fs@freebsd.org; Wed, 30 Jan 2013 11:20:03 +0100 Received: from [81.21.138.17] (helo=ronaldradial.versatec.local) by smtp.greenhost.nl with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1U0Ulu-0007jg-Ew for freebsd-fs@freebsd.org; Wed, 30 Jan 2013 11:20:02 +0100 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: freebsd-fs@freebsd.org Subject: Re: Improving ZFS performance for large directories References: <19DB8F4A-6788-44F6-9A2C-E01DEA01BED9@dragondata.com> Date: Wed, 30 Jan 2013 11:20:02 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: <19DB8F4A-6788-44F6-9A2C-E01DEA01BED9@dragondata.com> User-Agent: Opera Mail/12.13 (Win32) X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: 0.8 X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.1 X-Scan-Signature: a9e4b997d6a751f3e45cb47a3c2b1d2c X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2013 10:20:05 -0000 On Wed, 30 Jan 2013 00:20:15 +0100, Kevin Day wrote: > > I'm trying to improve performance when using ZFS in large (>60000 files) > directories. A common activity is to use "getdirentries" to enumerate > all the files in the directory, then "lstat" on each one to get > information about it. Doing an "ls -l" in a large directory like this > can take 10-30 seconds to complete. Trying to figure out why, I did: > > ktrace ls -l /path/to/large/directory > kdump -R |sort -rn |more Does ls -lf /pat/to/large/directory make a difference. It makes ls not to sort the directory so it can use a more efficient way of traversing the directory. Ronald. > > to see what sys calls were taking the most time, I ended up with: > > 69247 ls 0.190729 STRU struct stat {dev=846475008, ino=46220085, > mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295, > atime=1333196714, stime=1201004393, ctime=1333196714.547566024, > birthtime=1333196714.547566024, size=30784, blksize=31232, blocks=62, > flags=0x0 } > 69247 ls 0.180121 STRU struct stat {dev=846475008, ino=46233417, > mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295, > atime=1333197088, stime=1209814737, ctime=1333197088.913571042, > birthtime=1333197088.913571042, size=3162220, blksize=131072, > blocks=6409, flags=0x0 } > 69247 ls 0.152370 RET getdirentries 4088/0xff8 > 69247 ls 0.139939 CALL stat(0x800d8f598,0x7fffffffcca0) > 69247 ls 0.130411 RET __acl_get_link 0 > 69247 ls 0.121602 RET __acl_get_link 0 > 69247 ls 0.105799 RET getdirentries 4064/0xfe0 > 69247 ls 0.105069 RET getdirentries 4068/0xfe4 > 69247 ls 0.096862 RET getdirentries 4028/0xfbc > 69247 ls 0.085012 RET getdirentries 4088/0xff8 > 69247 ls 0.082722 STRU struct stat {dev=846475008, ino=72941319, > mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295, > atime=1348686155, stime=1348347621, ctime=1348686155.768875422, > birthtime=1348686155.768875422, size=6686225, blksize=131072, > blocks=13325, flags=0x0 } > 69247 ls 0.070318 STRU struct stat {dev=846475008, ino=46211679, > mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295, > atime=1333196475, stime=1240230314, ctime=1333196475.038567672, > birthtime=1333196475.038567672, size=829895, blksize=131072, > blocks=1797, flags=0x0 } > 69247 ls 0.068060 RET getdirentries 4048/0xfd0 > 69247 ls 0.065118 RET getdirentries 4088/0xff8 > 69247 ls 0.062536 RET getdirentries 4096/0x1000 > 69247 ls 0.061118 RET getdirentries 4020/0xfb4 > 69247 ls 0.055038 STRU struct stat {dev=846475008, ino=46220358, > mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295, > atime=1333196720, stime=1274282669, ctime=1333196720.972567345, > birthtime=1333196720.972567345, size=382344, blksize=131072, blocks=773, > flags=0x0 } > 69247 ls 0.054948 STRU struct stat {dev=846475008, ino=75025952, > mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295, > atime=1351071350, stime=1349726805, ctime=1351071350.800873870, > birthtime=1351071350.800873870, size=2575559, blksize=131072, > blocks=5127, flags=0x0 } > 69247 ls 0.054828 STRU struct stat {dev=846475008, ino=65021883, > mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295, > atime=1335730367, stime=1332843230, ctime=1335730367.541567371, > birthtime=1335730367.541567371, size=226347, blksize=131072, blocks=517, > flags=0x0 } > 69247 ls 0.053743 STRU struct stat {dev=846475008, ino=46222016, > mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295, > atime=1333196765, stime=1257110706, ctime=1333196765.206574132, > birthtime=1333196765.206574132, size=62112, blksize=62464, blocks=123, > flags=0x0 } > 69247 ls 0.052015 RET getdirentries 4060/0xfdc > 69247 ls 0.051388 RET getdirentries 4068/0xfe4 > 69247 ls 0.049875 RET getdirentries 4088/0xff8 > 69247 ls 0.049156 RET getdirentries 4032/0xfc0 > 69247 ls 0.048609 RET getdirentries 4040/0xfc8 > 69247 ls 0.048279 RET getdirentries 4032/0xfc0 > 69247 ls 0.048062 RET getdirentries 4064/0xfe0 > 69247 ls 0.047577 RET getdirentries 4076/0xfec > (snip) > > the STRU are returns from calling lstat(). > > It looks like both getdirentries and lstat are taking quite a while to > return. The shortest return for any lstat() call is 0.000004 seconds, > the maximum is 0.190729 and the average is around 0.0004. Just from > lstat() alone, that makes "ls" take over 20 seconds. > > I'm prepared to try an L2arc cache device (with > secondarycache=metadata), but I'm having trouble determining how big of > a device I'd need. We've got >30M inodes now on this filesystem, > including some files with extremely long names. Is there some way to > determine the amount of metadata on a ZFS filesystem? > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"