From owner-freebsd-fs@FreeBSD.ORG Wed Jan 30 10:36:42 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4900D351 for ; Wed, 30 Jan 2013 10:36:42 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-bk0-f50.google.com (mail-bk0-f50.google.com [209.85.214.50]) by mx1.freebsd.org (Postfix) with ESMTP id BEC36DEE for ; Wed, 30 Jan 2013 10:36:41 +0000 (UTC) Received: by mail-bk0-f50.google.com with SMTP id jg9so742690bkc.37 for ; Wed, 30 Jan 2013 02:36:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:subject:mime-version:content-type:from:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer; bh=XwHHaBNRncPTtC58MSyZcTj2pLTLfUFOxLhKoPVoOyk=; b=fiJn30P9vkLKlpzRL2/tToD0BPwYJcAb0HbHvf9eQO4E3IgA/svj5KZcqzDixPUdit sozWw/Iz/IGkeLTBps/SR2CQQXVqZactSDQhte7thzxuu7S9+uIFE8m+5VylGm7xEQJf 0jKamcJ6GQGm+dINt/LMFvqpzMZcNZxO8xgA6Qal2OyAZ9fp0xem6G/yHcIX4ueoeuPq 9VZzcqCwbXj7q2gKafOIDdZanufwrbidtW+MwCp5KUGRZ/49AN5A7msKqjLkb/P+q6Q0 kzQj0KIa0O8i3YNHEc9imX2CxIZ+pFGZOWsFwmda5hzQ+AQODLjQeipFHR3XN9ojlGSJ Cj6A== X-Received: by 10.204.12.206 with SMTP id y14mr1081502bky.132.1359542194602; Wed, 30 Jan 2013 02:36:34 -0800 (PST) Received: from ndenevsa.sf.moneybookers.net (g1.moneybookers.com. [217.18.249.148]) by mx.google.com with ESMTPS id z5sm383371bkv.11.2013.01.30.02.36.33 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 30 Jan 2013 02:36:34 -0800 (PST) Subject: Re: Improving ZFS performance for large directories Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Content-Type: text/plain; charset=us-ascii From: Nikolay Denev In-Reply-To: <19DB8F4A-6788-44F6-9A2C-E01DEA01BED9@dragondata.com> Date: Wed, 30 Jan 2013 12:36:35 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <5267B97C-ED47-4AAB-8415-12D6987E9371@gmail.com> References: <19DB8F4A-6788-44F6-9A2C-E01DEA01BED9@dragondata.com> To: Kevin Day X-Mailer: Apple Mail (2.1499) Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2013 10:36:42 -0000 On Jan 30, 2013, at 1:20 AM, Kevin Day wrote: >=20 > I'm trying to improve performance when using ZFS in large (>60000 = files) directories. A common activity is to use "getdirentries" to = enumerate all the files in the directory, then "lstat" on each one to = get information about it. Doing an "ls -l" in a large directory like = this can take 10-30 seconds to complete. Trying to figure out why, I = did: >=20 > ktrace ls -l /path/to/large/directory > kdump -R |sort -rn |more >=20 > to see what sys calls were taking the most time, I ended up with: >=20 > 69247 ls 0.190729 STRU struct stat {dev=3D846475008, = ino=3D46220085, mode=3D-rw-r--r-- , nlink=3D1, uid=3D0, gid=3D0, = rdev=3D4294967295, atime=3D1333196714, stime=3D1201004393, = ctime=3D1333196714.547566024, birthtime=3D1333196714.547566024, = size=3D30784, blksize=3D31232, blocks=3D62, flags=3D0x0 } > 69247 ls 0.180121 STRU struct stat {dev=3D846475008, = ino=3D46233417, mode=3D-rw-r--r-- , nlink=3D1, uid=3D0, gid=3D0, = rdev=3D4294967295, atime=3D1333197088, stime=3D1209814737, = ctime=3D1333197088.913571042, birthtime=3D1333197088.913571042, = size=3D3162220, blksize=3D131072, blocks=3D6409, flags=3D0x0 } > 69247 ls 0.152370 RET getdirentries 4088/0xff8 > 69247 ls 0.139939 CALL stat(0x800d8f598,0x7fffffffcca0) > 69247 ls 0.130411 RET __acl_get_link 0 > 69247 ls 0.121602 RET __acl_get_link 0 > 69247 ls 0.105799 RET getdirentries 4064/0xfe0 > 69247 ls 0.105069 RET getdirentries 4068/0xfe4 > 69247 ls 0.096862 RET getdirentries 4028/0xfbc > 69247 ls 0.085012 RET getdirentries 4088/0xff8 > 69247 ls 0.082722 STRU struct stat {dev=3D846475008, = ino=3D72941319, mode=3D-rw-r--r-- , nlink=3D1, uid=3D0, gid=3D0, = rdev=3D4294967295, atime=3D1348686155, stime=3D1348347621, = ctime=3D1348686155.768875422, birthtime=3D1348686155.768875422, = size=3D6686225, blksize=3D131072, blocks=3D13325, flags=3D0x0 } > 69247 ls 0.070318 STRU struct stat {dev=3D846475008, = ino=3D46211679, mode=3D-rw-r--r-- , nlink=3D1, uid=3D0, gid=3D0, = rdev=3D4294967295, atime=3D1333196475, stime=3D1240230314, = ctime=3D1333196475.038567672, birthtime=3D1333196475.038567672, = size=3D829895, blksize=3D131072, blocks=3D1797, flags=3D0x0 } > 69247 ls 0.068060 RET getdirentries 4048/0xfd0 > 69247 ls 0.065118 RET getdirentries 4088/0xff8 > 69247 ls 0.062536 RET getdirentries 4096/0x1000 > 69247 ls 0.061118 RET getdirentries 4020/0xfb4 > 69247 ls 0.055038 STRU struct stat {dev=3D846475008, = ino=3D46220358, mode=3D-rw-r--r-- , nlink=3D1, uid=3D0, gid=3D0, = rdev=3D4294967295, atime=3D1333196720, stime=3D1274282669, = ctime=3D1333196720.972567345, birthtime=3D1333196720.972567345, = size=3D382344, blksize=3D131072, blocks=3D773, flags=3D0x0 } > 69247 ls 0.054948 STRU struct stat {dev=3D846475008, = ino=3D75025952, mode=3D-rw-r--r-- , nlink=3D1, uid=3D0, gid=3D0, = rdev=3D4294967295, atime=3D1351071350, stime=3D1349726805, = ctime=3D1351071350.800873870, birthtime=3D1351071350.800873870, = size=3D2575559, blksize=3D131072, blocks=3D5127, flags=3D0x0 } > 69247 ls 0.054828 STRU struct stat {dev=3D846475008, = ino=3D65021883, mode=3D-rw-r--r-- , nlink=3D1, uid=3D0, gid=3D0, = rdev=3D4294967295, atime=3D1335730367, stime=3D1332843230, = ctime=3D1335730367.541567371, birthtime=3D1335730367.541567371, = size=3D226347, blksize=3D131072, blocks=3D517, flags=3D0x0 } > 69247 ls 0.053743 STRU struct stat {dev=3D846475008, = ino=3D46222016, mode=3D-rw-r--r-- , nlink=3D1, uid=3D0, gid=3D0, = rdev=3D4294967295, atime=3D1333196765, stime=3D1257110706, = ctime=3D1333196765.206574132, birthtime=3D1333196765.206574132, = size=3D62112, blksize=3D62464, blocks=3D123, flags=3D0x0 } > 69247 ls 0.052015 RET getdirentries 4060/0xfdc > 69247 ls 0.051388 RET getdirentries 4068/0xfe4 > 69247 ls 0.049875 RET getdirentries 4088/0xff8 > 69247 ls 0.049156 RET getdirentries 4032/0xfc0 > 69247 ls 0.048609 RET getdirentries 4040/0xfc8 > 69247 ls 0.048279 RET getdirentries 4032/0xfc0 > 69247 ls 0.048062 RET getdirentries 4064/0xfe0 > 69247 ls 0.047577 RET getdirentries 4076/0xfec > (snip) >=20 > the STRU are returns from calling lstat(). >=20 > It looks like both getdirentries and lstat are taking quite a while to = return. The shortest return for any lstat() call is 0.000004 seconds, = the maximum is 0.190729 and the average is around 0.0004. Just from = lstat() alone, that makes "ls" take over 20 seconds. >=20 > I'm prepared to try an L2arc cache device (with = secondarycache=3Dmetadata), but I'm having trouble determining how big = of a device I'd need. We've got >30M inodes now on this filesystem, = including some files with extremely long names. Is there some way to = determine the amount of metadata on a ZFS filesystem? >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" What are your : vfs.zfs.arc_meta_limit and vfs.zfs.arc_meta_used = sysctls? Maybe increasing the limit can help? Regards,