From owner-freebsd-fs@FreeBSD.ORG Sun Jul 24 18:00:36 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7C055106564A; Sun, 24 Jul 2011 18:00:36 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mail.digiware.nl (mail.ip6.digiware.nl [IPv6:2001:4cb8:1:106::2]) by mx1.freebsd.org (Postfix) with ESMTP id 140E08FC1A; Sun, 24 Jul 2011 18:00:36 +0000 (UTC) Received: from rack1.digiware.nl (localhost.digiware.nl [127.0.0.1]) by mail.digiware.nl (Postfix) with ESMTP id EB22B153434; Sun, 24 Jul 2011 20:00:34 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from mail.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NEq-LaHqnyMG; Sun, 24 Jul 2011 20:00:32 +0200 (CEST) Received: from [IPv6:2001:4cb8:3:1:c02b:ce62:71ff:9cbc] (unknown [IPv6:2001:4cb8:3:1:c02b:ce62:71ff:9cbc]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.digiware.nl (Postfix) with ESMTPSA id 235F3153433; Sun, 24 Jul 2011 20:00:32 +0200 (CEST) Message-ID: <4E2C5DBF.3050104@digiware.nl> Date: Sun, 24 Jul 2011 20:00:31 +0200 From: Willem Jan Withagen User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: Ivan Voras References: <13577F3E-DE59-44F4-98F7-9587E26499B8@gmail.com> In-Reply-To: X-Enigmail-Version: 1.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS and large directories - caveat report X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jul 2011 18:00:36 -0000 On 21-7-2011 19:18, Ivan Voras wrote: > On 21 July 2011 18:38, Luiz Otavio O Souza wrote: > >> The general usage on this server is fine, but the periodic (daily) scripts take almost a day to complete and the server is slow as hell while the daily scripts are running. > > Yes, this is how my problem was first diagnosed. > >> So, yes, i can confirm that running 'find' on a ZFS FS with a lot of files is very, very slow (and looks like it isn't related to how the files are distributed on the FS). > > Only it's not just "find" - it's any directory operations - including > file creation and removal. I cannot say that is not related to how > files are distributed on the file system, except the unusually long > operations on the parent of the shard directories in my case. A little late in the thread: Running on 8.2-stable, ZFS version 15 Quad core, 8Gb memory, /home is on a 6-disk(sata) raidz2 fs. The dicertory is a 3 week revolving log of images taken from a security com. So if anything it directory-file should be horribly thrashed It is around 170.000 files in one directory. [/home/sonycam] wjw@zfs.digiware.nl> ls periodical | wc 177364 177364 3369916 0.421u 6.999s 2:00.15 6.1% 37+1522k 0+0io 0pf+0w [/home/sonycam] wjw@zfs.digiware.nl> ls -asl periodical | wc 177401 1774002 13659785 1.747u 11.087s 1:42.98 12.4% 36+1562k 0+0io 0pf+0w Repeated finds after this complete with 10 secs. On average I seen about 100 IOPS/disk and readin is at 5Mbyte/disk. But I do not feel the system is really loaded while doing the ls. I can easily login again, and do other work. --WjW