From owner-freebsd-stable@FreeBSD.ORG Tue Aug 2 08:07:06 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9B35C1065675 for ; Tue, 2 Aug 2011 08:07:06 +0000 (UTC) (envelope-from seanrees@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 60DB88FC13 for ; Tue, 2 Aug 2011 08:07:06 +0000 (UTC) Received: by vxg33 with SMTP id 33so6621549vxg.13 for ; Tue, 02 Aug 2011 01:07:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=EwyOVgKo/BipgixWlCaMQX1cUJls5kcrhz2NVayQtFA=; b=TbPneLO6yUmCvsbrTz4ff1lpG6Pg+5h28vgkcu2I3+17/XnzA1iib0jDscoynys3Ht j7Sc4uahDslmXYTnfhDJInIXRxqKrTfF+kcro4sKECf7hosWvhM0Pti0bCPNganNkwRI CXsuePuXFMi9eb6KiSz6AaeFGRizJpDy8Pmps= MIME-Version: 1.0 Received: by 10.52.64.143 with SMTP id o15mr5479030vds.45.1312270743931; Tue, 02 Aug 2011 00:39:03 -0700 (PDT) Received: by 10.52.168.68 with HTTP; Tue, 2 Aug 2011 00:39:03 -0700 (PDT) Date: Tue, 2 Aug 2011 08:39:03 +0100 Message-ID: From: "seanrees@gmail.com" To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: ZFS directory with a large number of files X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2011 08:07:06 -0000 Hi there, I Googled around and checked the PRs and wasn't successful in finding any reports of what I'm seeing. I'm hoping someone here can help me debug what's going on. On my FreeBSD 8.2-S machine (built circa 12th June), I created a directory and populated it over the course of 3 weeks with about 2 million individual files. As you might imagine, a 'ls' of this directory took quite some time. The files were conveniently named with a timestamp in the filename (still images from a security camera, once per second) so I've since moved them all to timestamped directories (yyyy/MM/dd/hh/mm). What I found though was the original directory the images were in is still very slow to ls -- and it only has 1 file in it, another directory. To clarify: % ls second [lots of time and many many files enumerated] % # rename files using rename script % ls second [wait ages] 2011 dead % mkdir second2 && mv second/2011 second2 % ls second2 [fast!] 2011 % ls second [still very slow] dead % time ls second dead/ gls -F --color 0.00s user 1.56s system 0% cpu 3:09.61 total (timings are similar for /bin/ls) This data is stored on a striped ZFS pool (version 15, though the kernel reports version 28 is available but zpool upgrade seems to disagree), 2T in size. I've run zpool scrub with no effect. ZFS is busily driving the disks away; my iostat monitoring has all three drives in the zpool running at 40-60% busy for the duration of the ls (it was quiet before). I've attached truss to the ls process. It spends a lot of time here: fstatfs(0x5,0x7fffffffe0d0,0x800ad5548,0x7fffffffdfd8,0x0,0x0) = 0 (0x0) I'm thinking there's some old ZFS metadata that it's looking into, but I'm not sure how to best dig into this to understand what's going on under the hood. Can anyone perhaps point me the right direction on this? Thanks, Sean