Date: Tue, 02 Aug 2011 10:29:02 +0200 From: "Ronald Klop" <ronald-freebsd8@klop.yi.org> To: freebsd-stable@freebsd.org, "seanrees@gmail.com" <seanrees@gmail.com> Subject: Re: ZFS directory with a large number of files Message-ID: <op.vzku6oaa8527sy@212-182-167-131.ip.telfort.nl> In-Reply-To: <CAJGy1F0d7jeyaFuNdXe%2BucTL2r7R4suCyu8xG7WRHenMFZH-6g@mail.gmail.com> References: <CAJGy1F0d7jeyaFuNdXe%2BucTL2r7R4suCyu8xG7WRHenMFZH-6g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Not an in depth solution for ZFS, but maybe a solution for you. mkdir images2 mv images/* images2 rmdir images Ronald. On Tue, 02 Aug 2011 09:39:03 +0200, seanrees@gmail.com =20 <seanrees@gmail.com> wrote: > Hi there, > > I Googled around and checked the PRs and wasn't successful in finding > any reports of what I'm seeing. I'm hoping someone here can help me > debug what's going on. > > On my FreeBSD 8.2-S machine (built circa 12th June), I created a > directory and populated it over the course of 3 weeks with about 2 > million individual files. As you might imagine, a 'ls' of this > directory took quite some time. > > The files were conveniently named with a timestamp in the filename > (still images from a security camera, once per second) so I've since > moved them all to timestamped directories (yyyy/MM/dd/hh/mm). What I > found though was the original directory the images were in is still > very slow to ls -- and it only has 1 file in it, another directory. > > To clarify: > % ls second > [lots of time and many many files enumerated] > % # rename files using rename script > % ls second > [wait ages] > 2011 dead > % mkdir second2 && mv second/2011 second2 > % ls second2 > [fast!] > 2011 > % ls second > [still very slow] > dead > % time ls second > dead/ > gls -F --color 0.00s user 1.56s system 0% cpu 3:09.61 total > > (timings are similar for /bin/ls) > > This data is stored on a striped ZFS pool (version 15, though the > kernel reports version 28 is available but zpool upgrade seems to > disagree), 2T in size. I've run zpool scrub with no effect. ZFS is > busily driving the disks away; my iostat monitoring has all three > drives in the zpool running at 40-60% busy for the duration of the ls > (it was quiet before). > > I've attached truss to the ls process. It spends a lot of time here: > fstatfs(0x5,0x7fffffffe0d0,0x800ad5548,0x7fffffffdfd8,0x0,0x0) =3D 0 (0= x0) > > I'm thinking there's some old ZFS metadata that it's looking into, but > I'm not sure how to best dig into this to understand what's going on > under the hood. > > Can anyone perhaps point me the right direction on this? > > Thanks, > > Sean > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.or= g"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?op.vzku6oaa8527sy>