Date: Wed, 3 Aug 2011 16:51:59 +1000 From: Peter Jeremy <peterjeremy@acm.org> To: "seanrees@gmail.com" <seanrees@gmail.com> Cc: freebsd-stable@freebsd.org Subject: Re: ZFS directory with a large number of files Message-ID: <20110803065159.GC78870@server.vk2pj.dyndns.org> In-Reply-To: <CAJGy1F0d7jeyaFuNdXe%2BucTL2r7R4suCyu8xG7WRHenMFZH-6g@mail.gmail.com> References: <CAJGy1F0d7jeyaFuNdXe%2BucTL2r7R4suCyu8xG7WRHenMFZH-6g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--GPJrCs/72TxItFYR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2011-Aug-02 08:39:03 +0100, "seanrees@gmail.com" <seanrees@gmail.com> wr= ote: >On my FreeBSD 8.2-S machine (built circa 12th June), I created a >directory and populated it over the course of 3 weeks with about 2 >million individual files. As you might imagine, a 'ls' of this >directory took quite some time. > >The files were conveniently named with a timestamp in the filename >(still images from a security camera, once per second) so I've since >moved them all to timestamped directories (yyyy/MM/dd/hh/mm). What I >found though was the original directory the images were in is still >very slow to ls -- and it only has 1 file in it, another directory. I've also seen this behaviour on Solaris 10 after cleaning out a directory with a large number of files (though not as pathological as your case). I tried creating and deleting entries in an unsuccessful effort to trigger directory compaction. I wound up moving the remaining contents into a new directory, deleting the original one and renaming the new directory. It would appear te be a garbage collection bug in ZFS. On 2011-Aug-02 13:10:27 +0300, Daniel Kalchev <daniel@digsys.bg> wrote: >On 02.08.11 12:46, Daniel O'Connor wrote: >> I am pretty sure UFS does not have this problem. i.e. once you=20 >> delete/move the files out of the directory its performance would be=20 >> good again.=20 > >UFS would be the classic example of poor performance if you do this. Traditional UFS (including Solaris) behave badly in this scenario but 4.4BSD derivatives will release unused space at the end of a directory and have smarts to more efficiently skip unused entries at the start of a directory. --=20 Peter Jeremy --GPJrCs/72TxItFYR Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) iEYEARECAAYFAk448A8ACgkQ/opHv/APuIfmjQCgg4ijcCrG0q7oX4cLwKxDd9io TWcAoLDGoEoEkURljItE768LbQddMILj =ZKso -----END PGP SIGNATURE----- --GPJrCs/72TxItFYR--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110803065159.GC78870>