Date: Sat, 06 Aug 2011 07:51:58 +0200 From: "Hartmann, O." <ohartman@zedat.fu-berlin.de> To: "Patrick M. Hausen" <hausen@punkt.de> Cc: freebsd-stable@freebsd.org, Christian Weisgerber <naddy@mips.inka.de> Subject: Re: ZFS directory with a large number of files Message-ID: <4E3CD67E.7090201@zedat.fu-berlin.de> In-Reply-To: <BA47D829-B2E3-419B-AC50-FD3F6FCC54EF@punkt.de> References: <CAJGy1F0d7jeyaFuNdXe%2BucTL2r7R4suCyu8xG7WRHenMFZH-6g@mail.gmail.com> <20110802094226.GA93114@icarus.home.lan> <42039B84-D6CE-4780-AA70-8500B1B32036@gsoft.com.au> <4E37CD13.1070402@digsys.bg> <j1h18t$jh1$1@lorvorc.mips.inka.de> <BA47D829-B2E3-419B-AC50-FD3F6FCC54EF@punkt.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On 08/05/11 21:47, Patrick M. Hausen wrote: > Hi, all, > > Am 05.08.2011 um 17:12 schrieb Christian Weisgerber: >> Daniel Kalchev<daniel@digsys.bg> wrote: >> >>> On 02.08.11 12:46, Daniel O'Connor wrote: >>>> I am pretty sure UFS does not have this problem. i.e. once you >>>> delete/move the files out of the directory its performance would be >>>> good again. >>> UFS would be the classic example of poor performance if you do this. >> "Classic" indeed. UFS dirhash has pretty much taken care of this >> a decade ago. > While dirhash is quite an improvement, it is definitely no silver bullet. > > When I asked Kirk McKusick at last year's EuroBSDCon if having > a six-figure number of files in a single directory was a clever idea > (I just had a customer who ran into that situation), he just smiled > and shook his head. > > The directory in question was the typo3temp/pics directory that > TYPO3 uses to scale images that are part of the website, so they > are handed to the browser in exactly the size they are supposed > to be rendered. The performance impact was quite heavy, because > at some point requests started to pile up, PHP scripts did not finish > in time, fcgi slots stayed used ... most of you will know that scenario. > At some threshold a machine goes from "loaded, maybe a bit slow, > but generally responsive" to "no f*ing way". > > Best regards, > Patrick > I have similar situation here, but with a numerical simulation software, which drops for each timestep of integration a file of all integrated objects. Since the code is adopted and not very clever written in terms of doinf its I/O, I have to deal with this. While performing dynamical high resolution integrations of several hundreds of thousand objects over a time scale of 1 Ga produces even with a larger dump-delta creates a lot of files. making those files bigger results in a situation where they are hard to analyse, so its a tradeoff situation. ZFS and UFS2 perform bad on this situation, UFS2 even more than ZFS, but also ZFS is still a pain in the ass. Oliver
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4E3CD67E.7090201>