Date: Tue, 2 Aug 2011 02:42:26 -0700 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: "seanrees@gmail.com" <seanrees@gmail.com> Cc: freebsd-stable@freebsd.org Subject: Re: ZFS directory with a large number of files Message-ID: <20110802094226.GA93114@icarus.home.lan> In-Reply-To: <CAJGy1F0V65YB7L_1T-26O_gUkUUzn6mef036iuAw6HRGjxFRQA@mail.gmail.com> References: <CAJGy1F0d7jeyaFuNdXe%2BucTL2r7R4suCyu8xG7WRHenMFZH-6g@mail.gmail.com> <20110802090830.GA92646@icarus.home.lan> <CAJGy1F0V65YB7L_1T-26O_gUkUUzn6mef036iuAw6HRGjxFRQA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Aug 02, 2011 at 10:16:35AM +0100, seanrees@gmail.com wrote: > On Tue, Aug 2, 2011 at 10:08 AM, Jeremy Chadwick > <freebsd@jdc.parodius.com> wrote: > > On Tue, Aug 02, 2011 at 08:39:03AM +0100, seanrees@gmail.com wrote: > >> On my FreeBSD 8.2-S machine (built circa 12th June), I created a > >> directory and populated it over the course of 3 weeks with about 2 > >> million individual files. > > > > I'll keep this real simple: > > > > Why did you do this? > > > > I hope this was a stress test of some kind. ?If not: > > Not really, but it turned into one. > > The camera I was using had the ability (rather handily) to upload a > still image once per second via FTP to a server of my choosing. It > didn't have the ability to organize them for me in a neat directory > hierarchy. So on holidays I went for 3 weeks and came back to ~2M > images in the same directory. I equate this to the following conversation with a doctor: "I shoved 2 million cotton balls into my ear, and now my hearing is sub-par". "Why did you do this?" In this situation, the correct reply to the physician is: "because I wasn't thinking, doc". I suppose the reason I'm being so brash is that people doing this always brings into question the thought process that went into the decision to do said thing (both on your part, as well as the engineers of your camera who thought such a feature would be a wise choice, especially with an interval of 1 picture per second[1]). When I was being taught the ropes of system administration at Oregon State, the team of crotchety UNIX admins there made it quite clear that there were things you just Did Not Do(tm) to computer systems. Shoving thousands of files into a single directory with no hierarchy was one of them. Again I will point this out: it doesn't matter what filesystem is used, they all will suffer (just to different degrees) in this situation. This is just one of those things where I have to say: Please Don't Do This(tm). At least this is an educational experience for you. > > This is the 2nd or 3rd mail in recent months from people saying "I > > decided to do something utterly stupid with my filesystem[1] and now I'm > > asking why performance sucks". > > > > Why can people not create proper directory tree layouts to avoid this > > problem regardless of what filesystem is used? ?I just don't get it. > > I'm not sure it's utterly stupid; I didn't expect legendarily fast > performance from 'ls' or anything else that enumerated the contents of > the directory when all the files were there. You shouldn't expect any kind of performance, at all, on any filesystem, when/if you do this. The effects can linger for quite some time. I can't come up with a good analogy at the moment. > Now that the files are neatly organized, I expected fstatfs() on the > directory to become fast again. It isn't. I'd like to understand why > (or maybe learn a new trick or two about inspecting ZFS...) Ronald's recommendation should address the problem, or at least diminish it in the least. [1]: I would also strongly advocate contacting your camera manufacturer and asking them to add some extremely simple/basic code for adding a directory hierarchy when the images are put on an FTP server. The amount of code is extremely nominal. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110802094226.GA93114>