From owner-freebsd-stable@FreeBSD.ORG Tue Aug 2 09:16:37 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 59827106566C for ; Tue, 2 Aug 2011 09:16:37 +0000 (UTC) (envelope-from seanrees@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 192138FC13 for ; Tue, 2 Aug 2011 09:16:36 +0000 (UTC) Received: by vxg33 with SMTP id 33so6664123vxg.13 for ; Tue, 02 Aug 2011 02:16:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=4hQGSPwwJ/xf3w/7HGjqY/h2LlLnoWj2UwJAluvtAV8=; b=W5cu0iA95zA6kZTawL84M2CiwLOwXgvughX6X1xBt79r1mopBKtYnkUXW07SPAi9oF mqD4klMaZCNFCKJp8hkd6P40VBVkkNcIc/RVVeILoPap/VhXsipmgBeST61fYU/s4Cfy B9x6VHylP0PWDMtXZyJqnRA9LzT8NwyCg7SNI= MIME-Version: 1.0 Received: by 10.52.182.67 with SMTP id ec3mr5516747vdc.246.1312276596008; Tue, 02 Aug 2011 02:16:36 -0700 (PDT) Received: by 10.52.168.68 with HTTP; Tue, 2 Aug 2011 02:16:35 -0700 (PDT) In-Reply-To: <20110802090830.GA92646@icarus.home.lan> References: <20110802090830.GA92646@icarus.home.lan> Date: Tue, 2 Aug 2011 10:16:35 +0100 Message-ID: From: "seanrees@gmail.com" To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: ZFS directory with a large number of files X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2011 09:16:37 -0000 inline On Tue, Aug 2, 2011 at 10:08 AM, Jeremy Chadwick wrote: > On Tue, Aug 02, 2011 at 08:39:03AM +0100, seanrees@gmail.com wrote: >> On my FreeBSD 8.2-S machine (built circa 12th June), I created a >> directory and populated it over the course of 3 weeks with about 2 >> million individual files. > > I'll keep this real simple: > > Why did you do this? > > I hope this was a stress test of some kind. =A0If not: Not really, but it turned into one. The camera I was using had the ability (rather handily) to upload a still image once per second via FTP to a server of my choosing. It didn't have the ability to organize them for me in a neat directory hierarchy. So on holidays I went for 3 weeks and came back to ~2M images in the same directory. > This is the 2nd or 3rd mail in recent months from people saying "I > decided to do something utterly stupid with my filesystem[1] and now I'm > asking why performance sucks". > > Why can people not create proper directory tree layouts to avoid this > problem regardless of what filesystem is used? =A0I just don't get it. I'm not sure it's utterly stupid; I didn't expect legendarily fast performance from 'ls' or anything else that enumerated the contents of the directory when all the files were there. Now that the files are neatly organized, I expected fstatfs() on the directory to become fast again. It isn't. I'd like to understand why (or maybe learn a new trick or two about inspecting ZFS...) Sean