From owner-freebsd-performance@FreeBSD.ORG Fri Dec 21 21:29:37 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 047B516A469 for ; Fri, 21 Dec 2007 21:29:37 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id D3F8213C448 for ; Fri, 21 Dec 2007 21:29:36 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id E980E1A4D7C; Fri, 21 Dec 2007 13:28:08 -0800 (PST) Date: Fri, 21 Dec 2007 13:28:08 -0800 From: Alfred Perlstein To: Alexandre Biancalana Message-ID: <20071221212808.GE16982@elvis.mu.org> References: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> <20071221201625.GZ16982@elvis.mu.org> <8e10486b0712211249v4c5571ddud21b277f686992b2@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8e10486b0712211249v4c5571ddud21b277f686992b2@mail.gmail.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-performance@freebsd.org Subject: Re: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2007 21:29:37 -0000 * Alexandre Biancalana [071221 12:48] wrote: > On 12/21/07, Alfred Perlstein wrote: > > Hi Alfred ! > > > > > There is a lot of very good tuning advice in this thread, however > > one thing to note is that having ~1 million files in a directory > > is not a very good thing to do on just about any filesystem. > > I think I was not clear, I will try explain better. > > This Backup Server has a /backup zfs filesystem of 4TB. > > Each host that do backups to this server has a /backup/ and > /backup//YYYYMMDD zfs filesystems, the last contains the > backups for some day of that server. > > My problem is with some hosts that have in your directory structure a > lot of small files, independent of the hierarchy. Can you not tar these files together? > > One trick that a lot of people do is hashing the directories themselves > > so that you use some kind of computation to break this huge dir into > > multiple smaller dirs. > > I have the two cases, when you have a lot of files inside on directory > without any directory organization/distribution but I also have > problems with hosts that have files organized in a hierarchy like > YYYY/MM/DD/ having no more that 200 files in the day directory > level, but almost one million of files in total. > > Just for info, I made the previous suggested tuning (raise dirhash, > maxvnodes) but this improve nothing. > > Thanks for your hint! What application are you scanning these files with? I know I had issues with rsync in particular where I had to have it rsync smaller pieces of a collection for it to work nicely instead of going for the whole heirarchy. -- - Alfred Perlstein