From owner-freebsd-questions@FreeBSD.ORG Wed Mar 10 18:20:12 2010 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3E3AB106566C for ; Wed, 10 Mar 2010 18:20:12 +0000 (UTC) (envelope-from nvidican@m2.vidican.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id D63078FC19 for ; Wed, 10 Mar 2010 18:20:11 +0000 (UTC) Received: by wyb32 with SMTP id 32so4653712wyb.13 for ; Wed, 10 Mar 2010 10:20:10 -0800 (PST) MIME-Version: 1.0 Sender: nvidican@m2.vidican.com Received: by 10.216.85.133 with SMTP id u5mr1274953wee.91.1268245209992; Wed, 10 Mar 2010 10:20:09 -0800 (PST) X-Originating-IP: [136.1.1.154] In-Reply-To: <201003101151.50738.web@3dresearch.com> References: <201003101151.50738.web@3dresearch.com> Date: Wed, 10 Mar 2010 13:20:09 -0500 X-Google-Sender-Auth: 09e40f83a4314f42 Message-ID: <795fc2b81003101020q3a0a73e6u5b43ec4dc1da04ce@mail.gmail.com> From: Nathan Vidican To: Janos Dohanics Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: User Questions Subject: Re: Directory with large number of files X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 18:20:12 -0000 In some passed projects of mine where large quantities of image files had to be referenced I used a hierarchical tree structure based on the filename. For example: instead of '/disk/images/0018391.jpg' I used '/disk/images/10000/8/3/91.jpg' instead of '/disk/images/abcdsomename.jpg' I used '/disk/images/a/b/cdsomename.jpg' In most cases it was to store large amounts of small image files to reference from web-apps and such with a database key. Usually I tried to work either numerically or alphabetically scaling to the database itself. This also allows you to (if needed) use multiple disks/mount-points down the line if the scale grows (ie: move /disk/images/a or /disk/images/10000 to a dedicated disk/mount if room runs out or for performance reasons). Not sure on the system-side what the advantages versus using a single large directory will be - but I noted it was a much easier way to maintain things and scaled very well (in one case had a directory with well over a million images tied to a database of a product catalog, we broke it down by department/section and then by numeric key and eventually had to dedicate larger/faster storage to certain sections thus off-loading the work from all-in-one and requiring minimal effort to do so). Food for thought anyways - hope it helps. -- Nathan Vidican nathan@vidican.com On Wed, Mar 10, 2010 at 11:51 AM, Janos Dohanics wrote: > I have a directory with over 180,000 files in 4 subdirectories. Over > 170,000 > of the files are in one of the subdirectories. The files are image files, > mostly <50K. The machine has 2 GB RAM. > > What would be a reasonable setting for vfs.ufs.dirhash_maxmem? Is there a > disadvantage to setting vfs.ufs.dirhash_maxmem too high? > > Is there any other setting to change to find files faster in a directory > with > such large number of files? > > Thank you for your advice. > > -- > Janos Dohanics > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" >