From owner-freebsd-questions Thu Jan 2 6: 1:37 2003 Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B7EA337B401 for ; Thu, 2 Jan 2003 06:01:34 -0800 (PST) Received: from sentry.24cl.com (174.113.sn.ct.dsl.thebiz.net [216.238.113.174]) by mx1.FreeBSD.org (Postfix) with ESMTP id A825E43EA9 for ; Thu, 2 Jan 2003 06:01:33 -0800 (PST) (envelope-from MyRaQ@mgm51.com) Received: from ntmm (unknown [63.119.50.193]) by sentry.24cl.com (Postfix) with ESMTP id 64E3629496 for ; Thu, 2 Jan 2003 09:01:27 -0500 (EST) Message-ID: <200301020901270548.09C1C68B@sentry.24cl.com> X-Mailer: Calypso Version 3.30.00.00 (1) Date: Thu, 02 Jan 2003 09:01:27 -0500 Reply-To: myraq@mgm51.com From: "MikeM" To: freebsd-questions@FreeBSD.ORG Subject: Lots of files in a directory Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-freebsd-questions@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I'm considering setting up my server as a mirror site for the freedb.org lookup database. Unfortunately, I've seem to have run into a stumbling block. The server app requires over 250,000 files in a single directory. Each file is about 2k in size. It was a surprise to me (but probably not to those on this list) that the file system does not handle that many files in an expeditious manner (I'm being kind here). Is there anything I can do so that the file system works faster with such a large number of files? I'm looking for an increase in the area of 5 to 1. For example, the command "rm -rf misc" where "misc" is the directory containing the 250,000 files takes a couple of hours to run. If "misc" is my current working directory, and I type "rm *" I get the message that there are too many arguments being passed into rm. Some details: FreeBSD 4.5, dmesg below. Filesystem containing the files: /dev/da0s1f on /usr (ufs, local, with quotas) Copyright (c) 1992-2002 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.5-RELEASE-p20 #12: Sat Sep 28 11:03:59 EDT 2002 [snip]:/usr/obj/usr/src/sys/STABLE4FW Timecounter "i8254" frequency 1193182 Hz CPU: Pentium III/Pentium III Xeon/Celeron (797.48-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x686 Stepping = 6 Features=0x383fbff real memory = 805224448 (786352K bytes) avail memory = 778473472 (760228K bytes) Preloaded elf kernel "kernel" at 0xc0468000. Pentium Pro MTRR support enabled md0: Malloc disk npx0: on motherboard npx0: INT 16 interface pcib0: on motherboard pci0: on pcib0 pci0: at 1.0 fxp0: port 0x2200-0x223f mem 0xfea00000-0xfeafffff,0xfeb7f000-0xfeb7ffff irq 10 at device 2.0 on pci0 fxp0: Ethernet address 00:06:29:1f:27:61 inphy0: on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto isab0: at device 15.0 on pci0 isa0: on isab0 atapci0: port 0x700-0x70f at device 15.1 on pci0 ata0: at 0x1f0 irq 14 on atapci0 ohci0: mem 0xfeb7e000-0xfeb7efff irq 7 at device 15.2 on pci0 usb0: OHCI version 1.0, legacy support usb0: on ohci0 usb0: USB revision 1.0 uhub0: (unknown) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered pcib1: on motherboard pci1: on pcib1 ahc0: port 0x2300-0x23ff mem 0xeffff000-0xefffffff irq 9 at device 3.0 on pci1 aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/255 SCBs orm0: