From owner-freebsd-current@FreeBSD.ORG Wed Apr 21 12:54:24 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 76E1B16A4CE for ; Wed, 21 Apr 2004 12:54:24 -0700 (PDT) Received: from smtp0.server.rpi.edu (smtp0.server.rpi.edu [128.113.53.41]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1886843D49 for ; Wed, 21 Apr 2004 12:54:22 -0700 (PDT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by smtp0.server.rpi.edu (8.12.8/8.12.8) with ESMTP id i3LJsFEd019212; Wed, 21 Apr 2004 15:54:17 -0400 Mime-Version: 1.0 X-Sender: drosih@mail.rpi.edu Message-Id: In-Reply-To: <40867A5D.9010600@centtech.com> References: <40867A5D.9010600@centtech.com> Date: Wed, 21 Apr 2004 15:54:14 -0400 To: Eric Anderson , freebsd-current@freebsd.org From: Garance A Drosihn Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Scanned-By: CanIt (www . canit . ca) Subject: Re: Directories with 2million files X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Apr 2004 19:54:24 -0000 At 8:42 AM -0500 4/21/04, Eric Anderson wrote: >... I'm not sure if there is a limit to this number, but at >least we know it works to 2million. I'm running 5.2.1-RELEASE. > >However, several tools seem to choke on that many files - mainly >ls and du. Find works just fine. Here's what my directory looks >like (from the parent): > >drwxr-xr-x 2 anderson anderson 50919936 Apr 21 08:25 data > >and when I cd into that directory, and do an ls: > >$ ls -al | wc -l >ls: fts_read: Cannot allocate memory > 0 > >Watching memory usage, it goes up to about 515Mb, and runs out >of memory (can't swap it), and then dies. (I only have 768Mb in >this machine). An `ls -al' is going to be doing a lot of work, most of which you probably do not care about. (Certainly not if you're just piping it to `wc'!). Depending on what you are looking for, an `ls -1Af' might work better. If you really do want the -l (lowercase L) instead of -1 (digit one), it *might* help to add the -h option. I probably should look at the source code to see if that's really true, but it's so much easier to just have you type in the command and see what happens... Another option is to use the `stat' command instead of `ls'. (I don't know if `stat' will work any better, I'm just saying it's another option you might want to try...). One advantage is that you'd have much better control over what information is printed. >du does the exact same thing. Just a plain `du'? If all you want is the total, did you try `du -s'? I would not expect any problem from `du -s'. >I'd work on some patches, but I'm not worth much when it comes >to C/C++. If someone has some patches, or code to try, let me >know - I'd be more than willing to test, possibly even give out >an account on the machine. It is probably possible to make `ls' behave better in this situation, though I don't know how much of a special-case we would need to make it. -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu