From owner-freebsd-current@FreeBSD.ORG Fri Apr 23 06:27:35 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1DEE716A4CE for ; Fri, 23 Apr 2004 06:27:35 -0700 (PDT) Received: from otter3.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 86D9D43D55 for ; Fri, 23 Apr 2004 06:27:34 -0700 (PDT) (envelope-from anderson@centtech.com) Received: from centtech.com (neutrino.centtech.com [10.177.171.220]) by otter3.centtech.com (8.12.3/8.12.3) with ESMTP id i3NDRXE8010720; Fri, 23 Apr 2004 08:27:33 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <408919BA.5070702@centtech.com> Date: Fri, 23 Apr 2004 08:27:22 -0500 From: Eric Anderson User-Agent: Mozilla Thunderbird 0.5 (X11/20040406) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Tim Kientzle References: <40867A5D.9010600@centtech.com> <40887100.3040606@kientzle.com> In-Reply-To: <40887100.3040606@kientzle.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-current@freebsd.org Subject: Re: Directories with 2million files X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Apr 2004 13:27:35 -0000 Tim Kientzle wrote: > Eric Anderson wrote: > >> First, let me say that I am impressed (but not shocked) - FreeBSD >> quietly handled my building of a directory with 2055476 files in it. >> However, several tools seem to choke on that many files ... >> >> $ ls -al | wc -l >> ls: fts_read: Cannot allocate memory >> 0 >> >> Watching memory usage, it goes up to about 515Mb, and runs out of >> memory (can't swap it), and then dies. (I only have 768Mb in this >> machine). > > > Not "can't swap", but "doesn't need to swap." Your 'top' output > shows you've got plenty of free swap, so that's not the issue. > I suspect you have got a per-process data limit of 512MB, so the > kernel is killing the process when it gets too big. Up that > limit, and it should succeed. First, what I was referring to was the fact that after using 512mb of ram, it dies with a memory allocation error. So, since it looked like maybe it *did* run out of memory (I have 768Mb ram, but at the time it died, I have many other things running on it, eating up memory - so it was possible). Your right - there is a 512mb per-process limit (I didn't realize it was set so low), which I just found out with your commands below: > > What does "limit -d" say? Resource limits (current): datasize 524288 kb Ouch! That seems pretty low to me. 1gb would be closer to reasonable if you ask me, but I'm nobody, so take it with a grain of salt. > What is the 'datasize' set to in /etc/login.conf? :datasize=unlimited: > What are you using for DFLDSIZ in your kernel config file? > (See /usr/src/sys/conf/NOTES for more information on DFLDSIZ, > which I believe defaults to 512MB.) Defaults - I'm running the GENERIC kernel on this machine.. 512MB appears to be the default - I also think this should be 1gb default. At least, MAXDSIZ should be higher than 512MB, and it's not clear if the 512MB default limit it mentions is applicable to the DFLDSIZ, MAXSSIZ, MAXDSIZ or all three limits. > If you're using directories with over 2million files, > you probably have other processes that could use > more memory as well, so upping this limit is advisable. Thanks - I'll do that. > The Real Fix > > Of course, 'ls' should probably not be using this > much memory. [..snip..] Thanks Tim for the thoughts.. Again, the offer is open - I'll test any code anyone has for me, and even give an account to an interested hacker.. Eric -- ------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Today is the tomorrow you worried about yesterday. ------------------------------------------------------------------