From owner-freebsd-stable@FreeBSD.ORG  Thu Jan 20 09:45:57 2005
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 3472016A4CE
	for <stable@freebsd.org>; Thu, 20 Jan 2005 09:45:57 +0000 (GMT)
Received: from mail09.syd.optusnet.com.au (mail09.syd.optusnet.com.au
	[211.29.132.190])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 206FB43D31
	for <stable@freebsd.org>; Thu, 20 Jan 2005 09:45:56 +0000 (GMT)
	(envelope-from PeterJeremy@optushome.com.au)
Received: from cirb503493.alcatel.com.au
	(c211-30-75-229.belrs2.nsw.optusnet.com.au [211.30.75.229])
	j0K9jqFU023702
	(version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO);
	Thu, 20 Jan 2005 20:45:54 +1100
Received: from cirb503493.alcatel.com.au (localhost.alcatel.com.au
	[127.0.0.1])j0K9jqxP067359;	Thu, 20 Jan 2005 20:45:52 +1100 (EST)
	(envelope-from pjeremy@cirb503493.alcatel.com.au)
Received: (from pjeremy@localhost)j0K9jq0F067358;
	Thu, 20 Jan 2005 20:45:52 +1100 (EST)	(envelope-from pjeremy)
Date: Thu, 20 Jan 2005 20:45:52 +1100
From: Peter Jeremy <PeterJeremy@optushome.com.au>
To: Phillip Salzman <phill@sysctl.net>
Message-ID: <20050120094551.GK79646@cirb503493.alcatel.com.au>
References: <00b001c4fea0$7533d490$6745a8c0@MESE>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <00b001c4fea0$7533d490$6745a8c0@MESE>
User-Agent: Mutt/1.4.2i
cc: stable@freebsd.org
Subject: Re: Very large directory
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.1
Precedence: list
List-Id: Production branch of FreeBSD source code
	<freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 20 Jan 2005 09:45:57 -0000

On Wed, 2005-Jan-19 21:30:53 -0600, Phillip Salzman wrote:
>They've been running for a little while now - and recently we've noticed a
>lot of disk space disappearing.  Shortly after that, a simple du into our
>/var/spool returned a not so nice error:
>
>	du: fts_read: Cannot allocate memory
>
>No matter what command I run on that directory, I just don't seem to have
>enough available resources  to show the files let alone delete them (echo *,
>ls, find, rm -rf, etc.)

I suspect you will need to write something that uses dirent(3) to scan
the offending directory and delete (or whatever) the files one by one.

Skeleton code (in perl) would look like:

chdir $some_dir or die "Can't cd $some_dir: $!";
opendir(DIR, ".") or die "Can't opendir: $!";
while (my $file = readdir(DIR)) {
	next if ($file eq '.' || $file eq '..');
	next if (&this_file_is_still_needed($file));
	unlink $file or warn "Unable to delete $file: $!";
}
closedir DIR;

If you've reached the point where you can't actually read the entire
directory into user memory, expect the cleanup to take quite a while.

Once you've finished the cleanup, you should confirm that the directory
has shrunk to a sensible size.  If not, you need to re-create the
directory and move the remaining files into the new directory.

-- 
Peter Jeremy