From owner-freebsd-questions@FreeBSD.ORG Thu Jan 4 06:18:07 2007 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 929F316A407 for ; Thu, 4 Jan 2007 06:18:07 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from gaia.nimnet.asn.au (nimbin.lnk.telstra.net [139.130.45.143]) by mx1.freebsd.org (Postfix) with ESMTP id 178A913C457 for ; Thu, 4 Jan 2007 06:18:05 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (smithi@localhost) by gaia.nimnet.asn.au (8.8.8/8.8.8R1.4) with SMTP id RAA03788; Thu, 4 Jan 2007 17:17:57 +1100 (EST) (envelope-from smithi@nimnet.asn.au) Date: Thu, 4 Jan 2007 17:17:57 +1100 (EST) From: Ian Smith To: Kurt Buff In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: James Long , freebsd-questions@freebsd.org Subject: Re: Batch file question - average size of file in directory X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jan 2007 06:18:07 -0000 On Wed, 3 Jan 2007, Kurt Buff wrote: > On 1/3/07, Ian Smith wrote: > > > From: James Long > > > > From: "Kurt Buff" [..] > > > > I've got a directory with a large number of gzipped files in it (over > > > > 110k) along with a few thousand uncompressed files. > > > > If it were me I'd mv those into a bunch of subdirectories; things get > > really slow with more than 500 or so files per directory .. anyway .. > > I just store them for a while - delete them after two weeks if they're > not needed again. The overhead isn't enough to worry about at this > point. Fair enough. We once had a security webcam gadget ftp'ing images into a directory every minute, 1440/day, but a php script listing the files for display was timing out just on the 'ls' when over ~2000 files on a 2.4G P4, prompting better (in that case, directory per day) organisation. [..] > > > while read fname; do > > - > if file $fname | grep -q "compressed" > > + if file $fname | grep -q "gzip compressed" > > > then > > - > echo -n "$(zcat $fname | wc -c)+" > > + echo -n "$(gunzip -l $fname | grep -v comp | awk '{print $2}')+" That was off the top of my (then tired) head, and will of course barf if 'comp' appears anywhere in a filename; it should be 'grep -v ^comp'. > Ah - yes, I think that's much better. I should have thought of awk. That's the extent of my awk-foo, see Giorgos' post for fancier stuff :) And thanks to James for the base script to bother playing with .. Cheers, Ian