Date: Wed, 3 Jan 2007 10:48:05 -0800 From: "Kurt Buff" <kurt.buff@gmail.com> To: "Ian Smith" <smithi@nimnet.asn.au> Cc: James Long <list@museum.rain.com>, freebsd-questions@freebsd.org Subject: Re: Batch file question - average size of file in directory Message-ID: <a9f4a3860701031048l7969e409ged9e3e583ed4ebdd@mail.gmail.com> In-Reply-To: <Pine.BSF.3.96.1070104025642.7024B-100000@gaia.nimnet.asn.au> References: <20070103120041.C104816A569@hub.freebsd.org> <Pine.BSF.3.96.1070104025642.7024B-100000@gaia.nimnet.asn.au>
next in thread | previous in thread | raw e-mail | index | archive | help
On 1/3/07, Ian Smith <smithi@nimnet.asn.au> wrote:
> > Message: 17
> > Date: Tue, 2 Jan 2007 19:50:01 -0800
> > From: James Long <list@museum.rain.com>
>
> > > Message: 28
> > > Date: Tue, 2 Jan 2007 10:20:08 -0800
> > > From: "Kurt Buff" <kurt.buff@gmail.com>
>
> > > I don't even have a clue how to start this one, so am looking for a little help.
> > >
> > > I've got a directory with a large number of gzipped files in it (over
> > > 110k) along with a few thousand uncompressed files.
>
> If it were me I'd mv those into a bunch of subdirectories; things get
> really slow with more than 500 or so files per directory .. anyway ..
I just store them for a while - delete them after two weeks if they're
not needed again. The overhead isn't enough to worry about at this
point.
> > > I'd like to find the average uncompressed size of the gzipped files,
> > > and ignore the uncompressed files.
> > >
> > > How on earth would I go about doing that with the default shell (no
> > > bash or other shells installed), or in perl, or something like that.
> > > I'm no scripter of any great expertise, and am just stumbling over
> > > this trying to find an approach.
> > >
> > > Many thanks for any help,
> > >
> > > Kurt
> >
> > Hi, Kurt.
>
> And hi, James,
>
> > Can I make some assumptions that simplify things? No kinky filenames,
> > just [a-zA-Z0-9.]. My approach specifically doesn't like colons or
> > spaces, I bet. Also, you say gzipped, so I'm assuming it's ONLY gzip,
> > no bzip2, etc.
> >
> > Here's a first draft that might give you some ideas. It will output:
> >
> > foo.gz : 3456
> > bar.gz : 1048576
> > (etc.)
> >
> > find . -type f | while read fname; do
> > file $fname | grep -q "compressed" && echo "$fname : $(zcat $fname | wc -c)"
> > done
>
> % file cat7/tuning.7.gz
> cat7/tuning.7.gz: gzip compressed data, from Unix
>
> Good check, though grep "gzip compressed" excludes bzip2 etc.
>
> But you REALLY don't want to zcat 110 thousand files just to wc 'em,
> unless it's a benchmark :) .. may I suggest a slight speedup, template:
>
> % gunzip -l cat7/tuning.7.gz
> compressed uncompr. ratio uncompressed_name
> 13642 38421 64.5% cat7/tuning.7
>
> > If you really need a script that will do the math for you, then
> > pip the output of this into bc:
> >
> > #!/bin/sh
> >
> > find . -type f | {
> >
> > n=0
> > echo scale=2
> > echo -n "("
> > while read fname; do
> - > if file $fname | grep -q "compressed"
> + if file $fname | grep -q "gzip compressed"
> > then
> - > echo -n "$(zcat $fname | wc -c)+"
> + echo -n "$(gunzip -l $fname | grep -v comp | awk '{print $2}')+"
> > n=$(($n+1))
> > fi
> > done
> > echo "0) / $n"
> >
> > }
> >
> > That should give you the average decompressed size of the gzip'ped
> > files in the current directory.
>
> HTH, Ian
Ah - yes, I think that's much better. I should have thought of awk.
At some point, I'd like to do a bit more processing of file sizes,
such as trying to find out the number of IP packets each file would
take during an SMTP transaction, so that I could categorize overhead a
bit, but for now the average uncompressed file size is good enough.
Thanks again for your help!
Kurt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a9f4a3860701031048l7969e409ged9e3e583ed4ebdd>