Date: 18 Apr 2002 04:00:35 -0300 From: "Jim C." <notjames@snafu.concon.homeip.net> To: Dan Nelson <dnelson@allantgroup.com> Cc: "Jim C." <notjames@concon.homeip.net>, Michael E Mercer <mmercer@nc.rr.com>, Taylor Dondich <thexder@lvcm.com>, questions@FreeBSD.ORG Subject: Re: More of a scripting question I guess. Message-ID: <1019113236.72577.84.camel@snafu.concon.homeip.net> In-Reply-To: <20020418044346.GB21402@dan.emsphone.com> References: <000901c1e678$bc14fa80$6600a8c0@penguin> <20020418015926.GI72244@dan.emsphone.com> <3CBE2F51.41E19E8A@nc.rr.com> <20020418023400.GJ72244@dan.emsphone.com> <1019106179.72577.32.camel@snafu.concon.homeip.net> <20020418044346.GB21402@dan.emsphone.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 2002-04-18 at 01:43, Dan Nelson wrote:
> In the last episode (Apr 18), Jim C. said:
> > On Wed, 2002-04-17 at 23:34, Dan Nelson wrote:
> > > In the last episode (Apr 17), Michael E Mercer said:
> > > > another way to do it is this with the same outcome:
> > > >
> > > > while read address ; do
> > > > somecommand $address
> > > > done < file.txt
> > >
> > > But that puts the input file at the end of the command, which makes it
> > > harder to see where the input is coming from, especially if you have a
> > > lot of stuff in your loop.
> >
> > But its still more resource intensive. If you pipe a huge file into
> > while the machine has to store that file into memory before processing
> > it whereas with the redirection it simply gets filtered straight into
> > the while loop line by line.
>
> No it doesn't;
It does. (not significantly but it does)
the 'read' command will pull one line at a time out of
> the pipe.
Correct. However, the pipe "fills up" before read starts to take lines
out of it. This is where the resources get taken.
The only extra overhead is the cost of sending your input
> file through the pipe.
>
Which is what I was talking about. Consider the following:
Using mkfile I created a bogus file with only 2Mb of "stuff" in it.
__ snip __
-rw-r--r-- 1 notjames notjames 364 Apr 18 02:10 bmpipetest.pl
-rw------- 1 notjames notjames 2097152 Apr 18 01:50 file
-rw-r--r-- 1 notjames notjames 76 Apr 18 01:48 redirected.ksh
-rw-r--r-- 1 notjames notjames 80 Apr 18 01:48 unnamed_pipe.ksh
[notjames@snafu ~/build]$ cat unnamed_pipe.ksh
#!/bin/ksh
cat file | while read line;
do
echo $line > /dev/null 2>&1
done
[notjames@snafu ~/build]$ cat redirected.ksh
#!/bin/ksh
while read line;
do
echo $Line > /dev/null 2>&1
done < file
[notjames@snafu ~/build]$
[notjames@snafu ~/build]$ cat bmpipetest.pl
#!/usr/bin/perl -w
use Benchmark;
use strict;
my $count = 10;
# Use Perl code in strings...
my @times = timethese($count, {
'pipe' => \&pipe,
'redirect' => \&redirect
});
print "@times\n";
sub pipe {
system('ksh unnamed_pipe.ksh');
}
sub redirect {
system('ksh redirected.ksh');
}
__ end snippage __
pipe: 144 wallclock secs ( 0.00 usr 0.00 sys + 12.77 cusr 98.50
csys = 0.00 CPU)
redirect: 1 wallclock secs ( 0.00 usr 0.00 sys + 0.55 cusr 0.68
csys = 0.00 CPU)
[notjames@snafu ~/build]$ time ksh unnamed_pipe.ksh
real 0m6.101s
user 0m1.400s
sys 0m4.416s
[notjames@snafu ~/build]$ time ksh redirected.ksh
real 0m0.141s
user 0m0.054s
sys 0m0.070s
[notjames@snafu ~/build]$
I decided to run this test on a larger real (I created it with bogus
stuff) file (about 15 Mb) and ran a top to check and see what was going
on (after changing the above scripts some to point to the larger file):
(unnamed_pipe - 1 snap shot)
57066 notjames 61 0 520K 372K RUN 0:56 83.80% 80.57% ksh
(redirected - 1 snap shot)
57079 notjames 64 0 516K 368K RUN 0:57 86.78% 83.59% ksh
[notjames@snafu ~/build]$ time ksh unnamed_pipe.ksh
real 3m29.525s
user 1m18.960s
sys 1m43.168s
[notjames@snafu ~/build]$ time ksh redirected.ksh
real 3m19.777s
user 1m8.451s
sys 1m47.442s
and the benchmark (changed to 3 iterations each since it takes so long
and I need to get to bed):
Benchmark: timing 3 iterations of pipe, redirect...
pipe: 622 wallclock secs ( 0.00 usr 0.00 sys + 240.34 cusr 336.41
csys = 0.00 CPU)
(warning: too few iterations for a reliable count)
redirect: 547 wallclock secs ( 0.00 usr 0.00 sys + 207.83 cusr 309.48
csys = 0.00 CPU)
(warning: too few iterations for a reliable count)
Based on these tests I guarantee you the pipe is taking more system
resources (albeit not much) than the redirect is.
- Jim
> If you had done:
>
> lines=`cat myfile`
> for address in $lines ; do
> mycommand $address
> done
>
> , that does pull the whole input file into memory.
>
> --
> Dan Nelson
> dnelson@allantgroup.com
>
> To Unsubscribe: send mail to majordomo@FreeBSD.org
> with "unsubscribe freebsd-questions" in the body of the message
>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1019113236.72577.84.camel>