Date: Thu, 3 Jan 2002 15:50:26 +0100 From: Cliff Sarginson <cliff@raggedclown.net> To: freebsd-questions@freebsd.org Subject: Re: sh redirect optimization + fifos Message-ID: <20020103145026.GA8739@raggedclown.net> In-Reply-To: <200201031428.g03ESKO03852@chk.phattydomain.com> References: <200201031428.g03ESKO03852@chk.phattydomain.com>
index | next in thread | previous in thread | raw e-mail
On Thu, Jan 03, 2002 at 06:28:20AM -0800, chk no wrote:
> English:
> A function named "fout" which will read from stdin, break
> up input by lines, and output each line one at a time to
> the given named pipe (fifo).
>
> sh:
> fout () { while read a;do echo $a > $1;done; }
>
>
> The idea being that you could, for example:
>
> $ mkfifo blarg
> $ cat /usr/share/dict/words | fout blarg
>
> (then, elsewhere)
>
> $ cat blarg
> A
> $ cat blarg
> a
> $ cat blarg
> aa
> $ cat blarg
> aal
> $ cat blarg
> aalii
>
> etc., getting one word at a time.
>
> However, if you go & try this, it doesn't operate as outlined above.
> For each access of the fifo, it returns anywhere from 1 to ~5000
> words. sh caches the lines redirected with ">", and then writes
> them in chunks.
Probably does. Probably doesn't with stderr though.
> This is a great optimization for flat files, but
> it goes aginst the intention of the code when used with fifos.
>
Against the intention of the code :) Maybe, but that's the way it
is.
> Changing the function to:
>
> fout () { while read a;do echo $a > $1;sleep .001;done; }
>
> fixes this issue, and makes the fifo accesses return one line at a
> time, but only if sh gets a chance to run often enough. If the load
> average goes above 10 or so,
10 ! Jeez, you got more problems to worry about than this if you
are getting load averages of 10 on your system.
> multiple lines per read will start to
> slip through. Bumping the sleep value all the way up to .1 buys
> some more reliability, but breakes down (on my system) at about
> a load average of 30.
30 Now ! ?
> Bumping the value higher makes the code
> unacceptably slow, and is still prone to breakage if the LA rises
> high enough.
>
>
> How is this supposed to be done?
>
Well not this way. Firstly the sleeps will obviously only put the
problem back a step.
You are also reading data as a stream here, what you are trying to
do, as far as I can see is to read variable length records, where a
record is line of indeterminate length ending in a new line.
I don't think you can do this very elegantly in the shell.
I would suggest using message queues, and writing a little
program in perl,C or possibly awk, to do the reading. A Fifo will work well enough
for a data stream, or fixed length messages, but it ain't really
very handy for what you want to do.
I find your load averages staggering, does anything ever actually
get done on your machine, it must spend 99% of it's time context
switching.
--
Regards
Cliff
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-questions" in the body of the message
help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020103145026.GA8739>