Date: Thu, 3 Jan 2002 15:50:26 +0100 From: Cliff Sarginson <cliff@raggedclown.net> To: freebsd-questions@freebsd.org Subject: Re: sh redirect optimization + fifos Message-ID: <20020103145026.GA8739@raggedclown.net> In-Reply-To: <200201031428.g03ESKO03852@chk.phattydomain.com> References: <200201031428.g03ESKO03852@chk.phattydomain.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Jan 03, 2002 at 06:28:20AM -0800, chk no wrote: > English: > A function named "fout" which will read from stdin, break > up input by lines, and output each line one at a time to > the given named pipe (fifo). > > sh: > fout () { while read a;do echo $a > $1;done; } > > > The idea being that you could, for example: > > $ mkfifo blarg > $ cat /usr/share/dict/words | fout blarg > > (then, elsewhere) > > $ cat blarg > A > $ cat blarg > a > $ cat blarg > aa > $ cat blarg > aal > $ cat blarg > aalii > > etc., getting one word at a time. > > However, if you go & try this, it doesn't operate as outlined above. > For each access of the fifo, it returns anywhere from 1 to ~5000 > words. sh caches the lines redirected with ">", and then writes > them in chunks. Probably does. Probably doesn't with stderr though. > This is a great optimization for flat files, but > it goes aginst the intention of the code when used with fifos. > Against the intention of the code :) Maybe, but that's the way it is. > Changing the function to: > > fout () { while read a;do echo $a > $1;sleep .001;done; } > > fixes this issue, and makes the fifo accesses return one line at a > time, but only if sh gets a chance to run often enough. If the load > average goes above 10 or so, 10 ! Jeez, you got more problems to worry about than this if you are getting load averages of 10 on your system. > multiple lines per read will start to > slip through. Bumping the sleep value all the way up to .1 buys > some more reliability, but breakes down (on my system) at about > a load average of 30. 30 Now ! ? > Bumping the value higher makes the code > unacceptably slow, and is still prone to breakage if the LA rises > high enough. > > > How is this supposed to be done? > Well not this way. Firstly the sleeps will obviously only put the problem back a step. You are also reading data as a stream here, what you are trying to do, as far as I can see is to read variable length records, where a record is line of indeterminate length ending in a new line. I don't think you can do this very elegantly in the shell. I would suggest using message queues, and writing a little program in perl,C or possibly awk, to do the reading. A Fifo will work well enough for a data stream, or fixed length messages, but it ain't really very handy for what you want to do. I find your load averages staggering, does anything ever actually get done on your machine, it must spend 99% of it's time context switching. -- Regards Cliff To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020103145026.GA8739>