From owner-freebsd-questions Thu Jan 3 6:50:33 2002 Delivered-To: freebsd-questions@freebsd.org Received: from post.mail.nl.demon.net (post-10.mail.nl.demon.net [194.159.73.20]) by hub.freebsd.org (Postfix) with ESMTP id C3FEB37B405 for ; Thu, 3 Jan 2002 06:50:28 -0800 (PST) Received: from [212.238.194.207] (helo=tanya.raggedclown.net) by post.mail.nl.demon.net with esmtp (Exim 3.33 #1) id 16M9CN-0004LW-00 for freebsd-questions@freebsd.org; Thu, 03 Jan 2002 14:50:27 +0000 Received: by tanya.raggedclown.net (Postfix on SuSE Linux 7.3 (i386), from userid 500) id 28CEB1176; Thu, 3 Jan 2002 15:50:26 +0100 (CET) Date: Thu, 3 Jan 2002 15:50:26 +0100 From: Cliff Sarginson To: freebsd-questions@freebsd.org Subject: Re: sh redirect optimization + fifos Message-ID: <20020103145026.GA8739@raggedclown.net> References: <200201031428.g03ESKO03852@chk.phattydomain.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200201031428.g03ESKO03852@chk.phattydomain.com> User-Agent: Mutt/1.3.24i Sender: owner-freebsd-questions@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Thu, Jan 03, 2002 at 06:28:20AM -0800, chk no wrote: > English: > A function named "fout" which will read from stdin, break > up input by lines, and output each line one at a time to > the given named pipe (fifo). > > sh: > fout () { while read a;do echo $a > $1;done; } > > > The idea being that you could, for example: > > $ mkfifo blarg > $ cat /usr/share/dict/words | fout blarg > > (then, elsewhere) > > $ cat blarg > A > $ cat blarg > a > $ cat blarg > aa > $ cat blarg > aal > $ cat blarg > aalii > > etc., getting one word at a time. > > However, if you go & try this, it doesn't operate as outlined above. > For each access of the fifo, it returns anywhere from 1 to ~5000 > words. sh caches the lines redirected with ">", and then writes > them in chunks. Probably does. Probably doesn't with stderr though. > This is a great optimization for flat files, but > it goes aginst the intention of the code when used with fifos. > Against the intention of the code :) Maybe, but that's the way it is. > Changing the function to: > > fout () { while read a;do echo $a > $1;sleep .001;done; } > > fixes this issue, and makes the fifo accesses return one line at a > time, but only if sh gets a chance to run often enough. If the load > average goes above 10 or so, 10 ! Jeez, you got more problems to worry about than this if you are getting load averages of 10 on your system. > multiple lines per read will start to > slip through. Bumping the sleep value all the way up to .1 buys > some more reliability, but breakes down (on my system) at about > a load average of 30. 30 Now ! ? > Bumping the value higher makes the code > unacceptably slow, and is still prone to breakage if the LA rises > high enough. > > > How is this supposed to be done? > Well not this way. Firstly the sleeps will obviously only put the problem back a step. You are also reading data as a stream here, what you are trying to do, as far as I can see is to read variable length records, where a record is line of indeterminate length ending in a new line. I don't think you can do this very elegantly in the shell. I would suggest using message queues, and writing a little program in perl,C or possibly awk, to do the reading. A Fifo will work well enough for a data stream, or fixed length messages, but it ain't really very handy for what you want to do. I find your load averages staggering, does anything ever actually get done on your machine, it must spend 99% of it's time context switching. -- Regards Cliff To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message