From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 10 13:23:26 2011 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C88EE106564A; Mon, 10 Oct 2011 13:23:26 +0000 (UTC) (envelope-from jeremie@le-hen.org) Received: from smtp5-g21.free.fr (unknown [IPv6:2a01:e0c:1:1599::14]) by mx1.freebsd.org (Postfix) with ESMTP id 988C48FC08; Mon, 10 Oct 2011 13:23:24 +0000 (UTC) Received: from endor.tataz.chchile.org (unknown [82.233.239.98]) by smtp5-g21.free.fr (Postfix) with ESMTP id C4CDBD4828F; Mon, 10 Oct 2011 15:23:18 +0200 (CEST) Received: from felucia.tataz.chchile.org (felucia.tataz.chchile.org [192.168.1.9]) by endor.tataz.chchile.org (Postfix) with ESMTP id 8CBE87FE; Mon, 10 Oct 2011 13:23:17 +0000 (UTC) Received: by felucia.tataz.chchile.org (Postfix, from userid 1000) id 6C1DA91D7; Mon, 10 Oct 2011 13:23:17 +0000 (UTC) Date: Mon, 10 Oct 2011 15:23:17 +0200 From: Jeremie Le Hen To: freebsd-hackers@FreeBSD.org Message-ID: <20111010132317.GA96131@felucia.tataz.chchile.org> References: <20110219185043.GA6573@felucia.tataz.chchile.org> <20110804171928.GB96031@felucia.tataz.chchile.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110804171928.GB96031@felucia.tataz.chchile.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: jhb@FreeBSD.org Subject: Re: [RFC] stdbuf: Force stdio's streams initial buffering mode (patch) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Oct 2011 13:23:26 -0000 Hi, On Thu, Aug 04, 2011 at 07:19:28PM +0200, Jeremie Le Hen wrote: > On Sat, Feb 19, 2011 at 07:50:43PM +0100, Jeremie Le Hen wrote: > > I've been annoyed multiple time when running a command such like > > iostat -x 1 | grep -v ad10 | cat -n > > > > The problem stems from two factors: > > - grep's stdio sees that its stdout is not a terminal, so stdout is > > full buffered and not line-buffered; > > - iostat produces output too slowly so the aforementioned buffer takes > > numerous seconds to be filled and flushed to the last command. > > > > This problems is not specific to FreeBSD, it is actually a consequence > > of POSIX specification. I've checked this on Solaris and Linux. > > > > I've attached a small patch for stdio, so if the environment variable > > STDIO_IOLBF is set, the output streams will be line-oriented by default. > > iostat -x 1 | env STDIO_IOLBF=1 grep -v ad10 | cat -n > > I improved the whole picture. Now there is a shared library > libstdbuf.so which can be loaded with LD_PRELOAD and configured through > a environment variables. It is able to initial control buffering for > stdin, stdout and stderr (no buffering, line buffering, block > buffering). There is also an utility named stdbuf(1) which can be used > to run a command with the appropriate environment variables. Those are > named after a similar tool in recent versions of GNU coreutils; of > course, I also borrowed the interface for POLA. > > Here is how to use it (example taken from the manpage): > > vmstat 1 | stdbuf -o L awk '$2 > 1 || $3 > 1' | cat -n > > I think that using a preloaded shared library is better performance-wise > because libc doesn't bother looking up configuration variables in the > environment upon each execve(2), especially for something which is to be > rarely used. > > Manpages for both stdbuf(1) and libstdbuf(3) are provided. There is > surely room for improvement, so feel free to propose corrections to > them. > > The patch can be applied as-is I think, although I've developped it > against an old -CURRENT source tree (about 6 months ago). > > I'm looking for some feedback and review and hopefully this feature will > be committed for FreeBSD 9.0-RELEASE. It's now too late for 9.0-RELEASE :). But I'm still willing to see this feature committed eventually! While talking about this implementation with a friend, he told me that having a separate shared library sounds a little bit hackish and two or three getenv(3) in the libc startup path shouldn't eat too much CPU cycles. Any opinion on this? Anyone willing to commit this? Thanks. Regards, -- Jeremie Le Hen Men are born free and equal. Later on, they're on their own. Jean Yanne