From owner-freebsd-hackers@FreeBSD.ORG Wed Nov 5 19:21:18 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D2941065676 for ; Wed, 5 Nov 2008 19:21:18 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx27.mail.ru (mx27.mail.ru [194.67.23.23]) by mx1.freebsd.org (Postfix) with ESMTP id BC2B08FC12 for ; Wed, 5 Nov 2008 19:21:17 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from [217.25.27.27] (port=56447 helo=[217.25.27.27]) by mx27.mail.ru with asmtp id 1KxnwJ-000G2v-00; Wed, 05 Nov 2008 22:21:15 +0300 Message-ID: <4911F229.2010502@mail.ru> Date: Wed, 05 Nov 2008 23:21:13 +0400 From: rihad User-Agent: Icedove 1.5.0.14eol (X11/20080724) MIME-Version: 1.0 To: Peter Jeremy References: <4911A23B.7050104@mail.ru> <20081105181309.GB51239@server.vk2pj.dyndns.org> In-Reply-To: <20081105181309.GB51239@server.vk2pj.dyndns.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mailman-Approved-At: Wed, 05 Nov 2008 21:22:23 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: Asynchronous pipe I/O X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2008 19:21:18 -0000 Peter Jeremy wrote: > On 2008-Nov-05 17:40:11 +0400, rihad wrote: >> Imagine this shell pipeline: >> >> sh prog1 | sh prog2 >> >> >> As given above, prog1 blocks if prog2 hasn't yet read previously written >> data (actually, newline separated commands) or is busy. What I want is >> for prog1 to never block: >> >> sh prog1 | buffer | sh prog2 > > There's also misc/mbuffer which is supposed to be an enhancement of > misc/buffer - though I haven't used either. I have a program I wrote > to do this but it's not in a releasable state. > That thing only works for certain types of devices (tape?): $ while :; do echo hi; sleep 1; done | mbuffer -q | cat warning: Could not stat output device (unsupported by system)! This can result in incorrect written data when using multiple volumes. Continue at your own risk! ^C $ While writing this email I tried another incantation, and buffer finally worked! $ while :; do echo hi; sleep 1; done | buffer -s1 -b100 | cat hi hi hi hi hi ^C (each line was output one second apart). Once again thank to everybody participating! >> Wouldn't such an intermediary tool be a great way to boost performance >> for certain types of solutions? > > I've found that for dump|restore or dump|gzip, I can get quite significant > speedups by adding a buffer that is several hundred MB in the middle. > Well, if OS buffers aren't good enough, then throwing some memory at disk I/O surely helps ;)