From owner-freebsd-hackers@FreeBSD.ORG Wed Nov 5 16:30:37 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2EF61065673 for ; Wed, 5 Nov 2008 16:30:37 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx4.mail.ru (fallback.mail.ru [194.67.57.14]) by mx1.freebsd.org (Postfix) with ESMTP id 8EC7C8FC23 for ; Wed, 5 Nov 2008 16:30:37 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx38.mail.ru (mx38.mail.ru [194.67.23.16]) by mx4.mail.ru (mPOP.Fallback_MX) with ESMTP id 5D14A7EFA for ; Wed, 5 Nov 2008 16:40:14 +0300 (MSK) Received: from [217.25.27.27] (port=36252 helo=[217.25.27.27]) by mx38.mail.ru with asmtp id 1KxicG-0006Ys-00 for freebsd-hackers@freebsd.org; Wed, 05 Nov 2008 16:40:12 +0300 Message-ID: <4911A23B.7050104@mail.ru> Date: Wed, 05 Nov 2008 17:40:11 +0400 From: rihad User-Agent: Icedove 1.5.0.14eol (X11/20080724) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: OK X-Mailman-Approved-At: Wed, 05 Nov 2008 16:46:55 +0000 Subject: Asynchronous pipe I/O X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rihad@mail.ru List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2008 16:30:37 -0000 Imagine this shell pipeline: sh prog1 | sh prog2 As given above, prog1 blocks if prog2 hasn't yet read previously written data (actually, newline separated commands) or is busy. What I want is for prog1 to never block: sh prog1 | buffer | sh prog2 I first thought that the aptly named misc/buffer port would do exactly what I wanted: buffering prog1 output for prog2 to read it at its earliest convenience. That way prog1 would never block (unless it hit buffer's memory limits). Alas, misc/buffer was originally designed for tape backups, and despite its author's stating its applicability to other uses: "This is a program designed initially to speed up writing tapes on remote tape drives, but may be used as a general pipe buffering utility." buffer never starts writing unless the limit given by -s is crossed, which is 10 kbytes by default and cannot be less than 496 bytes, which is too much for me. Ideally I want it to start writing immediately, whenever new data hits its pools. Unfortunately, the -p 0 option doesn't work either: -s size Size in bytes of each block. The default blocksize is 10k to match the normal output of the tar(1) program. -p percentage Only start a write when the given percentage of the internal queue is full. A percentage around 75 often proves best. Defaults to zero. Wouldn't such an intermediary tool be a great way to boost performance for certain types of solutions? Thanks for any tips (Sorry if this was an inappropriate place to ask)