From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 18 23:12:04 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B62A9106568F for ; Sat, 18 Oct 2008 23:12:04 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by mx1.freebsd.org (Postfix) with ESMTP id 7901E8FC0A for ; Sat, 18 Oct 2008 23:12:04 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.3/8.14.3) with ESMTP id m9INC38w020985 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sat, 18 Oct 2008 18:12:03 -0500 (CDT) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.3/8.14.3/Submit) id m9INC28o020984; Sat, 18 Oct 2008 18:12:02 -0500 (CDT) (envelope-from dan) Date: Sat, 18 Oct 2008 18:12:02 -0500 From: Dan Nelson To: Ivan Voras Message-ID: <20081018231201.GM99270@dan.emsphone.com> References: <20081018213502.GL99270@dan.emsphone.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-OS: FreeBSD 7.1-PRERELEASE User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-hackers@freebsd.org Subject: Re: Pipes, cat buffer size X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Oct 2008 23:12:04 -0000 In the last episode (Oct 19), Ivan Voras said: > Dan Nelson wrote: > > In the last episode (Oct 18), Ivan Voras said: > >> I'm working on a program that's intended to be used as a "filter", > >> as in "something | myprogram > file". I'm trying it with cat and > >> I'm seeing my read()s return small blocks, 64 kB in size. I > >> suppose this is because cat writes in 64 kB blocks. So: > >> > >> a) Is there a way to programatically, per-process, set the pipe buffer > >> size? The program in question is a compressor and it's particularly > >> inefficient when given small blocks and I'm wondering if the system can > >> buffer enough data for it. > > > > Why not keep reading until you reach your desired compression block > > size? Bzip2's default blocksize is 900k, for example. > > Of course. But that's not the point :) From what I see (didn't look at > the code), Linux for example does some kind of internal buffering that > decouples how the reader and the writer interact. I think that with > FreeBSD's current behaviour the writer could write 1-byte buffers and > the reader will be forced to read each byte individually. I don't know > if there's some ulterior reason for this. No; take a look at /sys/kern/sys_pipe.c . Depending on how much data is in the pipe, it switches between async in-kernel buffering (<8192 bytes), and direct page wiring between sender and receiver (basically zero-copy). > >> b) Is there any objection to the following patch to cat: > > > > It might be simpler to just use "dd if=myfile obs=1m" instead of > > patching cat. > > I believe patching cat to bring its block size into the century of the > fruitbat has its own benefits. -- Dan Nelson dnelson@allantgroup.com