From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 19 00:19:29 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0692E10656A3 for ; Sun, 19 Oct 2008 00:19:29 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.224]) by mx1.freebsd.org (Postfix) with ESMTP id CA2908FC1E for ; Sun, 19 Oct 2008 00:19:28 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so1053442rvf.43 for ; Sat, 18 Oct 2008 17:19:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender :to:subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references :x-google-sender-auth; bh=WXSYCadypVWBjHSChaODv/GtlxPbpWLNr8xezPz6M1k=; b=HbQzeKxWLae17agolonW2MIJA+UlSUyUTqRuEiy7OdZziGWtofnRS9kg+/AqC40nXj kHsl7HD6lLbHzNGxyq7MH4v1tc5oHD562mebSyFxsn/EsQmE2okB5Ne8lzXED2DcNhbn dP60VuTBGqpHizXvxZamYlDDigcAWRiEHcO6k= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references:x-google-sender-auth; b=NjO/u5lRw6upWSUR2kq8tJFKSMn7fK2bTiaZz3hqaV9VJVU24PRhQWCEuAtFZZwQGu djSjjHBvXINQf1Ws27uG/qeQxVkZZOnuLg+1C+Jqf9nnmgiKhH5NlEG7fDz6T67nXu48 oEaIn45vRGDE70DKoSgOTHQwJvVV+KGUieYe4= Received: by 10.141.151.18 with SMTP id d18mr3659849rvo.75.1224375568133; Sat, 18 Oct 2008 17:19:28 -0700 (PDT) Received: by 10.141.153.13 with HTTP; Sat, 18 Oct 2008 17:19:28 -0700 (PDT) Message-ID: <9bbcef730810181719x4387a14yec74bdb6893d1a2a@mail.gmail.com> Date: Sun, 19 Oct 2008 02:19:28 +0200 From: "Ivan Voras" Sender: ivoras@gmail.com To: "Dan Nelson" In-Reply-To: <20081018231201.GM99270@dan.emsphone.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20081018213502.GL99270@dan.emsphone.com> <20081018231201.GM99270@dan.emsphone.com> X-Google-Sender-Auth: e8934d3392509a56 Cc: freebsd-hackers@freebsd.org Subject: Re: Pipes, cat buffer size X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Oct 2008 00:19:29 -0000 2008/10/19 Dan Nelson : > In the last episode (Oct 19), Ivan Voras said: >> Of course. But that's not the point :) From what I see (didn't look at >> the code), Linux for example does some kind of internal buffering that >> decouples how the reader and the writer interact. I think that with >> FreeBSD's current behaviour the writer could write 1-byte buffers and >> the reader will be forced to read each byte individually. I don't know >> if there's some ulterior reason for this. > > No; take a look at /sys/kern/sys_pipe.c . Depending on how much data > is in the pipe, it switches between async in-kernel buffering (<8192 > bytes), and direct page wiring between sender and receiver (basically > zero-copy). Ok, maybe it's just not behaving as I thought it should. See this test program: ---- #include #include #include #define BSIZE (1024*1024) void main() { int r; char buf[BSIZE]; while (1) { r = read(0, buf, BSIZE); fprintf(stderr, "read %d bytes\n", r); if (r <= 0) break; } } ---- and this command line: > dd bs=1 if=/dev/zero| ./reader The output of this on RELENG_7 is: read 8764 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes read 1 bytes ... The first value puzzles me - so it actually is doing some kind of buffering. Linux isn't actually much better, but the intention is there: $ dd if=/dev/zero bs=1 | ./bla read 1 bytes read 38 bytes read 8 bytes read 2 bytes read 2 bytes read 2 bytes read 2 bytes read 4 bytes read 3 bytes read 2 bytes read 2 bytes read 2 bytes read 2 bytes read 2 bytes read 2 bytes read 3 bytes read 3 bytes read 112 bytes read 2 bytes read 2 bytes ... Maybe FreeBSD switches between the writer and the reader too soon so the buffer doesn't get filled? Using cat (which started all this), FreeBSD consistently processes 4096 byte buffers, while Linux's sizes are all over the place - from 4 kB to 1 MB, randomly fluctuating. My goal would be (if it's possible - it might not be) to maximize coalescing in an environment where the reader does something with the data (e.g. compression) so there should be a reasonable amount of backlogged input data. But if it works in general, it may simply be that it isn't really applicable to my purpose (and I should modify the reader to read multiple blocks). Though it won't help me, I still think that modifying cat is worth it :)