From owner-freebsd-stable@FreeBSD.ORG Tue Mar 28 18:27:56 2006 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B89B916A466 for ; Tue, 28 Mar 2006 18:27:56 +0000 (UTC) (envelope-from mi+mx@aldan.algebra.com) Received: from zig.murex.com (mail.murex.com [194.98.239.11]) by mx1.FreeBSD.org (Postfix) with ESMTP id 97C044404A for ; Tue, 28 Mar 2006 18:02:09 +0000 (GMT) (envelope-from mi+mx@aldan.algebra.com) Received: from interscan.fr.murex.com (interscan.fr.murex.com [172.21.17.207] (may be forged)) by zig.murex.com with ESMTP id k2SI48Tg007449; Tue, 28 Mar 2006 20:04:08 +0200 (CEST) Received: from mxmail.murex.com (interscan.murex.fr [127.0.0.1]) by interscan.fr.murex.com (8.11.6/8.11.6) with ESMTP id k2SIXMH29436; Tue, 28 Mar 2006 20:33:22 +0200 Received: from [172.21.130.86] ([172.21.130.86]) by mxmail.murex.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 28 Mar 2006 20:01:37 +0200 From: Mikhail Teterin Organization: Virtual Estates, Inc. To: Peter Jeremy Date: Tue, 28 Mar 2006 13:01:35 -0500 User-Agent: KMail/1.9.1 References: <200603232352.k2NNqPS8018729@gate.bitblocks.com> <20060325103927.GE703@turion.vk2pj.dyndns.org> <20060328102721.GA2352@turion.vk2pj.dyndns.org> In-Reply-To: <20060328102721.GA2352@turion.vk2pj.dyndns.org> MIME-Version: 1.0 Content-Type: text/plain; charset="koi8-u" Content-Transfer-Encoding: 8bit Content-Disposition: inline Message-Id: <200603281301.36087.mi+mx@aldan.algebra.com> X-OriginalArrivalTime: 28 Mar 2006 18:01:37.0424 (UTC) FILETIME=[A850C900:01C65291] Cc: stable@freebsd.org Subject: Re: Reading via mmap stinks (Re: weird bugs with mmap-ing via NFS) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 18:27:57 -0000 в╕второк 28 березень 2006 05:27, Peter Jeremy написав: > I'd suggest that each mmap be capable of storing several hundred msec of > data as a minumum (maybe 10MB input and 5MB output, preferably more). > Synchronisation can be done by writing tokens into pipes shared with the > mmap's, optimised by sharing read/write pointers (so you only really need > the tokens when the shared buffer is full/empty). Thank you very much, Peter, for your suggestions. Unfortunately, I have no control whatsoever over the dump-ing part of the process. The dump is done by Sybase database servers -- old, clunky, and closed-source software, running on slow CPU (but good I/O) Sun hardware. You are right, of course, that my application (mzip being only part of it) needs to keep the dumper and the compressor in sync. Without any cooperation from the former, however, I see no other way but to temporarily throttle the NFS-bandwidth via firewall, when the compressor falls behind (as can be detected by the increased proportion of sys-time, I guess). Much as I apreciate the (past and future) help and suggestions, I'm not asking you, nor the mailing list to solve my particular problem here :-) I only gave the details of my need and application to illustrate a missed general optimization opportunity in FreeBSD -- reading large files via mmap need not be slower than via read. If anything, it should be (slightly) faster. After many days Matt has finally stated (admitted? ;-): read() uses a different heuristic then mmap() to implement the read-ahead. There is also code in there which depresses the page priority of 'old' already-read pages in the sequential case. There is no reason not to implement similar smarts in the mmap-handling code to similarly depress the priority of the in-memory pages in the MADV_SEQUENTIAL case, thus freeing more RAM for aggressive read-ahead. As I admitted before, actually implementing this far exceeds my own capabilities, so all I can do is pester, whoever cares, to do it instead :-) C'mon, guys... -mi