From owner-freebsd-stable@FreeBSD.ORG Sat Mar 25 21:27:40 2006 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 697F916A420; Sat, 25 Mar 2006 21:27:40 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail07.syd.optusnet.com.au (mail07.syd.optusnet.com.au [211.29.132.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1485943D45; Sat, 25 Mar 2006 21:27:33 +0000 (GMT) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-19-236.belrs4.nsw.optusnet.com.au [220.239.19.236]) by mail07.syd.optusnet.com.au (8.12.11/8.12.11) with ESMTP id k2PLRLov018653 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Sun, 26 Mar 2006 08:27:22 +1100 Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.13.4/8.13.4) with ESMTP id k2PLRLY7007851; Sun, 26 Mar 2006 08:27:21 +1100 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.13.4/8.13.4/Submit) id k2PLRKl3007850; Sun, 26 Mar 2006 08:27:20 +1100 (EST) (envelope-from peter) Date: Sun, 26 Mar 2006 08:27:20 +1100 From: Peter Jeremy To: Mikhail Teterin Message-ID: <20060325212720.GI703@turion.vk2pj.dyndns.org> References: <200603232352.k2NNqPS8018729@gate.bitblocks.com> <200603241518.01027.mi+mx@aldan.algebra.com> <20060325103927.GE703@turion.vk2pj.dyndns.org> <200603250920.14208@aldan> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200603250920.14208@aldan> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.11 Cc: alc@freebsd.org, Mikhail Teterin , stable@freebsd.org Subject: Re: Reading via mmap stinks (Re: weird bugs with mmap-ing via NFS) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 21:27:40 -0000 On Sat, 2006-Mar-25 09:20:13 -0500, Mikhail Teterin wrote: >I'm sorry, that should be http://aldan.algebra.com/~mi/mzip.c -- I checked >this time :-( It doesn't look like it's doing anything especially weird. As Matt pointed out, creating files with mmap() is not a good idea because the syncer can cause massive fragmentation when allocating space. I can't test is as-is because it insists on mmap'ing its output and I only have one disk and you can't mmap /dev/null. Since your program is already written to mmap the input and output in pieces, it would be trivial to convert it to use read/write. >= I tried writing a program that just mmap'd my entire (2GB) test file >= and summed all the longwords in it. > >The files I'm dealing with are database dumps -- 10-80Gb :-) Maybe, that's, >what triggers some pessimal case?.. I tried generating an 11GB test file and got results consistent with my previous tests: grep using read or mmap, as well as mmap'ing the entire file give similar times with the disk mostly saturated. I suggest you try converting mzip.c to use read/write and see if the problem is still present. -- Peter Jeremy