From owner-freebsd-hackers Tue Apr 24 2:32:13 2001 Delivered-To: freebsd-hackers@freebsd.org Received: from wit379112.student.utwente.nl (wit379119.student.utwente.nl [130.89.232.129]) by hub.freebsd.org (Postfix) with ESMTP id 9DB5C37B422 for ; Tue, 24 Apr 2001 02:32:10 -0700 (PDT) (envelope-from niek@wit379112.student.utwente.nl) Received: by wit379112.student.utwente.nl (Postfix, from userid 1000) id EBF025D38; Tue, 24 Apr 2001 11:35:36 +0200 (CEST) Date: Tue, 24 Apr 2001 11:35:36 +0200 From: Niek Bergboer To: Mike Silbersack Cc: freebsd-hackers@freebsd.org Subject: Re: UFS block size vs. write speed Message-ID: <20010424113536.A61988@wit379119.student.utwente.nl> Reply-To: niek@bergboer.net References: <20010423134554.A57241@wit379119.student.utwente.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from silby@silby.com on Mon, Apr 23, 2001 at 12:09:26PM -0500 Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Mon, Apr 23, 2001 at 12:09:26PM -0500, Mike Silbersack wrote: > On Mon, 23 Apr 2001, Niek Bergboer wrote: > > I've implemented a quick hack on the BSD ftp-client: in the original > > recv-file function data is read from a socket into a buffer, which is > > then written to a file. I've mmap-ed the file, and rather than reading > > from the socket into the buffer, I read directly from the socket into > > the mmaped region. I use the MAP_SHARED and MAP_NOSYNC flags, and > > especially the latter makes a huge difference. > > I seem to remember Matt Dillon saying that if you extend a file through > mmap'd writes, it will become very fragmented. Check the archives for > this mailing list during decemeber or so, I believe the thread had > something to so with usenet servers. Yes, the file gets fragmented indeed because I first extend it using ftruncate() and then mmap() it. The fragmentation gets especially bad when the MAP_ASYNC flag is passed to mmap(). However, in order to get a "write-cache"-like system I need to use MAP_ASYNC. Now, what I'm wondering about is the following: when one uses ftruncate() to extend a file, _no_ blocks get allocated in the filesystem because all blocks are empty (it was nice to see that I was actually able to make a 800 GB "file" on a 15 GB harddisk). This non-allocation seems to be the problem: if more than one process is writing, the blocks will have to be allocated dynamically as one writes into the mmap()'ed area. So is it possible to _allocate_ the blocks without actually _writing_ them full of zeros? If ftruncate() can do this, you would only have to write a very small amount of meta-data to the filesystem in order to allocate a non-fragmented set of blocks. In fact, I couldn't care less if the allocated blocks contain random data (rather than zeros), since I'll be overwriting them immediately. > Mike "Silby" Silbersack Niek Bergboer -- Conscience doth make cowards of us all. -- Shakespeare To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message