From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 04:46:38 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 57E5C16A4CE for ; Wed, 5 Jan 2005 04:46:38 +0000 (GMT) Received: from mail03.syd.optusnet.com.au (mail03.syd.optusnet.com.au [211.29.132.184]) by mx1.FreeBSD.org (Postfix) with ESMTP id 77DED43D39 for ; Wed, 5 Jan 2005 04:46:37 +0000 (GMT) (envelope-from PeterJeremy@optushome.com.au) Received: from cirb503493.alcatel.com.au (c211-30-75-229.belrs2.nsw.optusnet.com.au [211.30.75.229]) j054kUaf032622 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Wed, 5 Jan 2005 15:46:31 +1100 Received: from cirb503493.alcatel.com.au (localhost.alcatel.com.au [127.0.0.1])j054kUxP038432; Wed, 5 Jan 2005 15:46:30 +1100 (EST) (envelope-from pjeremy@cirb503493.alcatel.com.au) Received: (from pjeremy@localhost)j054kQKh038431; Wed, 5 Jan 2005 15:46:26 +1100 (EST) (envelope-from pjeremy) Date: Wed, 5 Jan 2005 15:46:26 +1100 From: Peter Jeremy To: Julian Elischer Message-ID: <20050105044626.GI34222@cirb503493.alcatel.com.au> References: <20050104224043.GM784@darkness.comp.waw.pl> <41DB2B24.6050005@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <41DB2B24.6050005@elischer.org> User-Agent: Mutt/1.4.2i cc: arch@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 04:46:38 -0000 On Tue, 2005-Jan-04 15:47:48 -0800, Julian Elischer wrote: >One thing that needs to be done is an 2ndary storage fsck. >that doesn't try put everything in RAM. Agreed but VM is different to physical RAM. >Basically this will mean extracting all the metadata from filesystems into >files and running sort operations of various kinds on them >to order the data in ways that allows consistencies to be checked. How do you determine where the disk space comes from? You can't safely use the filesystem being checked because you can't safely write to it until the fsck has completed. _Any_ filesystem picked automatically may not have sufficient free space to do an fsck. (Older Unices prompted for a temporary directory for work files). Overall, the most likely location for free space is swap. I'd therefore suggest that rather than moving back to using temporary files, we look at how to better use swap space: - Look into reducing VM requirements (600MB per TB seems high) - Tweak data structures to maximise locality of reference (traversing linked lists is extremely slow when your list won't fit into RAM) - Give fsck the ability to split into multiple processes to avoid process size limits - usr socketpair()s to allow a "master" process to delegate work ala dump(8). The downside of using swap is the possibility of over-writing a crash dump. -- Peter Jeremy