Date: Tue, 04 Jan 2005 23:10:01 -0800 From: Julian Elischer <julian@elischer.org> To: Peter Jeremy <PeterJeremy@optushome.com.au> Cc: arch@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. Message-ID: <41DB92C9.50801@elischer.org> In-Reply-To: <20050105044626.GI34222@cirb503493.alcatel.com.au> References: <20050104224043.GM784@darkness.comp.waw.pl> <41DB2B24.6050005@elischer.org> <20050105044626.GI34222@cirb503493.alcatel.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
Peter Jeremy wrote: > On Tue, 2005-Jan-04 15:47:48 -0800, Julian Elischer wrote: > >>One thing that needs to be done is an 2ndary storage fsck. >>that doesn't try put everything in RAM. > > > Agreed but VM is different to physical RAM. > > >>Basically this will mean extracting all the metadata from filesystems into >>files and running sort operations of various kinds on them >>to order the data in ways that allows consistencies to be checked. > > > How do you determine where the disk space comes from? You can't > safely use the filesystem being checked because you can't safely write > to it until the fsck has completed. _Any_ filesystem picked > automatically may not have sufficient free space to do an fsck. > (Older Unices prompted for a temporary directory for work files). One assumes you also have a smaller (say 100GB temp filesystem that you can newfs on boot.. or some other similar system.. Here we have our own fsck that acts differently, and we do /var with the regular fsck first and then use it as a place to store the logfiles of the main fsck runs on our TB raids using our modified fsck. > > Overall, the most likely location for free space is swap. I'd therefore > suggest that rather than moving back to using temporary files, we look > at how to better use swap space: swap may not be big enough either.. > - Look into reducing VM requirements (600MB per TB seems high) I only report what it does.. > - Tweak data structures to maximise locality of reference (traversing > linked lists is extremely slow when your list won't fit into RAM) > - Give fsck the ability to split into multiple processes to avoid process > size limits - usr socketpair()s to allow a "master" process to delegate > work ala dump(8). still only DELAYs the problem. > > The downside of using swap is the possibility of over-writing a crash dump. >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?41DB92C9.50801>