Date: Tue, 04 Jan 2005 20:35:02 -0700 (MST) From: "M. Warner Losh" <imp@bsdimp.com> To: julian@elischer.org Cc: pjd@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. Message-ID: <20050104.203502.85411551.imp@bsdimp.com> In-Reply-To: <41DB2B24.6050005@elischer.org> References: <20050104224043.GM784@darkness.comp.waw.pl> <41DB2B24.6050005@elischer.org>
next in thread | previous in thread | raw e-mail | index | archive | help
In message: <41DB2B24.6050005@elischer.org>
Julian Elischer <julian@elischer.org> writes:
:
:
: Pawel Jakub Dawidek wrote:
:
: >Hi.
: >
: >I want you to look at two patches which makes du(1) 64bit clean.
: >This work is part of the BigDisk project:
: >
: > http://www.freebsd.org/projects/bigdisk/
: >
: >
: One thing that needs to be done is an 2ndary storage fsck.
: that doesn't try put everything in RAM.
: Basically this will mean extracting all the metadata from filesystems into
: files and running sort operations of various kinds on them
: to order the data in ways that allows consistencies to be checked.
: It will take a bit longer than a RAM fsck but maybe not as much as
: one might fear.
: We all remember those "sort a mag-tape larger than RAM"
: lessons from CS101 don't we?
: At least it doesn't have to be "in place" so merge sorts are OK. :-)
:
: why?
:
: A bitmap of 1TB of 512 byte records is 244MB so with a 4BG machine
: with 3GB available to the process you can't even fit the bitmaps into
: memory for a 12TB Filesystem let alone other metadata.
:
: Going to 2048 byte frags helps but you still run into a limit.
: last I tried it, you need about 600MB per TB of fileysstem to check.
:
: So I think a special fsck that uses files is a must for really big
: filesystems, unless they (the filesystems) can be broken up in
: a logical way (IBM did that many years ago I believe).
: I think you should add that to your list.
I think that a big amount of this could be reduced by using simple
arrays rather than lists which are more memory efficient...
Warner
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050104.203502.85411551.imp>