Date: Tue, 04 Jan 2005 22:57:23 -0800 From: Julian Elischer <julian@elischer.org> To: "M. Warner Losh" <imp@bsdimp.com> Cc: pjd@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. Message-ID: <41DB8FD3.3030101@elischer.org> In-Reply-To: <20050104.203502.85411551.imp@bsdimp.com> References: <20050104224043.GM784@darkness.comp.waw.pl> <41DB2B24.6050005@elischer.org> <20050104.203502.85411551.imp@bsdimp.com>
next in thread | previous in thread | raw e-mail | index | archive | help
M. Warner Losh wrote: > In message: <41DB2B24.6050005@elischer.org> > Julian Elischer <julian@elischer.org> writes: > : > : > : Pawel Jakub Dawidek wrote: > : > : >Hi. > : > > : >I want you to look at two patches which makes du(1) 64bit clean. > : >This work is part of the BigDisk project: > : > > : > http://www.freebsd.org/projects/bigdisk/ > : > > : > > : One thing that needs to be done is an 2ndary storage fsck. > : that doesn't try put everything in RAM. > : Basically this will mean extracting all the metadata from filesystems into > : files and running sort operations of various kinds on them > : to order the data in ways that allows consistencies to be checked. > : It will take a bit longer than a RAM fsck but maybe not as much as > : one might fear. > : We all remember those "sort a mag-tape larger than RAM" > : lessons from CS101 don't we? > : At least it doesn't have to be "in place" so merge sorts are OK. :-) > : > : why? > : > : A bitmap of 1TB of 512 byte records is 244MB so with a 4BG machine > : with 3GB available to the process you can't even fit the bitmaps into > : memory for a 12TB Filesystem let alone other metadata. > : > : Going to 2048 byte frags helps but you still run into a limit. > : last I tried it, you need about 600MB per TB of fileysstem to check. > : > : So I think a special fsck that uses files is a must for really big > : filesystems, unless they (the filesystems) can be broken up in > : a logical way (IBM did that many years ago I believe). > : I think you should add that to your list. > > I think that a big amount of this could be reduced by using simple > arrays rather than lists which are more memory efficient... > > Warner but that just puts the problem a bit further away.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?41DB8FD3.3030101>