From owner-freebsd-fs@FreeBSD.ORG Wed Jul 30 12:38:31 2003 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1312937B401 for ; Wed, 30 Jul 2003 12:38:31 -0700 (PDT) Received: from rwcrmhc12.comcast.net (rwcrmhc12.comcast.net [216.148.227.85]) by mx1.FreeBSD.org (Postfix) with ESMTP id 96DE943F93 for ; Wed, 30 Jul 2003 12:38:30 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (rwcrmhc12) with ESMTP id <2003073019382501400aoae8e>; Wed, 30 Jul 2003 19:38:25 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id MAA24176; Wed, 30 Jul 2003 12:38:23 -0700 (PDT) Date: Wed, 30 Jul 2003 12:38:21 -0700 (PDT) From: Julian Elischer To: Brooks Davis In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: fs@freebsd.org Subject: Re: fsck memory usage X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2003 19:38:31 -0000 On Wed, 30 Jul 2003, Julian Elischer wrote: > > > On Wed, 30 Jul 2003, Brooks Davis wrote: > > > Does anyone have a formula to give fsck memory requirements based on > > disk size and newfs arguments. I'm going to be building some large > > (3.3TB) file systems and I'd like to figure out if I'm going to need > > to go 64-bit to have sufficent address space to fsck them. The mean > > filesize will be pretty big so I will be able to set bytes/i-node > > fairly high. > > I needed 700MB of memory in teh fsck process to be able to fsck a 1 TB > filesystem. > (actually that's what I needed for pass 1.. I ended up > splitting it into 2 smaller ones after fsck's > first 2 hours only got to pass 2..) > I came to the conclusion that the maximum you could fsck on a 32 bit machine was somewhere around 3TB (maybe a bit less) if you had 4GB of ram in the machine and a 2GB virtual space for the process I think it should be possible to produce a 'streaming' fsck that uses files of orderred metadata (i.e. sorted records) to check much larger filesystems, but you'd need a filesystem set asside specifically to hold the workfiles as they'd be bretty damned big.. Think of the old tape based mechanisms that used tbe done when tapes were bigger than ram.. e.g. merge sorts and similar.. files would be made containing all referenced inode and blocks and directorie entries etc. and then they would eb sorted into approriate orders (possible several) and a 'merge/select' pass woudl be made on them to generate output files containing lists of 'good' and 'bad' references etc.