From owner-freebsd-hackers@FreeBSD.ORG Fri Jan 9 11:35:58 2004 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4388116A4CE for ; Fri, 9 Jan 2004 11:35:58 -0800 (PST) Received: from beelzebubba.sysabend.org (alcatraz.inna.net [209.201.74.14]) by mx1.FreeBSD.org (Postfix) with ESMTP id E098943D1F for ; Fri, 9 Jan 2004 11:35:56 -0800 (PST) (envelope-from xyzzy@moo.sysabend.org) Received: from moo.sysabend.org (moo.sysabend.org [66.111.41.70]) by beelzebubba.sysabend.org (Postfix) with SMTP id B47FF14E99 for ; Fri, 9 Jan 2004 14:36:01 -0500 (EST) Received: (nullmailer pid 69073 invoked by uid 14); Fri, 09 Jan 2004 19:35:51 -0000 Date: Fri, 9 Jan 2004 11:35:51 -0800 From: Tom Arnold To: freebsd-hackers@freebsd.org Message-ID: <20040109193551.GD39751@moo.sysabend.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4i Organization: The Sysabend Dump X-Operating-System: CPM2.2 X-Bucket-Brigade-Devices: Rah! Subject: Large Filesystem Woes X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: xyzzy@sysabend.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2004 19:35:58 -0000 I previously posted this on -fs but got no responce so I'm trying -hackers. Building a box thats going to house many billions of small files. Think innd circa 1998 or someone trying to house AOLs mail system on cyrus or something. To this end I've hung a 3.3TB hardware raid off a BSD box broken into 4 partitions. 3 1TB and 1 300GB. Originally this was on a 4.9 box. da0s1 and da0s2 were formatted "stock" ( -f 2048 -b 16384 -i 8192 ) da1s1 and s2 were both formatted -f 512 -b 4096 -i 512. Needless to say fsck took a while if the box came up dirty. Switched to 5.2. Newfs'd the RAID for UFS2. First issue, if the machine came up dirty, bgfsck seemed to do its thing and the machine was online and usable after about 20 minutes however after a few hours I get this error : fsck: /dev/da1s1e: CANNOT CREATE SNAPSHOT /export/database/.snap/fsck_snapshot: File too large fsck: /dev/da1s1e: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. And the second thing I've noticed is I have lost a lot of space. Under 4.9 with UFS da1s1e was approx 870gigs and s2e was around 180, now I see : Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/da0s1e 992G 4.0K 912G 0% 2 134411260 0% /export/logs1 /dev/da0s2e 992G 4.0K 912G 0% 2 134411260 0% /export/logs2 /dev/da1s1e 510G 1.0K 469G 0% 2 2148661228 0% /export/database /dev/da1s2e 94G 1.0K 86G 0% 2 395214332 0% /export/spare I'm not certain if I've run into some kind of weird limit here or a bug or what and am looking for ideas to persue before I'm stuck going to an OS with something journaled. Thanks! -- ------------------------------------------------------------------------ - Tom Arnold - When I was small, I was in love, - - Sysabend - In love with everything. - - CareTaker - And now there's only you... - -------------- -- Thomas Dolby, "Cloudburst At Shingle Street" -