From owner-freebsd-stable@FreeBSD.ORG Tue Sep 30 04:32:52 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D926D106568C for ; Tue, 30 Sep 2008 04:32:52 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA03.emeryville.ca.mail.comcast.net (qmta03.emeryville.ca.mail.comcast.net [76.96.30.32]) by mx1.freebsd.org (Postfix) with ESMTP id B75398FC0C for ; Tue, 30 Sep 2008 04:32:52 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from OMTA13.emeryville.ca.mail.comcast.net ([76.96.30.52]) by QMTA03.emeryville.ca.mail.comcast.net with comcast id LsSc1a00517UAYkA3sYsY1; Tue, 30 Sep 2008 04:32:52 +0000 Received: from koitsu.dyndns.org ([67.180.253.227]) by OMTA13.emeryville.ca.mail.comcast.net with comcast id LsYq1a00J4v8bD78ZsYrGU; Tue, 30 Sep 2008 04:32:52 +0000 X-Authority-Analysis: v=1.0 c=1 a=TxirYYpeSEAA:10 a=QO6ccaido9wA:10 a=QycZ5dHgAAAA:8 a=-yhJEkN_MDzKlqzCys4A:9 a=edlQyZZIXfq769eLMXgA:7 a=7uWRdPfgoIcZitXqQ7ZnRmYUFrEA:4 a=EoioJ0NPDVgA:10 a=LY0hPdMaydYA:10 Received: by icarus.home.lan (Postfix, from userid 1000) id D732CC9419; Mon, 29 Sep 2008 21:32:50 -0700 (PDT) Date: Mon, 29 Sep 2008 21:32:50 -0700 From: Jeremy Chadwick To: Andrew Snow Message-ID: <20080930043250.GA36878@icarus.home.lan> References: <765067435.20080926223557@takeda.tk> <20080927064417.GA43638@icarus.home.lan> <588787159.20080927003750@takeda.tk> <5f67a8c40809282030l7888d942q548d570cd0b33be9@mail.gmail.com> <20080929040025.GA97332@icarus.home.lan> <48E080C0.9070103@modulus.org> <5f67a8c40809290809j58639df8ka65184151161cab6@mail.gmail.com> <5f67a8c40809290849m413eebe6sd31a493aea506932@mail.gmail.com> <200809291744.m8THiBlR034739@apollo.backplane.com> <48E1839E.3060006@modulus.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <48E1839E.3060006@modulus.org> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-stable@freebsd.org Subject: Re: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Sep 2008 04:32:52 -0000 On Tue, Sep 30, 2008 at 11:40:46AM +1000, Andrew Snow wrote: > > Matthew Dillon wrote: >> It can take 6 hours to fsck a full 1TB HD. It can >> take over a day to fsck larger setups. Putting in a few sleeps here >> and there just makes the run time even longer and perpetuates the pain. > > We have a box with millions of files spread over 2TB, on a 16 disk RAID. > Foreground fsck takes almost 8 hours, so background fsck, which takes > almost 24 hours or more, is my only option when I want to bring the box > back online quickly. And UFS Snapshots are so slow as to be completely > useless. > > I've now converted the volume to ZFS, and am now enjoying instant boot > time and higher speed I/O under heavy load, at the expense of memory > consumption. > >> My recommendation? Default UFS back to a synchronous fsck and stop >> treating ZFS (your only real alternative) as being so ultra-alpha that >> it shouldn't be used. > > Completely agree. ZFS is the way of the future for FreeBSD. In my > latest testing, the memory problems are now under control, there is just > stability problems with random lockups after days of heavy load unless I > turn off ZIL. So its nearly there. It just now occurred to me that this entire conversation should've been moved to freebsd-fs weeks ago. *laugh* Oh well. :-) You're the first person I've encountered who has had to disable the ZIL to get stability in ZFS; ouch, that must hurt. ZFS stability has been discussed on freebsd-fs numerous times, but the answers provided are always penultimate; no one (AFAIK) has examined how to solve this from the start (specifically new FreeBSD installations). Yes, I know sysinstall/sade doesn't support ZFS (though the PC-BSD folks have apparently implemented this), but that's not what I'm talking about. I'm talking about the most commonly-encountered problem: kmem exhaustion. People want to be able to install FreeBSD then say "Okay! Time to give ZFS a try!" on some separate disks, and have it work. They don't want to encounter kmem exhaustion half way through the migration process; that's just going to dishearten them. I'll be starting up a new topic on freebsd-fs later tonight with an idea I came up with for solving this out-of-the-box. I have a feeling I'm going to get told "so who's going to do all the work?" or downright flamed, but I hope it induces a discussion of ideas, specifically with regards to new FreeBSD installations. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |