From owner-freebsd-questions@FreeBSD.ORG Wed Oct 22 05:35:47 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD8D81065683 for ; Wed, 22 Oct 2008 05:35:47 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA03.emeryville.ca.mail.comcast.net (qmta03.emeryville.ca.mail.comcast.net [76.96.30.32]) by mx1.freebsd.org (Postfix) with ESMTP id A83B38FC0C for ; Wed, 22 Oct 2008 05:35:47 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from OMTA08.emeryville.ca.mail.comcast.net ([76.96.30.12]) by QMTA03.emeryville.ca.mail.comcast.net with comcast id VdW21a0010FhH24A3hbnX1; Wed, 22 Oct 2008 05:35:47 +0000 Received: from koitsu.dyndns.org ([69.181.141.110]) by OMTA08.emeryville.ca.mail.comcast.net with comcast id Vhbl1a00J2P6wsM8UhbmWH; Wed, 22 Oct 2008 05:35:46 +0000 X-Authority-Analysis: v=1.0 c=1 a=6I5d2MoRAAAA:8 a=QycZ5dHgAAAA:8 a=quzWZ-RWUDyiXxjVbL4A:9 a=jrrQAfUEdi3uCYP9G2YA:7 a=vrHdybAb9OFAw7tsqOSIdcX82tsA:4 a=EoioJ0NPDVgA:10 a=LY0hPdMaydYA:10 Received: by icarus.home.lan (Postfix, from userid 1000) id 7F298C9432; Tue, 21 Oct 2008 22:35:45 -0700 (PDT) Date: Tue, 21 Oct 2008 22:35:45 -0700 From: Jeremy Chadwick To: Anthony Chavez Message-ID: <20081022053545.GA49412@icarus.home.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-questions@freebsd.org Subject: Re: UFS2 or ZFS for 3TB disk? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Oct 2008 05:35:48 -0000 On Tue, Oct 21, 2008 at 11:06:07PM -0600, Anthony Chavez wrote: > The subject says it all. > > I will soon be installing an Areca ARC-1110 and 3x 1.5TB Seagate > Barracuda SATAs into a 3.2GHz Northwood P4 with 1GB of RAM, and I'm > wondering which would be the most stable filesystem to use. > > I've read the bigdisk page [1] and the various information about ZFS on > the FreeBSD Wiki [2]. I'm aware of the tuning requirements that ZFS > requires, and upgrading to 4GB of RAM would be quite possible as it was > understood beforehand that ZFS requires a large quantity of it. The official Solaris ZFS Wiki answers this question. The "I really don't want to this to get pedantic" answer is: caching. > 1. I'm aware of the fact that ZFS works better on 64-bit platforms, and > that alone has me thinking that it's not a good fit for this particular > machine. But apart from that, it seems that ZFS is not yet stable > enough for my environment (only about 25 users but in production > nonetheless). To me, [3] paints all sorts of ugly pictures, which can > be summarized as "count on ZFS-related panics and deadlocks happening > fairly regularly" and "disabling ZIL in the interest of stability will > put your data at risk." Comments about live systems using ZFS (on > 7.0-RELEASE or 7-STABLE) would be appreciated. You should review the past 3-4 months of freebsd-fs and freebsd-stable mailing list posts. Stabilising ZFS has been thoroughly discussed there. http://lists.freebsd.org/pipermail/freebsd-fs/ http://lists.freebsd.org/pipermail/freebsd-stable/ To date I've only heard of one person having to disable the ZIL to get stability on RELENG_7. Everyone else has followed the "standard" tuning procedures, which is to tune kmem_size, kmem_size_max, arc_min, and arc_max. prefetch_disable being set can also help performance-wise (many of us find that disabling prefetching provides less "stalls" or interruptions during extreme disk I/O). Remember that kmem_size and kmem_size_max cannot be increased past 2GB (and you should NOT set them to 2048M!) on RELENG_7 (i386 and amd64). If you want more memory made available to the kernel, you'll need to run CURRENT. None of this affects userland. What you need to keep in mind is that you need to tune things to get ZFS stable on RELENG_7, and you will need to keep tune it if you experience panics relating to ZFS or kmem exhaustion. The way to avoid panics is to start with something very restrictive (I advocate 1536M for kmem_size and kmem_size_max, 16M for arc_min, and 64M of arc_max, on a machine with 2GB of RAM), and then gradually increase arc_max by ~16M increments. You cannot tune this, reboot, let it run for 24 hours and go "IT WORKS!!!!!!!!". You will need to seriously test this over a period of many days, if not weeks. Each person's environment is slightly different, and it depends greatly upon the type of data being accessed, how often, and what sort of workload. There is no "standard". I use ZFS to manage a raidz1 zpool across 4 500GB disks (all SATA) on my home machine, which gets thrashed pretty hard due to the content sizes (500-600MB files). This machine has 2GB of RAM, and runs amd64. I also use it in my co-lo as a replacement for /home and on another machine to be used for backups (rsnapshot+rsync across ssh, results stored on a ZFS filesystem). This machine has 2GB of RAM, and runs i386. The "it works better on amd64 platforms" is true, but there is absolutely nothing that stops you from using it on i386. You should feel confident using it on both. If you really want to know of the performance difference of ZFS on amd64 vs. i386, assuming Google has nothing, you might ask pjd@ directly. I think the general deal is that ZFS was developed with 64-bit pointers and 64-bit values in mind, and while those are not entirely available on x86 (sans MMX/SSE/SSE2/SSE3 registers) without a bit more work, "long long" on i386 works fine. IMHO, you should be using ZFS for what you want to accomplish. If you do go the UFS2 route, I would recommend you set background_fsck="no" in your rc.conf, especially since it's obvious data integrity matters a lot to you (but then again, if it does matter that much, ZFS's checksum ability should trump your interest in UFS2 entirely). That's really all I'm going to say on the matter. (Sorry if that sounds rude, I'm just incredibly scared that my above comments will induce some pedantic flame war or battle of sort, which is not my intention.) > [1] http://freebsd.org/projects/bigdisk > [2] http://wiki.freebsd.org/ZFS > [3] http://wiki.freebsd.org/ZFSKnownProblems -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |