From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 05:56:46 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37F661065679; Wed, 9 Jul 2008 05:56:46 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 221508FC17; Wed, 9 Jul 2008 05:56:45 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 847061CC081; Tue, 8 Jul 2008 22:56:45 -0700 (PDT) Date: Tue, 8 Jul 2008 22:56:45 -0700 From: Jeremy Chadwick To: CZUCZY Gergely Message-ID: <20080709055645.GA40076@eos.sc1.parodius.com> References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <20080708221327.5c1d0e92@mort.in.publishing.hu> <4873CF6C.7000205@FreeBSD.org> <20080708225449.1070252d@mort.in.publishing.hu> <4873F4E9.3040203@FreeBSD.org> <20080709074420.24df3be4@mort.in.publishing.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080709074420.24df3be4@mort.in.publishing.hu> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org, Kris Kennaway Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 05:56:46 -0000 On Wed, Jul 09, 2008 at 07:44:20AM +0200, CZUCZY Gergely wrote: > On Wed, 09 Jul 2008 01:14:49 +0200 > Kris Kennaway wrote: > > > CZUCZY Gergely wrote: > > I don't know; empirically my setup is an upper bound. How large was > > "as large as it was allowed" for you? > Well, we cannot buy "upper bounds" all over, just because some > developer is unable to figure out things. I think you can't expect > FreeBSD users to spend as much money as possible, just because the devs > can't tell how much is enough... > It seems more like a twilight zone then a stable feature now ;) > > It was exactly as much as an amd64 installation would allow with 2GB of > physical memory. We've dismissed the setup around february, and I don't > have the configs anymore. It was an amd64 setup with 2GB of physical > memory. The bottom line here is that i386 and amd64 both have a kmem_size limit of 2GB. You can throw 32GB of RAM into an amd64 box, but FreeBSD will only utilise up to 2GB of that for kmem. That is purely a FreeBSD limitation, and is being dealt with in HEAD by Alan Cox. I believe he has a patch, or it may have been committed -- I don't follow HEAD. I can point people to a mailing list URL, if needed. This is one of the limitations Gergely is referring to. Since ZFS is incredibly memory-hungry, you're forced to tune ZFS to try and get it to "play nice" with that 2GB limit on STABLE/RELEASE systems. You also need to keep in mind that you can't just set kmem_size and kmem_size_max to 2048M, because the kernel needs memory for other things. The tuning parameters I use on my 2GB amd64 and 4GB amd64 boxes are: vm.kmem_size="1536M" vm.kmem_size_max="1536M" vfs.zfs.arc_min="16M" vfs.zfs.arc_max="64M" If you set kmem_size and kmem_size_max any higher than that, the machine will panic on boot, stating (indirectly) that there isn't enough memory available for the kernel to allocate for other things. Until I added the arc_min and arc_max setting, I could occasionally panic the machines under very heavy load (heavy zpool I/O), caused by kmem exhaustion. Since adding the arc_* tunings, I've tried very hard to crash the machines, and I cannot. But there's absolutely no guarantee those tuning parameters above will ensure FreeBSD won't panic due to kmem exhaustion. I believe this is the point Gergely is making about the "stability" of the whole thing. Now, with regards to prefetch_disable, folks can disable that if they want. I disable it on my above systems because for what they do, the overall performance appears better with prefetching disabled. I hope this helps shed some light here... -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |