From owner-freebsd-fs@FreeBSD.ORG Wed Dec 24 04:07:42 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A7911065673 for ; Wed, 24 Dec 2008 04:07:42 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mx.egr.msu.edu (surfnturf.egr.msu.edu [35.9.37.164]) by mx1.freebsd.org (Postfix) with ESMTP id 5CEE38FC18 for ; Wed, 24 Dec 2008 04:07:42 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from localhost (localhost [127.0.0.1]) by mx.egr.msu.edu (Postfix) with ESMTP id C80AC71EDBC; Tue, 23 Dec 2008 22:48:12 -0500 (EST) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mx.egr.msu.edu ([127.0.0.1]) by localhost (surfnturf.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WA+uYhJEBLYr; Tue, 23 Dec 2008 22:48:12 -0500 (EST) Received: from localhost (daemon.egr.msu.edu [35.9.44.65]) by mx.egr.msu.edu (Postfix) with ESMTP id 918A371ED5E; Tue, 23 Dec 2008 22:48:12 -0500 (EST) Received: by localhost (Postfix, from userid 21281) id 540B7B6A; Tue, 23 Dec 2008 22:48:12 -0500 (EST) Date: Tue, 23 Dec 2008 22:48:12 -0500 From: Adam McDougall To: Matt Simerson Message-ID: <20081224034812.GR87625@egr.msu.edu> References: <22C8092E-210F-4E91-AA09-CFD38966975C@spry.com> <494AE6F4.30506@modulus.org> <1424BEB3-69FE-4BA2-884F-4862B3D7BCFD@corp.spry.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1424BEB3-69FE-4BA2-884F-4862B3D7BCFD@corp.spry.com> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS performance gains real or imaginary? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Dec 2008 04:07:42 -0000 On Tue, Dec 23, 2008 at 12:43:47PM -0800, Matt Simerson wrote: On Dec 18, 2008, at 4:12 PM, Andrew Snow wrote: >> If so, then I really should be upgrading my production ZFS servers >> to the latest -HEAD. > > Thats correct, that is the only way to get the best working version > of ZFS. Of course, then everything is unstable and broken - eg. > SMBFS became unusable for me and would crash the server. I got bit by the ARP bug (kern/129730) which really annoyed our network manager. After applying a patch for that, I've got a working kernel and upgraded ZFS. Unfortunately, the newer kernel hangs much more frequently. Previously I was getting nearly a month between reboots. Now I don't even get 1 day between hangs. Worse yet, I upgraded the ZFS version of the pools, to see if that would make any difference. It did not, and now I can't revert. :-( I have these settings in /boot/loader.conf vm.kmem_size="1536M" vm.kmem_size_max="1536M" vfs.zfs.arc_max="100M" I have also experimented with vfs.zfs.prefetch_disable, vfs.zfs.arc_min in the past, and I'm open to suggestions on what might help under this workload (multiple concurrent rsync processes from remote systems to this one). Matt Specs: SuperMicro 24 disk, 8-core Xeon, 16GB RAM, 2 x 12 disk HW RAID striped (RAID 0). back01# uname -a FreeBSD back01.int.spry.com 8.0-CURRENT FreeBSD 8.0-CURRENT #2: Fri Dec 19 15:37:12 PST 2008 root@back01.int.spry.com:/usr/obj/usr/src/ sys/BACK01 amd64 Can you try: vm.kmem_size=2G vm.kmem_size_max=2G vfs.zfs.arc_max=512M This has been working for me on one amd64 system that only has 2G of ram but had similar problem frequency to yours. I don't know if its coincidence with the data that I am rsyncing lately, but: 10:47PM up 22 days, 7:12