From owner-freebsd-fs@FreeBSD.ORG Tue Dec 23 20:43:50 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A9CA810656AA for ; Tue, 23 Dec 2008 20:43:50 +0000 (UTC) (envelope-from matt@corp.spry.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.172]) by mx1.freebsd.org (Postfix) with ESMTP id 865EC8FC1E for ; Tue, 23 Dec 2008 20:43:50 +0000 (UTC) (envelope-from matt@corp.spry.com) Received: by wf-out-1314.google.com with SMTP id 24so4399011wfg.7 for ; Tue, 23 Dec 2008 12:43:50 -0800 (PST) Received: by 10.142.103.2 with SMTP id a2mr3289679wfc.164.1230065029491; Tue, 23 Dec 2008 12:43:49 -0800 (PST) Received: from imac24.simerson.net (c-67-168-10-190.hsd1.wa.comcast.net [67.168.10.190]) by mx.google.com with ESMTPS id 9sm29340525wfc.56.2008.12.23.12.43.48 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 23 Dec 2008 12:43:48 -0800 (PST) Message-Id: <1424BEB3-69FE-4BA2-884F-4862B3D7BCFD@corp.spry.com> From: Matt Simerson To: freebsd-fs@freebsd.org In-Reply-To: <494AE6F4.30506@modulus.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v930.3) Date: Tue, 23 Dec 2008 12:43:47 -0800 References: <22C8092E-210F-4E91-AA09-CFD38966975C@spry.com> <494AE6F4.30506@modulus.org> X-Mailer: Apple Mail (2.930.3) Subject: Re: ZFS performance gains real or imaginary? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Dec 2008 20:43:50 -0000 On Dec 18, 2008, at 4:12 PM, Andrew Snow wrote: >> If so, then I really should be upgrading my production ZFS servers >> to the latest -HEAD. > > Thats correct, that is the only way to get the best working version > of ZFS. Of course, then everything is unstable and broken - eg. > SMBFS became unusable for me and would crash the server. I got bit by the ARP bug (kern/129730) which really annoyed our network manager. After applying a patch for that, I've got a working kernel and upgraded ZFS. Unfortunately, the newer kernel hangs much more frequently. Previously I was getting nearly a month between reboots. Now I don't even get 1 day between hangs. Worse yet, I upgraded the ZFS version of the pools, to see if that would make any difference. It did not, and now I can't revert. :-( I have these settings in /boot/loader.conf vm.kmem_size="1536M" vm.kmem_size_max="1536M" vfs.zfs.arc_max="100M" I have also experimented with vfs.zfs.prefetch_disable, vfs.zfs.arc_min in the past, and I'm open to suggestions on what might help under this workload (multiple concurrent rsync processes from remote systems to this one). Matt Specs: SuperMicro 24 disk, 8-core Xeon, 16GB RAM, 2 x 12 disk HW RAID striped (RAID 0). back01# uname -a FreeBSD back01.int.spry.com 8.0-CURRENT FreeBSD 8.0-CURRENT #2: Fri Dec 19 15:37:12 PST 2008 root@back01.int.spry.com:/usr/obj/usr/src/ sys/BACK01 amd64 back01# zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT back01 18.1T 16.5T 1.60T 91% ONLINE - back01# zpool status pool: back01 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM back01 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 errors: No known data errors back01# zpool upgrade This system is currently running ZFS pool version 13. All pools are formatted using this version.