From owner-freebsd-fs@FreeBSD.ORG Fri Dec 9 14:20:40 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65AF81065675 for ; Fri, 9 Dec 2011 14:20:40 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mail.egr.msu.edu (gribble.egr.msu.edu [35.9.37.169]) by mx1.freebsd.org (Postfix) with ESMTP id 268A58FC15 for ; Fri, 9 Dec 2011 14:20:39 +0000 (UTC) Received: from gribble (localhost [127.0.0.1]) by mail.egr.msu.edu (Postfix) with ESMTP id 046EA735E6 for ; Fri, 9 Dec 2011 09:20:39 -0500 (EST) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mail.egr.msu.edu ([127.0.0.1]) by gribble (gribble.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4ZtdYQPu3F2f for ; Fri, 9 Dec 2011 09:20:38 -0500 (EST) Received: from [35.9.44.65] (daemon.egr.msu.edu [35.9.44.65]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: mcdouga9) by mail.egr.msu.edu (Postfix) with ESMTPSA id A5959735DC for ; Fri, 9 Dec 2011 09:20:38 -0500 (EST) Message-ID: <4EE21936.6020502@egr.msu.edu> Date: Fri, 09 Dec 2011 09:20:38 -0500 From: Adam McDougall User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4EE118C7.8030803@internet2.edu> <4EE12632.4070309@internet2.edu> In-Reply-To: <4EE12632.4070309@internet2.edu> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: ZFS hangs with 8.2-release X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Dec 2011 14:20:40 -0000 On 12/08/11 16:03, Dan Pritts wrote: > Upgrading is our intent...IF we stay with FreeBSD. Thus my question > about stability improvements in freebsd 9. > > Which I guess you've answered; we'll give it a go. > > thanks > danno >> Freddie Cash >> December 8, 2011 3:29 PM >> >> With a pool that big, you really should upgrade to 8-STABLE or >> 9-STABLE. Both of those support ZFSv28. You don't need to upgrade >> the pool/filesystems to ZFSv28, but the new code is much more stable >> and speedy. Plus, there are a lot of nice extra features in ZFSv28 >> compared to ZFSv15. >> > Some comments on your loader.conf based on experience for stability on my ZFS servers (I have not used zfs send/rec in years and generally don't use snapshots): - Ever since near the timeframe of ZFS v13 and some memory code improvements, I've been able to get ZFS to run in a stable manner if I make sure ZFS has enough ARC in a non-fragmented kmem space. Frequent ARC allocations can cause the ARC to become fragmented inside the virtual kmem space leading to slowness or stalls/panics in extreme cases. Additionally there is a bug in the code that adjusts the kmem_size based on the vm.kmem_size loader variable that you are setting; chances are if you inspect 'sysctl vm.kmem_size' you will find it considerably smaller than the 8G you set it to. I suggest: - set vm.kmem_size to double your ram to give the ARC plenty of elbow room against becoming fragmented within kmem (its a virtual address space, it is not constrained to your ram size). - to fix the bug in the kmem_size setting, edit /usr/src/sys/kern/kern_malloc.c and change cnt.v_page_count to mem_size in this code (it was recently committed to HEAD and is scheduled to be merged into other branches): vm_kmem_size = 2 * cnt.v_page_count * PAGE_SIZE; - I would still recommend setting vfs.zfs.arc_max to 2-4G if you can spare the ram unless it causes the system to starve other necessary functions or cause it to swap. I have seen ZFS speed CRAWL if the ARC is pushed too small, say 600m or under. - I wouldn't bother setting the arc_min unless you are trying to nudge it to use more ram than whatever it has chosen to use. - I wouldn't bother setting vm.kmem_size_max, it is HUGE by default - In my experience running with prefetch disabled is a significant impact to speed, once you are comfortable with doing some performance testing I would evaluate that and decide for yourself about "some discussion suggests that the prefetch sucks" - Be wary of using dedupe in v28, it seems to have a huge performance drag when working with files that were written while dedupe was enabled; I won't comment more on that except to suggest not adding that variable to your issue - These comments mostly relate to speed, but I had to give the ARC enough room to work without deadlocking the system so they may help you there.