From owner-freebsd-fs@FreeBSD.ORG Tue Jul 22 21:25:45 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 662D91065673 for ; Tue, 22 Jul 2008 21:25:45 +0000 (UTC) (envelope-from matt@corp.spry.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.174]) by mx1.freebsd.org (Postfix) with ESMTP id 505588FC0A for ; Tue, 22 Jul 2008 21:25:45 +0000 (UTC) (envelope-from matt@corp.spry.com) Received: by wf-out-1314.google.com with SMTP id 24so1436328wfg.7 for ; Tue, 22 Jul 2008 14:25:45 -0700 (PDT) Received: by 10.142.253.21 with SMTP id a21mr2019227wfi.254.1216760257618; Tue, 22 Jul 2008 13:57:37 -0700 (PDT) Received: from matts.spry.com ( [207.178.4.6]) by mx.google.com with ESMTPS id 29sm5991582wfg.0.2008.07.22.13.57.33 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 22 Jul 2008 13:57:36 -0700 (PDT) Message-Id: <5E8D64DE-EC9B-4B11-BCB4-17BA63650BB7@corp.spry.com> From: Matt Simerson To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v928.1) Date: Tue, 22 Jul 2008 13:57:27 -0700 X-Mailer: Apple Mail (2.928.1) Cc: pjd@freebsd.org Subject: ZFS hang issue and prefetch_disable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2008 21:25:45 -0000 Symptoms Deadlocks under heavy IO load on the ZFS file system with prefetch_disable=0. Setting vfs.zfs.prefetch_disable=1 results in a stable system. Configuration Two machines. Identically built. Both exhibit identical behavior. 8 cores (2 x E5420) x 2.5GHz, 16 GB RAM, 24 x 1TB disks. FreeBSD 7.0 amd64 dmesg: http://matt.simerson.net/computing/zfs/dmesg.txt Boot disk is a read only 1GB compact flash # cat /etc/fstab /dev/ad0s1a / ufs ro,noatime 2 2 # df -h / Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/ad0s1a 939M 555M 309M 64% / RAM has been boosted as suggested in ZFS Tuning Guide # cat /boot/loader.conf vm.kmem_size= 1610612736 vm.kmem_size_max= 1610612736 vfs.zfs.prefetch_disable=1 I haven't mucked much with the other memory settings as I'm using amd64 and according to the FreeBSD ZFS wiki, that isn't necessary. I've tried higher settings for kmem but that resulted in a failed boot. I have ample RAM And would love to use as much as possible for network and disk I/O buffers as that's principally all this system does. Disks & ZFS options Sun's "Best Practices" suggests limiting the number of disks in a raidz pool to no more than 6-10, IIRC. ZFS is configured as shown: http://matt.simerson.net/computing/zfs/zpool.txt I'm using all of the ZFS default properties except: atime=off, compression=on. Environment I'm using these machines as backup servers. I wrote an application that generates a list of the thousands of VPS accounts we host. For each host, it generates a rsnapshot configuration file and backs up up their VPS to these systems via rsync. The application manages concurrency and will span additional rsync processes if system i/o load is below a defined thresh-hold. Which is to say, I can crank up or down the amount of network and disk IO the system sees. With vfs.zfs.prefetch_disable=1, a hang will occur within a few hours (no more than a day). If I keep the i/o load (measured via iostat) down to a low level (< 200 iops) then I still get hangs but less frequently (1-6 days). The only way I have found to prevent the hangs is by setting vfs.zfs.prefetch_disable=1. Matt Simerson