Date: Thu, 31 Jul 2008 13:58:26 -0700 From: Matt Simerson <matt@corp.spry.com> To: freebsd-fs@freebsd.org Subject: ZFS hang issue and prefetch_disable - UPDATE Message-ID: <62D3072A-E41A-4CFC-971D-9924958F38C7@corp.spry.com> References: <20253C48-38CB-4A77-9C59-B993E7E5D78A@corp.spry.com>
next in thread | previous in thread | raw e-mail | index | archive | help
My announcement that vfs.zfs.prefetch_disable=1 resulted in a stable system was premature. One of my backup servers (see specs below) hung. When I got onto the console via KVM, it looked normal with no errors but didn't respond to Control-Alt-Delete. After a power cycle, zpool status showed 8 disks FAULTED and the action state was: http://www.sun.com/msg/ZFS-8000-5E Basically, that meant my ZFS file system and 7.5TB of data was gone. Ouch. I'm using a pair of ARECA 1231ML RAID controllers. Previously, I had them configured in JBOD with raidz2. This time around, I configured both controllers with one 12 disk RAID 6 volume. Now FreeBSD just sees two 10TB disks which I stripe with ZFS: zpool create back01 /dev/ da0 /dev/da1 I also did a bit more fiddling with /boot/loader.conf. Previous I had: vm.kmem_size="1536M" vm.kmem_size_max="1536M" vfs.zfs.prefetch_disable=1 This resulted in ZFS using 1.1GB of RAM (as measured using the technique described on the wiki) during normal use. The system in question hung during the nightly processing (which backs up some other systems via rsync) and my suspicions are that when I/O load picked up, it exhausted the available kernel memory and hung the system. So now I have these settings on one system: vm.kmem_size="1536M" vm.kmem_size_max="1536M" vfs.zfs.arc_min="16M" vfs.zfs.arc_max="64M" vfs.zfs.prefetch_disable=1 and the same except vfs.zfs.arc_max="256M" on the other. The one with 64M uses 256MB of RAM for ZFS and the one set at 256M uses 600MB of RAM. These are measured under heavy network and disk IO load being generated by multiple rsync processes pulling backups from remote nodes and storing it on ZFS. I am using ZFS compression. I get much better performance now with RAID 6 on the controller and ZFS striping than using raidz2. Unless tuning the arc_ settings made the difference. Either way, the system I just rebuilt is now quite a bit faster with RAID 6 than JBOD + raidz2. Hopefully tuning vfs.zfs.arc_max will result in stability. If it doesn't, my next choice is upgrading to -HEAD with the recent ZFS patch or ditching ZFS entirely and using geom_stripe. I don't like either option. Matt > From: Matt Simerson <matt@corp.spry.com> > Date: July 22, 2008 1:25:42 PM PDT > To: freebsd-fs@freebsd.org > Subject: ZFS hang issue and prefetch_disable > > Symptoms > > Deadlocks under heavy IO load on the ZFS file system with > prefetch_disable=0. Setting vfs.zfs.prefetch_disable=1 results in a > stable system. > > Configuration > > Two machines. Identically built. Both exhibit identical behavior. > 8 cores (2 x E5420) x 2.5GHz, 16 GB RAM, 24 x 1TB disks. > FreeBSD 7.0 amd64 > dmesg: http://matt.simerson.net/computing/zfs/dmesg.txt > > Boot disk is a read only 1GB compact flash > # cat /etc/fstab > /dev/ad0s1a / ufs ro,noatime 2 2 > > # df -h / > Filesystem 1K-blocks Used Avail Capacity Mounted on > /dev/ad0s1a 939M 555M 309M 64% / > > RAM has been boosted as suggested in ZFS Tuning Guide > # cat /boot/loader.conf > vm.kmem_size= 1610612736 > vm.kmem_size_max= 1610612736 > vfs.zfs.prefetch_disable=1 > > I haven't mucked much with the other memory settings as I'm using > amd64 and according to the FreeBSD ZFS wiki, that isn't necessary. > I've tried higher settings for kmem but that resulted in a failed > boot. I have ample RAM And would love to use as much as possible for > network and disk I/O buffers as that's principally all this system > does. > > Disks & ZFS options > > Sun's "Best Practices" suggests limiting the number of disks in a > raidz pool to no more than 6-10, IIRC. ZFS is configured as shown: http://matt.simerson.net/computing/zfs/zpool.txt > > I'm using all of the ZFS default properties except: atime=off, > compression=on. > > Environment > > I'm using these machines as backup servers. I wrote an application > that generates a list of the thousands of VPS accounts we host. For > each host, it generates a rsnapshot configuration file and backs up > up their VPS to these systems via rsync. The application manages > concurrency and will spawn additional rsync processes if system i/o > load is below a defined threshhold. Which is to say, I can crank up > or down the amount of disk IO the system sees. > > With vfs.zfs.prefetch_disable=0, I can trigger a hang within a few > hours (no more than a day). If I keep the i/o load (measured via > iostat) down to a low level (< 200 iops) then I still get hangs but > less frequently (1-6 days). The only way I have found to prevent > the hangs is by setting vfs.zfs.prefetch_disable=1.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?62D3072A-E41A-4CFC-971D-9924958F38C7>