Date: Sun, 7 Dec 2008 22:24:40 -0500 From: Adam McDougall <mcdouga9@egr.msu.edu> To: Stefan Bethke <stb@lassitu.de> Cc: FreeBSD Current <freebsd-current@freebsd.org> Subject: Re: ZFS gets stuck Message-ID: <20081208032440.GU93166@egr.msu.edu> In-Reply-To: <5C97E586-4296-4835-A103-FD273B2D7A4F@lassitu.de> References: <20081117205526.GC1733@garage.freebsd.pl> <20081202203308.GA13818@hyperion.scode.org> <200812021254.21242.fjwcash@gmail.com> <20081202232924.GA19134@hyperion.scode.org> <31C70CBC-488A-4A9A-A642-37855E8F1DD1@lassitu.de> <5C97E586-4296-4835-A103-FD273B2D7A4F@lassitu.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Dec 07, 2008 at 11:21:13AM +0100, Stefan Bethke wrote: For the past week, I've been stress testing two new boxes by running make -j4 universe. /usr/src is on ufs, /usr/obj on zfs, backed by a single disk pool. Every so often (about once every one or two days), processes start getting wedged on accessing the zfs file systems. FreeBSD lokschuppen.lassitu.de 8.0-CURRENT FreeBSD 8.0-CURRENT #1: Wed Dec 3 07:05:03 UTC 2008 root@lokschuppen.lassitu.de:/usr/obj/usr/ src/sys/EISENBOOT amd64 So far, I've had this in loader.conf: vfs.zfs.arc_max="512M" vfs.zfs.prefetch_disable="1" I'm now adding vfs.zfs.zil_disable="1" to see if that makes a difference. Is there anything in particular people would want me to check out? Kernel is GENERIC minus a number of devices, and without INVARIANTS and WITNESS. I have one system running a recent -current with 2G of ram that would tend to get stuck almost nightly during a nightly rsync, and have been trying to tune it to get rid of the problems. I've made it about 6 days so far using just: vm.kmem_size=2G vm.kmem_size_max=2G vfs.zfs.arc_max=512M With these settings, I had two rsyncs running constantly for probably 12 hours on the first day or two and still once nightly, so far so good. I tried leaving the kmem settings out and while vm.kmem_size_max appeared to auto tune to approx 4.5G, I still had a out of kmem panic (a low number around 600 or 900M, I don't remember) until I boosted both kmem settings to 2G. I previously had ~1.6G for the kmem settings from when that was the highest they could be set, and maybe that wasn't high enough with recent -current because it had hangs every few days as opposed to every few months with zfs "6" (same zfs as in -stable). I might just be having a string of good luck, and the data that rsync transfers might have played a role (it changes a varying amount each day). If it looks fairly stable then I will apply the settings to a different, busier server with ZFS.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20081208032440.GU93166>