Date: Mon, 29 Apr 2013 09:01:54 -0700 From: Freddie Cash <fjwcash@gmail.com> To: Daniel Kalchev <daniel@digsys.bg> Cc: FreeBSD Filesystems <freebsd-fs@freebsd.org>, Andriy Gapon <avg@freebsd.org> Subject: Re: Strange slowdown when cache devices enabled in ZFS Message-ID: <CAOjFWZ4og1rO0u=vBzupYXirgtqm1mpwwhf1kdXbjeteNPbbOQ@mail.gmail.com> In-Reply-To: <6F1DCA37-9159-47F2-9ABB-FDEC41665611@digsys.bg> References: <CAOjFWZ6Q=Vs3P-kfGysLzSbw4CnfrJkMEka4AqfSrQJFZDP_qw@mail.gmail.com> <51430744.6020004@FreeBSD.org> <CAOjFWZ5e2t0Y_KOxm%2BGhX%2BzXNPfOXb8HKF4uU%2BQ%2BN5eWQqLtdg@mail.gmail.com> <51487CE1.5090703@FreeBSD.org> <CAOjFWZ6uL5nJ3X0Bz-oxJf-o21k81HfkR3PgwM022R4W21_5ZQ@mail.gmail.com> <CAOjFWZ60r5X_vP3g7vFXWZt0E1Vk-9KcTANVm9ReqfUUmvNqQg@mail.gmail.com> <6F1DCA37-9159-47F2-9ABB-FDEC41665611@digsys.bg>
next in thread | previous in thread | raw e-mail | index | archive | help
The following settings in /etc/sysctl.conf prevent the "stalls" completely, even when the L2ARC devices are 100% full and all RAM is wired into the ARC. Been running without issues for 5 days now: vfs.zfs.l2arc_norw=0 # Default is 1 vfs.zfs.l2arc_feed_again=0 # Default is 1 vfs.zfs.l2arc_noprefetch=0 # Default is 0 vfs.zfs.l2arc_feed_min_ms=1000 # Default is 200 vfs.zfs.l2arc_write_boost=320000000 # Default is 8 MBps vfs.zfs.l2arc_write_max=160000000 # Default is 8 MBps With these settings, I'm also able to expand the ARC to use the full 128 GB of RAM in the biggest box, and to use both L2ARC devices (60 GB in total). And, can set primarycache and secondarycache to all (the default) instead of just metadata. The only two settings in sysctl.conf that I've changed since the "stalls" began are: vfs.zfs.l2arc_norw vfs.zfs.l2arc_feed_again The other settings were already set when the boxes went live a few months back. I've run the dtrace hotkernel script from -HEAD on each of the 9-STABLE boxes. Not really sure how to interpret the results. I ran the script (on the receiving box) while doing a full zfs send/recv with the norw set to 1 (default) and then again with it set to 0. Here's the top 20 entries from each run. There doesn't seem to be much of a difference in the output: ==> hotkernel.norw0 <== kernel`_sx_try_xlock 48996 0.1% kernel`bzero 50683 0.1% kernel`_mtx_lock_sleep 71089 0.1% kernel`hpet_get_timecount 81257 0.2% kernel`atomic_add_long 97927 0.2% kernel`_sx_xunlock 122009 0.2% kernel`_sx_xlock 122743 0.2% zfs.ko`l2arc_write_eligible 134717 0.3% kernel`bcopy 157756 0.3% zfs.ko`buf_hash 184670 0.4% zfs.ko`l2arc_feed_thread 295029 0.6% zfs.ko`list_next 309683 0.6% kernel`sched_idletd 316410 0.6% kernel`spinlock_exit 337550 0.7% zfs.ko`SHA256_Transform 427619 0.9% zfs.ko`lzjb_compress 500649 1.0% kernel`_sx_xlock_hard 537433 1.1% kernel`cpu_idle_mwait 3214405 6.5% kernel`acpi_cpu_c1 41039557 83.2% ==> hotkernel.norw1 <== kernel`copyin 49078 0.2% kernel`_mtx_lock_sleep 54643 0.2% kernel`_rw_wlock_hard 60637 0.2% kernel`hpet_get_timecount 64973 0.2% kernel`atomic_add_long 83166 0.3% zfs.ko`l2arc_write_eligible 104574 0.3% kernel`_sx_xunlock 108864 0.4% kernel`_sx_xlock 112804 0.4% zfs.ko`buf_hash 151566 0.5% kernel`bcopy 161269 0.5% zfs.ko`l2arc_feed_thread 240827 0.8% zfs.ko`list_next 252949 0.8% kernel`spinlock_exit 361583 1.2% kernel`sched_idletd 407961 1.3% zfs.ko`SHA256_Transform 546927 1.8% kernel`_sx_xlock_hard 629281 2.0% zfs.ko`lzjb_compress 634820 2.1% kernel`cpu_idle_mwait 3359336 10.9% kernel`acpi_cpu_c1 22177795 71.9% -- Freddie Cash fjwcash@gmail.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOjFWZ4og1rO0u=vBzupYXirgtqm1mpwwhf1kdXbjeteNPbbOQ>