Date: Wed, 24 Mar 2010 18:18:52 +0200 From: Dan Naumov <dan.naumov@gmail.com> To: freebsd-fs@freebsd.org, freebsd-questions@freebsd.org Subject: tuning vfs.zfs.vdev.max_pending and solving the issue of ZFS writes choking read IO Message-ID: <cf9b1ee01003240918g3a7d61adqaeccb609090a09ea@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello I am having a slight issue (and judging by Google results, similar issues have been seen by other FreeBSD and Solaris/OpenSolaris users) with writes choking the read IO. The issue I am having is described pretty well here: http://opensolaris.org/jive/thread.jspa?threadID=106453 It seems that under heavy write load, ZFS likes to aggregate a really huge amount of data before actually writing it to disks, resulting in sudden 10+ second stalls where it frantically tries to commit everything, completely choking read IO in the process and sometimes even the network (with a large enough write to a mirror pool using DD, I can cause my SSH sessions to drop dead, without actually running out of RAM. As soon as the data is committed, I can reconnect back). Beyond the issue of system interactivity (or rather, the near-disappearance thereof) during these enormous flushes, this kind of pattern seems really ineffective from the CPU utilization point of view. Instead of a relatively stable and consistent flow of reads and writes, allowing the CPU to be utilized as much as possible, when the system is committing the data the CPU basically stays IDLE for 10+ seconds (or as long as the flush takes) and the process of committing unwritten data to the pool seemingly completely trounces the priority of any read operations. Has anyone done any extensive testing of the effects of tuning vfs.zfs.vdev.max_pending on this issue? Is there some universally recommended value beyond the default 35? Anything else I should be looking at? - Sincerely, Dan Naumov
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?cf9b1ee01003240918g3a7d61adqaeccb609090a09ea>