From owner-freebsd-fs@FreeBSD.ORG Wed Mar 24 16:18:54 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22CF2106564A; Wed, 24 Mar 2010 16:18:54 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-bw0-f216.google.com (mail-bw0-f216.google.com [209.85.218.216]) by mx1.freebsd.org (Postfix) with ESMTP id 7ED428FC08; Wed, 24 Mar 2010 16:18:53 +0000 (UTC) Received: by mail-bw0-f216.google.com with SMTP id 8so1281969bwz.3 for ; Wed, 24 Mar 2010 09:18:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=9WvB4PoqQfeDySwmu/Qx5SY+cwfQ4ZQUlX+qo91boXw=; b=t+Fn5mNa+uyo9kEJ2zKWlcdTZONRCWhENN83VgF8rr7gfLBX7G5mySXU+PU0w3+iPF hWpDNtlIyskVTpWMIAYCxSbRXgKjCNI/KA+OriuC+tY0L1s4ep8zvm02ruj2sKOb+tVT hvMaFH2C+bN1Ns9owH2gE7nUsZ8+ZiQscsdO8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=cdxgD/dKonmgwFRvm11dBZ2Jo5zHK7O2ocoI3uOHMmPNYGSBxriIwlBY8ESsoFjw1M ULgd3i+a4V4I//k7yfVnbJFhIIblP5DLLD3ii3N9s84qId5MyKKFjJ21OG8tgxVtqLXl ZcBFulu6LGmGpppnn3OmVJY+2XM9ey1CIGXEM= MIME-Version: 1.0 Received: by 10.204.134.70 with SMTP id i6mr7528622bkt.74.1269447532762; Wed, 24 Mar 2010 09:18:52 -0700 (PDT) Date: Wed, 24 Mar 2010 18:18:52 +0200 Message-ID: From: Dan Naumov To: freebsd-fs@freebsd.org, freebsd-questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: tuning vfs.zfs.vdev.max_pending and solving the issue of ZFS writes choking read IO X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Mar 2010 16:18:54 -0000 Hello I am having a slight issue (and judging by Google results, similar issues have been seen by other FreeBSD and Solaris/OpenSolaris users) with writes choking the read IO. The issue I am having is described pretty well here: http://opensolaris.org/jive/thread.jspa?threadID=106453 It seems that under heavy write load, ZFS likes to aggregate a really huge amount of data before actually writing it to disks, resulting in sudden 10+ second stalls where it frantically tries to commit everything, completely choking read IO in the process and sometimes even the network (with a large enough write to a mirror pool using DD, I can cause my SSH sessions to drop dead, without actually running out of RAM. As soon as the data is committed, I can reconnect back). Beyond the issue of system interactivity (or rather, the near-disappearance thereof) during these enormous flushes, this kind of pattern seems really ineffective from the CPU utilization point of view. Instead of a relatively stable and consistent flow of reads and writes, allowing the CPU to be utilized as much as possible, when the system is committing the data the CPU basically stays IDLE for 10+ seconds (or as long as the flush takes) and the process of committing unwritten data to the pool seemingly completely trounces the priority of any read operations. Has anyone done any extensive testing of the effects of tuning vfs.zfs.vdev.max_pending on this issue? Is there some universally recommended value beyond the default 35? Anything else I should be looking at? - Sincerely, Dan Naumov