From owner-freebsd-questions@FreeBSD.ORG Sun Aug 11 04:11:27 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 09FFA5EE for ; Sun, 11 Aug 2013 04:11:27 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ie0-x22e.google.com (mail-ie0-x22e.google.com [IPv6:2607:f8b0:4001:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CDC512046 for ; Sun, 11 Aug 2013 04:11:26 +0000 (UTC) Received: by mail-ie0-f174.google.com with SMTP id w15so6316502iea.5 for ; Sat, 10 Aug 2013 21:11:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=HdBSPYEKDatFndIZzAU5vCpY1AmweCUbV0c1PcyJBN4=; b=JYKCxiWt5bZxx2WsI8GtXRmgr8wGV19ohOFWDARpFMsn3YulsOcSUnE3kvYDlCqTea zUMFfp41xLWQumRvJxrATLxs6vDeWYWwFEpuI17qJju6pZyXj91LNjPJfagqhtw2Uvy1 uabA6/srd7hhxTSapZvS2xw5ZYMXpL4SMDvNC9X//NcqcW/6v96XM/EO1ItB8zLrJ9ol 2WplFYPG+VGtjMXo+fecxKiT3mZg/stXxVOBGsahsob6eQpaJ2tYbIl4Vh9tTV8H+Y8F JHCLcELNhf6s++62hoFsdj0qpKx8mNcOJOONGyiE8l7ewx0lp51pWW1uMkKA8eawuNYQ HZVQ== MIME-Version: 1.0 X-Received: by 10.42.215.11 with SMTP id hc11mr7540512icb.9.1376194285435; Sat, 10 Aug 2013 21:11:25 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.42.150.196 with HTTP; Sat, 10 Aug 2013 21:11:25 -0700 (PDT) In-Reply-To: <5202C558.3010305@fjl.co.uk> References: <5202C558.3010305@fjl.co.uk> Date: Sun, 11 Aug 2013 00:11:25 -0400 X-Google-Sender-Auth: 01JUD0xpuNK5x3XcEiaENfJ_nzY Message-ID: Subject: Re: Terrible disk performance with LSI / FreeBSD 9.2-RC1 From: J David To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Aug 2013 04:11:27 -0000 To follow up on this issue, at one point the stats were down to this: extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b da0 0.0 0.0 0.0 0.0 0 0.0 0 da1 0.0 0.0 0.0 0.0 0 0.0 0 da2 127.9 0.0 202.3 0.0 1 47.5 100 da3 125.9 0.0 189.3 0.0 1 43.1 97 da4 127.9 0.0 189.8 0.0 1 45.8 100 da5 128.9 0.0 206.3 0.0 0 42.5 99 da6 127.9 0.0 202.3 0.0 1 46.2 98 da7 0.0 249.7 0.0 334.2 10 39.5 100 At some point, I figured out that 125 random iops is pretty much the limit for 7200 RPM SATA drives. So mostly what we're looking at here is the resilver of a raidz2 is the pathological worst case. Lesson learned; raidz2 is just really not viable without some kind of sort on the resilver operations. Wish I understood ZFS well enough to do something about that, but research suggests the problem is non-trivial. :( There also seems to be a separate ZFS issue related to having a very large number of snapshots (e.g. hourly for several months on a couple of filesystems). Some combination of the OS updates we've been doing trying to get this machine to 9.2-RC1 and deleting a ton of snapshots. It would be nice to know which it was; I guess we'll find out in a few months. So it seems like the combination of these two issues is mostly what is/was plaguing us. Thanks!