From owner-freebsd-current@FreeBSD.ORG Mon Dec 19 17:05:21 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 18643106564A; Mon, 19 Dec 2011 17:05:21 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-tul01m020-f182.google.com (mail-tul01m020-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id CC3FF8FC16; Mon, 19 Dec 2011 17:05:20 +0000 (UTC) Received: by obbwd18 with SMTP id wd18so2795566obb.13 for ; Mon, 19 Dec 2011 09:05:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=7hLMRkgWqFY0kO9sk2oWVJSuomSe5vidSQ/zqLaoMwg=; b=RDU8ZBrPyuQxvvVm03y0mKGc74hTgEbtBRTeBEP6r+ESD7idKhLYI6gHAafce52e3S cncEgg7xmKlWaISQaihYVtyoC83prEqaOTSmGpm7+bj0ET+NX9aOR8TX64w2nsmRlaDp e10Zo7w+d0t0WpDKrSuR0RV6zgZtTu7nNoDx8= MIME-Version: 1.0 Received: by 10.182.17.102 with SMTP id n6mr10778933obd.56.1324314320418; Mon, 19 Dec 2011 09:05:20 -0800 (PST) Received: by 10.182.62.227 with HTTP; Mon, 19 Dec 2011 09:05:20 -0800 (PST) In-Reply-To: <4EEF488E.1030904@freebsd.org> References: <4EEF488E.1030904@freebsd.org> Date: Mon, 19 Dec 2011 09:05:20 -0800 Message-ID: From: Garrett Cooper To: Stefan Esser Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD Current Subject: Re: Uneven load on drives in ZFS RAIDZ1 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Dec 2011 17:05:21 -0000 On Mon, Dec 19, 2011 at 6:22 AM, Stefan Esser wrote: > Hi ZFS users, > > for quite some time I have observed an uneven distribution of load > between drives in a 4 * 2TB RAIDZ1 pool. The following is an excerpt of > a longer log of 10 second averages logged with gstat: > > dT: 10.001s =A0w: 10.000s =A0filter: ^a?da?.$ > =A0L(q) =A0ops/s =A0 =A0r/s =A0 kBps =A0 ms/r =A0 =A0w/s =A0 kBps =A0 ms/= w =A0 %busy Name > =A0 =A00 =A0 =A0130 =A0 =A0106 =A0 4134 =A0 =A04.5 =A0 =A0 23 =A0 1033 = =A0 =A05.2 =A0 48.8| ada0 > =A0 =A00 =A0 =A0131 =A0 =A0111 =A0 3784 =A0 =A04.2 =A0 =A0 19 =A0 1007 = =A0 =A04.0 =A0 47.6| ada1 > =A0 =A00 =A0 =A0 90 =A0 =A0 66 =A0 2219 =A0 =A04.5 =A0 =A0 24 =A0 1031 = =A0 =A05.1 =A0 31.7| ada2 > =A0 =A01 =A0 =A0 81 =A0 =A0 58 =A0 2007 =A0 =A04.6 =A0 =A0 22 =A0 1023 = =A0 =A02.3 =A0 28.1| ada3 > > =A0L(q) =A0ops/s =A0 =A0r/s =A0 kBps =A0 ms/r =A0 =A0w/s =A0 kBps =A0 ms/= w =A0 %busy Name > =A0 =A01 =A0 =A0132 =A0 =A0104 =A0 4036 =A0 =A04.2 =A0 =A0 27 =A0 1129 = =A0 =A05.3 =A0 45.2| ada0 > =A0 =A00 =A0 =A0129 =A0 =A0103 =A0 3679 =A0 =A04.5 =A0 =A0 26 =A0 1115 = =A0 =A06.8 =A0 47.6| ada1 > =A0 =A01 =A0 =A0 91 =A0 =A0 61 =A0 2133 =A0 =A04.6 =A0 =A0 30 =A0 1129 = =A0 =A01.9 =A0 29.6| ada2 > =A0 =A00 =A0 =A0 81 =A0 =A0 56 =A0 1985 =A0 =A04.8 =A0 =A0 24 =A0 1102 = =A0 =A06.0 =A0 29.4| ada3 > > =A0L(q) =A0ops/s =A0 =A0r/s =A0 kBps =A0 ms/r =A0 =A0w/s =A0 kBps =A0 ms/= w =A0 %busy Name > =A0 =A01 =A0 =A0148 =A0 =A0108 =A0 4084 =A0 =A05.3 =A0 =A0 39 =A0 2511 = =A0 =A07.2 =A0 55.5| ada0 > =A0 =A01 =A0 =A0141 =A0 =A0104 =A0 3693 =A0 =A05.1 =A0 =A0 36 =A0 2505 = =A0 10.4 =A0 54.4| ada1 > =A0 =A01 =A0 =A0102 =A0 =A0 62 =A0 2112 =A0 =A05.6 =A0 =A0 39 =A0 2508 = =A0 =A05.5 =A0 35.4| ada2 > =A0 =A00 =A0 =A0 99 =A0 =A0 60 =A0 2064 =A0 =A06.0 =A0 =A0 39 =A0 2483 = =A0 =A03.7 =A0 36.1| ada3 This suggests (note that I said suggests) that there might be a slight difference in the data path speeds or physical media as someone else suggested; look at zpool iostat -v though before making a firm statement as to whether or not a drive is truly not performing to your assumed spec. gstat and zpool iostat -v suggest performance though -- they aren't the end-all-be-all for determining drive performance. If the latency numbers were high enough, I would suggest dd'ing out to the individual drives (i.e. remove the drive from the RAIDZ) to see if there's a noticeable discrepancy, as this can indicate a bad cable, backplane, or drive; from there I would start doing the physical swap routine and see if the issue moves with the drive or stays static with the controller channel and/or chassis slot. Cheers, -Garrett