From owner-freebsd-current@FreeBSD.ORG Mon Dec 19 16:40:39 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BD41E106566C for ; Mon, 19 Dec 2011 16:40:39 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from email2.allantgroup.com (email2.emsphone.com [199.67.51.116]) by mx1.freebsd.org (Postfix) with ESMTP id 81F7A8FC12 for ; Mon, 19 Dec 2011 16:40:39 +0000 (UTC) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by email2.allantgroup.com (8.14.4/8.14.4) with ESMTP id pBJGMKdt043093 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 19 Dec 2011 10:22:21 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.5/8.14.5) with ESMTP id pBJGMK9b055483 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 19 Dec 2011 10:22:20 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.5/8.14.5/Submit) id pBJGMKSe055480; Mon, 19 Dec 2011 10:22:20 -0600 (CST) (envelope-from dan) Date: Mon, 19 Dec 2011 10:22:20 -0600 From: Dan Nelson To: Stefan Esser Message-ID: <20111219162220.GK53453@dan.emsphone.com> References: <4EEF488E.1030904@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EEF488E.1030904@freebsd.org> X-OS: FreeBSD 8.2-STABLE User-Agent: Mutt/1.5.21 (2010-09-15) X-Virus-Scanned: clamav-milter 0.97.2 at email2.allantgroup.com X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.6 (email2.allantgroup.com [199.67.51.78]); Mon, 19 Dec 2011 10:22:21 -0600 (CST) X-Scanned-By: MIMEDefang 2.68 on 199.67.51.78 Cc: FreeBSD Current Subject: Re: Uneven load on drives in ZFS RAIDZ1 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Dec 2011 16:40:39 -0000 In the last episode (Dec 19), Stefan Esser said: > for quite some time I have observed an uneven distribution of load between > drives in a 4 * 2TB RAIDZ1 pool. The following is an excerpt of a longer > log of 10 second averages logged with gstat: > > dT: 10.001s w: 10.000s filter: ^a?da?.$ > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > 0 130 106 4134 4.5 23 1033 5.2 48.8| ada0 > 0 131 111 3784 4.2 19 1007 4.0 47.6| ada1 > 0 90 66 2219 4.5 24 1031 5.1 31.7| ada2 > 1 81 58 2007 4.6 22 1023 2.3 28.1| ada3 [...] > zpool status -v > pool: raid1 > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > raid1 ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > ada0p2 ONLINE 0 0 0 > ada1p2 ONLINE 0 0 0 > ada2p2 ONLINE 0 0 0 > ada3p2 ONLINE 0 0 0 Any read from your raidz device will hit three disks (the checksum is applied across the stripe, not on each block, so a full stripe is always read) so I think your extra IOs are coming from somewhere else. What's on p1 on these disks? Could that be the cause of your extra I/Os? Does "zpool iostat -v 10" give you even numbers across all disks? -- Dan Nelson dnelson@allantgroup.com