From owner-freebsd-current@FreeBSD.ORG Mon Dec 19 15:42:23 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7ADC6106566C for ; Mon, 19 Dec 2011 15:42:23 +0000 (UTC) (envelope-from peter.maloney@brockmann-consult.de) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.17.8]) by mx1.freebsd.org (Postfix) with ESMTP id 1044D8FC08 for ; Mon, 19 Dec 2011 15:42:22 +0000 (UTC) Received: from [10.3.0.26] ([141.4.215.32]) by mrelayeu.kundenserver.de (node=mrbap1) with ESMTP (Nemesis) id 0MATjG-1RR8lz3iVG-00B1mT; Mon, 19 Dec 2011 16:42:22 +0100 Message-ID: <4EEF5B5C.90905@brockmann-consult.de> Date: Mon, 19 Dec 2011 16:42:20 +0100 From: Peter Maloney User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15 MIME-Version: 1.0 To: freebsd-current@freebsd.org References: <4EEF488E.1030904@freebsd.org> In-Reply-To: <4EEF488E.1030904@freebsd.org> X-Enigmail-Version: 1.1.2 X-Provags-ID: V02:K0:bPqgbp5OjsoLqDCTFJyDn6hh3bG4/hn6HInIUlQIzuI lfZ1WGZYecDCsClCJ02/xbx+LO0UWvl4hWCxd24L6Azv8TUJHu asNsg3scCXkK3kAYPptA9TW198lHOcEkuuCqjTmaIV3IFFfTyh tu9BHu+AfiHSqaE+XpLYhQMRuIAoP426clq98u52XVXmleKh12 kD3O31Wze8U3Ez839sWjZIvmbMC/I2A69T2dqogtbe2yLYpwDZ OI/ndwLTi9KTjAGtuxfT4+AHtekRMOIaXnfc1/zeQAYxgifvT1 otj1TMlxQs8AngBUD+AV+4lG9V1SjE1wwkeAQ2QaItnldHU7Rn /5rFPb9QPQytoexLpkQtoKrz/SWOFzPYIkwXt14nX Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: Uneven load on drives in ZFS RAIDZ1 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Dec 2011 15:42:23 -0000 On 12/19/2011 03:22 PM, Stefan Esser wrote: > Hi ZFS users, > > for quite some time I have observed an uneven distribution of load > between drives in a 4 * 2TB RAIDZ1 pool. The following is an excerpt of > a longer log of 10 second averages logged with gstat: > > dT: 10.001s w: 10.000s filter: ^a?da?.$ > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > 0 130 106 4134 4.5 23 1033 5.2 48.8| ada0 > 0 131 111 3784 4.2 19 1007 4.0 47.6| ada1 > 0 90 66 2219 4.5 24 1031 5.1 31.7| ada2 > 1 81 58 2007 4.6 22 1023 2.3 28.1| ada3 > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > 1 132 104 4036 4.2 27 1129 5.3 45.2| ada0 > 0 129 103 3679 4.5 26 1115 6.8 47.6| ada1 > 1 91 61 2133 4.6 30 1129 1.9 29.6| ada2 > 0 81 56 1985 4.8 24 1102 6.0 29.4| ada3 > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > 1 148 108 4084 5.3 39 2511 7.2 55.5| ada0 > 1 141 104 3693 5.1 36 2505 10.4 54.4| ada1 > 1 102 62 2112 5.6 39 2508 5.5 35.4| ada2 > 0 99 60 2064 6.0 39 2483 3.7 36.1| ada3 > > ... > So: Can anybody reproduce this distribution requests? I don't have a raidz1 machine, and no time to make you a special raidz1 pool out of spare disks, but on my raidz2 I can only ever see unevenness when a disk is bad, or between different vdevs. But you only have one vdev. Check is that your disks are identical (are they? we can only assume so since you didn't say so). Show us output from: smartctl -i /dev/ada0 smartctl -i /dev/ada1 smartctl -i /dev/ada2 smartctl -i /dev/ada3 Since your tests show read ms/r to be pretty even, I guess your disks are not broken. But the ms/w is slightly different. So I think it seems that the first 2 disks are slower for writing (someone once said that refurbished disks are like this, even if identical), or the hard disk controller ports they use are slower. For example, maybe your motherboard has 6 ports, and you plugged disks 1,2,3 into port 1,2,3 and disk 4 into port 5. Disk 3 and 4 would have their own channel, but disk 1 and 2 share one. So if the disks are identical, I would guess your hard disk controller is to blame. To test this, first back it up. Then *fix your setup by using labels*. ie. use gpt/somelabel0 or gptid/....... rather than ada0p2. Check "ls /dev/gpt*" output for options on what labels you have already. Then try swapping disks around to see if the load changes. Make sure to back up... Swapping disks (or even removing one depending on controller, etc. when it fails) without labels can be bad. eg. You have ada1 ada2 ada3 ada4. Someone spills coffee on ada2; it fries and cannot be detected anymore, and you reboot. Now you have ada1 ada2 ada3. Then things are usually still fine (even though ada3 is now ada2 and ada4 is now ada3, because there is some zfs superblock stuff to keep track of things), but if you also had an ada5 that was not part of the pool, or was a spare or a log or something other than another disk in the same vdev as ada1, etc., bad things happen when it becomes ada4. Unfortunately, I don't know exactly what people do to cause the "bad things" that happen. When this happened to me, it just said my pool was faulted or degraded or something, and set a disk or two to UNAVAIL or FAULTED. I don't remember it automatically resilvering them, but when I read about these problems, I think it seems like some disks were resilvered afterwards. And last thing I can think of is to make sure your partitions are aligned, and identical. Show us output from: gpart show > Any idea, why this is happening and whether something should be changed > in ZFS to better distribute the load (leading to higher file system > performance)? > > Best regards, STefan > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" -- -------------------------------------------- Peter Maloney Brockmann Consult Max-Planck-Str. 2 21502 Geesthacht Germany Tel: +49 4152 889 300 Fax: +49 4152 889 333 E-mail: peter.maloney@brockmann-consult.de Internet: http://www.brockmann-consult.de --------------------------------------------