From owner-freebsd-fs@FreeBSD.ORG Tue Jul 16 12:16:44 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 61619E15 for ; Tue, 16 Jul 2013 12:16:44 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.21.123]) by mx1.freebsd.org (Postfix) with ESMTP id D481E1FA for ; Tue, 16 Jul 2013 12:16:43 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [193.68.6.1]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.6/8.14.6) with ESMTP id r6GCGcZ4023385 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Tue, 16 Jul 2013 15:16:39 +0300 (EEST) (envelope-from daniel@digsys.bg) Message-ID: <51E539A6.9090109@digsys.bg> Date: Tue, 16 Jul 2013 15:16:38 +0300 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130627 Thunderbird/17.0.7 MIME-Version: 1.0 To: freebsd-fs Subject: Re: ZFS vdev I/O questions References: <51E5316B.9070201@digsys.bg> <3472068604314C9887FE3BD4CD42B7C8@multiplay.co.uk> In-Reply-To: <3472068604314C9887FE3BD4CD42B7C8@multiplay.co.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Jul 2013 12:16:44 -0000 On 16.07.13 14:53, Mark Felder wrote: >> The other observation I have is with the first vdev: the 512b drives do >> a lot of I/O fast, complete first and then sit idle, while da0 continues >> to write for many more seconds. They consistently show many more IOPS >> than the other drives for this type of activity -- on streaming writes >> all drives behave more or less the same. It is only on this un-dedup >> scenario where the difference is so much pronounced. >> >> All the vdevs in the pool are with ashift=12 so the theory that ZFS >> actually issues 512b writes to these drives can't be true, can it? >> >> Another worry is this Seagate Barracuda 7200.14 (AF) >> (ST3000DM001-1CH166, firmware CC24) drive. It seems constantly >> under-performing. Does anyone know if it is so different from the >> ST3000DM001-9YN166 drives? Might be, I should just replace it? >> > A lot of information here. > > Those Hitachis are great drives. The addition of the Barracuda with > different performance characteristics could be part of the problem. I'm > glad you pointed out that the pool ashift=12 so we can try to rule that > out. I'd be quite interested in knowing if some or perhaps even all of > your issues go away simply by replacing that drive with another Hitachi. I don't have any more of these available and the vendor unfortunately could not supply more (but I am considering looking elsewhere) At the moment, I could only replace that Barracuda with one of the spare drives, which are Seagate SV35 (ST3000VX000-9YW166, firmware CV13) I have observed that these drives are not particularly good at random I/O however.... On 16.07.13 14:58, Steven Hartland wrote: > One thing to check with this is to add -d to your gstat to see > if your waiting deletes IO? I doubt it but worth checking. Here is with gstat -d. Actually, running this script makes more sense: while true ;do gstat -f 'da[0-9]*$' -d -b sleep 1 done (small random reads, heavy writes, then some reads) dT: 1.001s w: 1.000s filter: da[0-9]*$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 11 1 4 20.8 10 44 0.4 0 0 0.0 2.2 da0 0 15 5 20 7.1 10 40 5.8 0 0 0.0 5.1 da1 0 7 0 0 0.0 7 28 0.4 0 0 0.0 0.2 da2 1 7 0 0 0.0 7 28 4.4 0 0 0.0 1.1 da3 1 11 0 0 0.0 11 44 0.5 0 0 0.0 0.2 da4 0 11 0 0 0.0 11 44 0.4 0 0 0.0 0.2 da5 0 25 16 68 7.8 9 36 0.4 0 0 0.0 5.4 da6 0 21 12 52 14.2 9 36 0.3 0 0 0.0 6.0 da7 0 9 0 0 0.0 9 36 0.3 0 0 0.0 0.1 da8 0 35 23 176 12.5 12 48 0.4 0 0 0.0 9.4 da9 0 26 17 140 13.7 9 36 0.3 0 0 0.0 7.3 da10 0 8 0 0 0.0 8 32 0.3 0 0 0.0 0.1 da11 0 158 29 212 17.4 129 4083 81.9 0 0 0.0 57.8 da12 0 162 25 192 16.7 137 4423 84.8 0 0 0.0 61.9 da13 0 118 0 0 0.0 118 3844 95.5 0 0 0.0 45.6 da14 5 152 20 112 10.8 132 4127 66.9 0 0 0.0 45.8 da15 0 171 14 88 17.4 157 4990 89.2 0 0 0.0 69.6 da16 0 124 1 4 6.6 123 3840 71.5 0 0 0.0 37.6 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 4558 4557 6736 0.5 1 1 0.2 0 0 0.0 14.7 da20 0 120 0 0 0.0 119 8083 35.2 0 0 0.0 17.4 ada0 0 120 0 0 0.0 119 8083 34.0 0 0 0.0 16.8 ada1 dT: 1.001s w: 1.000s filter: da[0-9]*$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 1 1 4 14.0 0 0 0.0 0 0 0.0 1.4 da0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da1 0 2 2 8 18.8 0 0 0.0 0 0 0.0 3.7 da2 0 1 1 4 35.1 0 0 0.0 0 0 0.0 3.5 da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da4 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da5 0 3 3 12 12.8 0 0 0.0 0 0 0.0 3.8 da6 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da7 0 3 3 12 14.3 0 0 0.0 0 0 0.0 4.3 da8 0 1 1 4 9.8 0 0 0.0 0 0 0.0 1.0 da9 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da10 0 7 7 28 9.8 0 0 0.0 0 0 0.0 6.8 da11 0 1 1 4 7.4 0 0 0.0 0 0 0.0 0.7 da12 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da13 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da14 0 3 3 12 34.0 0 0 0.0 0 0 0.0 10.2 da15 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da16 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 2656 236 455 0.2 2420 12284 0.3 0 0 0.0 21.3 da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada1 dT: 1.001s w: 1.000s filter: da[0-9]*$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 4 4 16 22.8 0 0 0.0 0 0 0.0 9.1 da0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da1 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da2 0 1 1 4 17.4 0 0 0.0 0 0 0.0 1.7 da3 0 3 3 12 13.2 0 0 0.0 0 0 0.0 4.0 da4 0 2 2 8 28.2 0 0 0.0 0 0 0.0 5.6 da5 0 1 1 4 9.7 0 0 0.0 0 0 0.0 1.0 da6 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da7 0 3 3 12 8.8 0 0 0.0 0 0 0.0 2.6 da8 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da9 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da10 0 5 5 20 10.1 0 0 0.0 0 0 0.0 5.0 da11 0 4 4 16 10.4 0 0 0.0 0 0 0.0 4.1 da12 0 1 1 4 16.0 0 0 0.0 0 0 0.0 1.6 da13 0 1 1 4 13.8 0 0 0.0 0 0 0.0 1.4 da14 0 4 4 16 7.3 0 0 0.0 0 0 0.0 2.9 da15 0 1 1 4 13.1 0 0 0.0 0 0 0.0 1.3 da16 0 1 1 4 21.3 0 0 0.0 0 0 0.0 2.1 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 515 515 979 0.2 0 0 0.0 0 0 0.0 9.2 da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada1 dT: 1.001s w: 1.000s filter: da[0-9]*$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 1 1 4 32.3 0 0 0.0 0 0 0.0 3.2 da0 0 1 1 4 35.0 0 0 0.0 0 0 0.0 3.5 da1 0 4 4 16 4.3 0 0 0.0 0 0 0.0 1.7 da2 0 3 3 12 31.8 0 0 0.0 0 0 0.0 9.5 da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da4 0 4 4 16 6.9 0 0 0.0 0 0 0.0 2.8 da5 0 2 2 8 13.6 0 0 0.0 0 0 0.0 2.7 da6 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da7 0 4 4 16 7.4 0 0 0.0 0 0 0.0 3.0 da8 0 1 1 4 12.6 0 0 0.0 0 0 0.0 1.3 da9 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da10 0 6 6 24 10.0 0 0 0.0 0 0 0.0 6.0 da11 0 1 1 4 21.6 0 0 0.0 0 0 0.0 2.2 da12 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da13 1 1 1 4 10.8 0 0 0.0 0 0 0.0 1.1 da14 0 1 1 4 18.3 0 0 0.0 0 0 0.0 1.8 da15 0 1 1 4 9.9 0 0 0.0 0 0 0.0 1.0 da16 0 1 1 4 21.6 0 0 0.0 0 0 0.0 2.2 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 588 587 1130 0.2 1 2 0.6 0 0 0.0 10.5 da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada1 dT: 1.001s w: 1.000s filter: da[0-9]*$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 23 1412 0 0 0.0 1412 24423 15.0 0 0 0.0 99.6 da0 23 4699 0 0 0.0 4699 72429 4.1 0 0 0.0 97.2 da1 24 4294 0 0 0.0 4294 67749 4.6 0 0 0.0 99.0 da2 23 4923 0 0 0.0 4923 75746 3.8 0 0 0.0 96.2 da3 24 4452 0 0 0.0 4452 67729 4.3 0 0 0.0 97.5 da4 23 4558 0 0 0.0 4558 68992 4.2 0 0 0.0 97.3 da5 24 813 0 0 0.0 813 10675 27.2 0 0 0.0 99.6 da6 24 866 0 0 0.0 866 11338 25.5 0 0 0.0 99.5 da7 24 845 0 0 0.0 845 11058 25.8 0 0 0.0 99.5 da8 24 866 0 0 0.0 866 11174 25.3 0 0 0.0 98.8 da9 24 928 0 0 0.0 928 12061 23.6 0 0 0.0 99.2 da10 24 864 0 0 0.0 864 11150 25.6 0 0 0.0 99.2 da11 24 674 0 0 0.0 674 14587 32.4 0 0 0.0 99.9 da12 24 705 0 0 0.0 705 15043 31.7 0 0 0.0 100.1 da13 24 670 0 0 0.0 670 14527 33.0 0 0 0.0 99.5 da14 24 648 0 0 0.0 648 13360 34.3 0 0 0.0 99.5 da15 24 716 0 0 0.0 716 14835 30.9 0 0 0.0 99.5 da16 24 730 0 0 0.0 730 15227 30.0 0 0 0.0 98.7 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada1 dT: 1.001s w: 1.000s filter: da[0-9]*$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 24 1169 0 0 0.0 1169 15954 20.6 0 0 0.0 100.3 da0 0 1811 0 0 0.0 1811 20961 3.9 0 0 0.0 29.6 da1 0 2392 0 0 0.0 2392 27444 3.7 0 0 0.0 37.1 da2 0 1199 0 0 0.0 1199 14571 3.7 0 0 0.0 18.5 da3 0 2305 0 0 0.0 2305 26924 4.0 0 0 0.0 38.1 da4 0 1820 0 0 0.0 1820 21573 3.8 0 0 0.0 28.8 da5 24 744 0 0 0.0 744 9248 32.6 0 0 0.0 99.5 da6 24 771 0 0 0.0 771 10722 31.1 0 0 0.0 100.0 da7 24 716 0 0 0.0 716 9268 33.5 0 0 0.0 100.2 da8 24 754 0 0 0.0 754 9280 31.8 0 0 0.0 99.9 da9 24 769 0 0 0.0 769 9384 31.4 0 0 0.0 99.9 da10 24 742 0 0 0.0 742 9508 32.1 0 0 0.0 98.5 da11 24 665 0 0 0.0 665 9967 36.5 0 0 0.0 99.3 da12 24 740 0 0 0.0 740 10958 32.2 0 0 0.0 98.7 da13 24 639 0 0 0.0 639 9619 37.4 0 0 0.0 99.7 da14 24 695 0 0 0.0 695 10367 35.1 0 0 0.0 99.8 da15 24 583 0 0 0.0 583 8429 41.5 0 0 0.0 100.2 da16 24 697 0 0 0.0 697 10287 34.3 0 0 0.0 100.0 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada1 dT: 1.001s w: 1.000s filter: da[0-9]*$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 24 895 0 0 0.0 895 14773 26.9 0 0 0.0 101.6 da0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da4 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da5 24 702 0 0 0.0 702 9327 33.8 0 0 0.0 97.6 da6 24 675 0 0 0.0 675 9191 35.3 0 0 0.0 100.1 da7 24 668 0 0 0.0 668 8575 36.1 0 0 0.0 100.7 da8 24 744 0 0 0.0 744 9179 31.4 0 0 0.0 97.1 da9 24 750 0 0 0.0 750 9003 32.3 0 0 0.0 100.1 da10 24 586 0 0 0.0 586 6909 40.9 0 0 0.0 100.0 da11 24 711 0 0 0.0 711 13990 33.6 0 0 0.0 99.4 da12 24 653 0 0 0.0 653 10074 36.5 0 0 0.0 100.0 da13 24 645 0 0 0.0 645 13111 37.0 0 0 0.0 100.0 da14 24 687 0 0 0.0 687 12795 34.8 0 0 0.0 99.9 da15 24 678 0 0 0.0 678 12855 35.6 0 0 0.0 100.6 da16 24 696 0 0 0.0 696 13091 34.4 0 0 0.0 99.2 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada1 dT: 1.001s w: 1.000s filter: da[0-9]*$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 24 1177 0 0 0.0 1177 18586 20.6 0 0 0.0 100.0 da0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da4 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da5 24 780 0 0 0.0 780 11733 30.6 0 0 0.0 99.3 da6 23 776 0 0 0.0 776 12636 31.1 0 0 0.0 100.2 da7 24 636 0 0 0.0 636 9387 37.8 0 0 0.0 99.7 da8 24 778 0 0 0.0 778 13343 31.2 0 0 0.0 100.6 da9 24 821 0 0 0.0 821 14202 29.3 0 0 0.0 101.0 da10 24 786 0 0 0.0 786 12024 30.6 0 0 0.0 100.0 da11 24 666 0 0 0.0 666 9351 35.9 0 0 0.0 100.0 da12 24 720 0 0 0.0 720 9954 33.1 0 0 0.0 99.7 da13 24 706 0 0 0.0 706 10014 34.0 0 0 0.0 99.8 da14 24 801 0 0 0.0 801 11105 30.3 0 0 0.0 101.1 da15 24 738 0 0 0.0 738 10126 32.5 0 0 0.0 99.9 da16 24 670 0 0 0.0 670 9203 36.5 0 0 0.0 101.4 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada1 dT: 1.001s w: 1.000s filter: da[0-9]*$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 23 1650 0 0 0.0 1650 18674 14.4 0 0 0.0 100.0 da0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da4 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da5 24 669 0 0 0.0 669 13234 35.9 0 0 0.0 100.1 da6 24 573 0 0 0.0 573 12223 42.4 0 0 0.0 99.7 da7 23 519 0 0 0.0 519 10584 46.3 0 0 0.0 100.1 da8 24 659 0 0 0.0 659 15552 36.9 0 0 0.0 100.5 da9 24 687 0 0 0.0 687 18410 35.1 0 0 0.0 99.3 da10 23 671 0 0 0.0 671 13190 36.0 0 0 0.0 100.2 da11 24 807 0 0 0.0 807 9425 29.7 0 0 0.0 99.9 da12 24 665 0 0 0.0 665 7902 36.4 0 0 0.0 100.3 da13 24 861 0 0 0.0 861 10140 28.0 0 0 0.0 99.6 da14 24 714 0 0 0.0 714 8797 33.4 0 0 0.0 99.7 da15 24 753 0 0 0.0 753 9225 32.5 0 0 0.0 100.0 da16 24 760 0 0 0.0 760 9337 31.5 0 0 0.0 100.0 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada1 dT: 1.001s w: 1.000s filter: da[0-9]*$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da4 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da5 24 648 0 0 0.0 648 9954 37.0 0 0 0.0 100.0 da6 24 664 0 0 0.0 664 10194 36.1 0 0 0.0 100.0 da7 24 653 0 0 0.0 653 13163 36.7 0 0 0.0 99.9 da8 24 749 0 0 0.0 749 12716 31.8 0 0 0.0 100.4 da9 24 769 0 0 0.0 769 13239 31.4 0 0 0.0 100.3 da10 24 742 0 0 0.0 742 12017 32.4 0 0 0.0 100.1 da11 24 584 0 0 0.0 584 7393 40.8 0 0 0.0 101.1 da12 24 648 0 0 0.0 648 8336 37.7 0 0 0.0 99.7 da13 24 679 0 0 0.0 679 8756 35.3 0 0 0.0 100.8 da14 24 646 0 0 0.0 646 8160 37.0 0 0 0.0 100.5 da15 24 692 0 0 0.0 692 8900 35.5 0 0 0.0 99.9 da16 24 684 0 0 0.0 684 8692 35.6 0 0 0.0 101.7 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada1 dT: 1.001s w: 1.000s filter: da[0-9]*$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da4 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da5 24 662 0 0 0.0 662 14641 35.8 0 0 0.0 100.0 da6 24 705 0 0 0.0 705 16535 34.2 0 0 0.0 99.1 da7 24 644 0 0 0.0 644 10120 37.4 0 0 0.0 101.8 da8 24 560 0 0 0.0 560 14721 43.4 0 0 0.0 99.5 da9 24 677 0 0 0.0 677 20864 35.2 0 0 0.0 100.0 da10 24 685 0 0 0.0 685 19009 35.6 0 0 0.0 100.3 da11 24 714 0 0 0.0 714 10328 34.0 0 0 0.0 101.7 da12 24 738 0 0 0.0 738 10476 32.2 0 0 0.0 100.0 da13 24 587 0 0 0.0 587 8465 40.6 0 0 0.0 100.1 da14 24 720 0 0 0.0 720 9888 33.8 0 0 0.0 100.1 da15 24 603 0 0 0.0 603 8258 39.1 0 0 0.0 100.1 da16 24 623 0 0 0.0 623 9025 38.3 0 0 0.0 99.9 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada1 dT: 1.001s w: 1.000s filter: da[0-9]*$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da4 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da5 0 76 0 0 0.0 76 1702 39.6 0 0 0.0 10.9 da6 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da7 0 298 0 0 0.0 298 8704 33.8 0 0 0.0 40.5 da8 0 112 0 0 0.0 112 3065 48.0 0 0 0.0 22.1 da9 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da10 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da11 24 645 0 0 0.0 645 9942 37.3 0 0 0.0 100.9 da12 24 581 0 0 0.0 581 8660 41.1 0 0 0.0 99.8 da13 24 717 0 0 0.0 717 10910 33.6 0 0 0.0 101.4 da14 24 737 0 0 0.0 737 11737 32.6 0 0 0.0 99.9 da15 24 556 0 0 0.0 556 8796 42.3 0 0 0.0 99.5 da16 24 538 0 0 0.0 538 8444 44.1 0 0 0.0 100.1 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada1 dT: 1.001s w: 1.000s filter: da[0-9]*$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da4 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da5 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da6 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da7 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da8 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da9 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da10 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da11 0 642 0 0 0.0 642 8875 37.2 0 0 0.0 100.0 da12 24 672 0 0 0.0 672 9323 36.0 0 0 0.0 100.0 da13 24 765 0 0 0.0 765 10602 31.5 0 0 0.0 101.6 da14 0 684 0 0 0.0 684 9902 31.0 0 0 0.0 87.1 da15 23 683 0 0 0.0 683 10210 35.6 0 0 0.0 100.4 da16 24 654 0 0 0.0 654 9603 37.1 0 0 0.0 99.9 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada1 dT: 1.000s w: 1.000s filter: da[0-9]*$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 114 109 3811 1.9 4 16 0.5 0 0 0.0 12.6 da0 0 349 343 12129 0.7 4 16 0.5 0 0 0.0 24.5 da1 0 232 226 8322 1.0 4 16 0.5 0 0 0.0 23.6 da2 0 230 224 8338 2.7 4 16 0.5 0 0 0.0 28.0 da3 0 349 343 12129 0.7 4 16 0.5 0 0 0.0 23.9 da4 0 122 116 3775 1.1 4 16 0.5 0 0 0.0 21.5 da5 0 5 0 0 0.0 4 16 0.4 0 0 0.0 4.8 da6 0 5 0 0 0.0 4 16 0.3 0 0 0.0 4.2 da7 0 356 351 12577 0.8 4 16 0.4 0 0 0.0 14.4 da8 0 349 344 12573 0.7 4 16 0.3 0 0 0.0 15.5 da9 0 359 354 12573 0.6 4 16 0.4 0 0 0.0 9.5 da10 0 360 355 12589 1.0 4 16 0.3 0 0 0.0 14.0 da11 0 6 0 0 0.0 4 16 0.3 0 0 0.0 16.9 da12 0 5 0 0 0.0 4 16 0.3 0 0 0.0 3.6 da13 0 5 0 0 0.0 4 16 0.3 0 0 0.0 5.3 da14 0 5 0 0 0.0 4 16 0.3 0 0 0.0 3.8 da15 0 5 0 0 0.0 4 16 0.3 0 0 0.0 4.9 da16 0 9 4 16 15.7 4 16 0.4 0 0 0.0 5.8 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da20 0 1 0 0 0.0 0 0 0.0 1 3839 0.4 0.0 ada0 0 2 0 0 0.0 0 0 0.0 2 3839 0.5 0.0 ada1 ^C There were some heavy writes to da20 (the L2ARC) but no delete requests. While deletes on ada0 and ada1 do happen as expected. The L2ARC should delete, as it is nearly full. zpool iostat -v storage 10 capacity operations bandwidth pool alloc free read write read write -------------- ----- ----- ----- ----- ----- ----- storage 44.8T 3.95T 77 9.34K 5.03M 43.1M raidz2 15.1T 1.13T 27 3.09K 2.37M 14.1M gpt/disk00 - - 8 485 303K 6.66M gpt/disk01 - - 16 477 603K 6.56M gpt/disk02 - - 9 476 320K 6.56M gpt/disk03 - - 8 476 322K 6.56M gpt/disk04 - - 16 477 603K 6.56M gpt/disk05 - - 8 476 301K 6.55M raidz2 14.7T 1.53T 32 3.13K 2.57M 14.5M gpt/disk06 - - 14 444 502K 6.74M gpt/disk07 - - 11 444 385K 6.74M gpt/disk08 - - 14 444 552K 6.74M gpt/disk09 - - 15 445 582K 6.73M gpt/disk10 - - 7 444 280K 6.72M gpt/disk11 - - 9 444 369K 6.72M raidz2 15.0T 1.29T 17 3.10K 87.0K 14.1M gpt/disk12 - - 4 434 40.8K 6.65M gpt/disk13 - - 2 434 16.3K 6.65M gpt/disk14 - - 0 433 8.75K 6.62M gpt/disk15 - - 4 435 40.6K 6.66M gpt/disk16 - - 2 433 15.3K 6.61M gpt/disk17 - - 0 434 8.54K 6.65M logs - - - - - - ada0p2 1.01M 7.94G 0 2 0 185K ada1p2 1.01M 7.94G 0 2 0 185K cache - - - - - - da20p2 215G 80M 439 674 703K 2.84M -------------- ----- ----- ----- ----- ----- ----- Daniel