Date: Tue, 16 Jul 2013 15:16:38 +0300 From: Daniel Kalchev <daniel@digsys.bg> To: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: ZFS vdev I/O questions Message-ID: <51E539A6.9090109@digsys.bg> In-Reply-To: <3472068604314C9887FE3BD4CD42B7C8@multiplay.co.uk> References: <51E5316B.9070201@digsys.bg> <3472068604314C9887FE3BD4CD42B7C8@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
On 16.07.13 14:53, Mark Felder wrote:
>> The other observation I have is with the first vdev: the 512b drives do
>> a lot of I/O fast, complete first and then sit idle, while da0 continues
>> to write for many more seconds. They consistently show many more IOPS
>> than the other drives for this type of activity -- on streaming writes
>> all drives behave more or less the same. It is only on this un-dedup
>> scenario where the difference is so much pronounced.
>>
>> All the vdevs in the pool are with ashift=12 so the theory that ZFS
>> actually issues 512b writes to these drives can't be true, can it?
>>
>> Another worry is this Seagate Barracuda 7200.14 (AF)
>> (ST3000DM001-1CH166, firmware CC24) drive. It seems constantly
>> under-performing. Does anyone know if it is so different from the
>> ST3000DM001-9YN166 drives? Might be, I should just replace it?
>>
> A lot of information here.
>
> Those Hitachis are great drives. The addition of the Barracuda with
> different performance characteristics could be part of the problem. I'm
> glad you pointed out that the pool ashift=12 so we can try to rule that
> out. I'd be quite interested in knowing if some or perhaps even all of
> your issues go away simply by replacing that drive with another Hitachi.
I don't have any more of these available and the vendor unfortunately
could not supply more (but I am considering looking elsewhere)
At the moment, I could only replace that Barracuda with one of the spare
drives, which are Seagate SV35 (ST3000VX000-9YW166, firmware CV13)
I have observed that these drives are not particularly good at random
I/O however....
On 16.07.13 14:58, Steven Hartland wrote:
> One thing to check with this is to add -d to your gstat to see
> if your waiting deletes IO? I doubt it but worth checking.
Here is with gstat -d. Actually, running this script makes more sense:
while true ;do
gstat -f 'da[0-9]*$' -d -b
sleep 1
done
(small random reads, heavy writes, then some reads)
dT: 1.001s w: 1.000s filter: da[0-9]*$
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
ms/d %busy Name
0 11 1 4 20.8 10 44 0.4 0 0
0.0 2.2 da0
0 15 5 20 7.1 10 40 5.8 0 0
0.0 5.1 da1
0 7 0 0 0.0 7 28 0.4 0 0
0.0 0.2 da2
1 7 0 0 0.0 7 28 4.4 0 0
0.0 1.1 da3
1 11 0 0 0.0 11 44 0.5 0 0
0.0 0.2 da4
0 11 0 0 0.0 11 44 0.4 0 0
0.0 0.2 da5
0 25 16 68 7.8 9 36 0.4 0 0
0.0 5.4 da6
0 21 12 52 14.2 9 36 0.3 0 0
0.0 6.0 da7
0 9 0 0 0.0 9 36 0.3 0 0
0.0 0.1 da8
0 35 23 176 12.5 12 48 0.4 0 0
0.0 9.4 da9
0 26 17 140 13.7 9 36 0.3 0 0
0.0 7.3 da10
0 8 0 0 0.0 8 32 0.3 0 0
0.0 0.1 da11
0 158 29 212 17.4 129 4083 81.9 0 0
0.0 57.8 da12
0 162 25 192 16.7 137 4423 84.8 0 0
0.0 61.9 da13
0 118 0 0 0.0 118 3844 95.5 0 0
0.0 45.6 da14
5 152 20 112 10.8 132 4127 66.9 0 0
0.0 45.8 da15
0 171 14 88 17.4 157 4990 89.2 0 0
0.0 69.6 da16
0 124 1 4 6.6 123 3840 71.5 0 0
0.0 37.6 da17
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da19
0 4558 4557 6736 0.5 1 1 0.2 0 0
0.0 14.7 da20
0 120 0 0 0.0 119 8083 35.2 0 0
0.0 17.4 ada0
0 120 0 0 0.0 119 8083 34.0 0 0
0.0 16.8 ada1
dT: 1.001s w: 1.000s filter: da[0-9]*$
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
ms/d %busy Name
0 1 1 4 14.0 0 0 0.0 0 0
0.0 1.4 da0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da1
0 2 2 8 18.8 0 0 0.0 0 0
0.0 3.7 da2
0 1 1 4 35.1 0 0 0.0 0 0
0.0 3.5 da3
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da4
1 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da5
0 3 3 12 12.8 0 0 0.0 0 0
0.0 3.8 da6
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da7
0 3 3 12 14.3 0 0 0.0 0 0
0.0 4.3 da8
0 1 1 4 9.8 0 0 0.0 0 0
0.0 1.0 da9
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da10
0 7 7 28 9.8 0 0 0.0 0 0
0.0 6.8 da11
0 1 1 4 7.4 0 0 0.0 0 0
0.0 0.7 da12
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da13
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da14
0 3 3 12 34.0 0 0 0.0 0 0
0.0 10.2 da15
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da16
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da17
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da19
0 2656 236 455 0.2 2420 12284 0.3 0 0
0.0 21.3 da20
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada1
dT: 1.001s w: 1.000s filter: da[0-9]*$
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
ms/d %busy Name
0 4 4 16 22.8 0 0 0.0 0 0
0.0 9.1 da0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da1
1 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da2
0 1 1 4 17.4 0 0 0.0 0 0
0.0 1.7 da3
0 3 3 12 13.2 0 0 0.0 0 0
0.0 4.0 da4
0 2 2 8 28.2 0 0 0.0 0 0
0.0 5.6 da5
0 1 1 4 9.7 0 0 0.0 0 0
0.0 1.0 da6
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da7
0 3 3 12 8.8 0 0 0.0 0 0
0.0 2.6 da8
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da9
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da10
0 5 5 20 10.1 0 0 0.0 0 0
0.0 5.0 da11
0 4 4 16 10.4 0 0 0.0 0 0
0.0 4.1 da12
0 1 1 4 16.0 0 0 0.0 0 0
0.0 1.6 da13
0 1 1 4 13.8 0 0 0.0 0 0
0.0 1.4 da14
0 4 4 16 7.3 0 0 0.0 0 0
0.0 2.9 da15
0 1 1 4 13.1 0 0 0.0 0 0
0.0 1.3 da16
0 1 1 4 21.3 0 0 0.0 0 0
0.0 2.1 da17
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da19
0 515 515 979 0.2 0 0 0.0 0 0
0.0 9.2 da20
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada1
dT: 1.001s w: 1.000s filter: da[0-9]*$
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
ms/d %busy Name
0 1 1 4 32.3 0 0 0.0 0 0
0.0 3.2 da0
0 1 1 4 35.0 0 0 0.0 0 0
0.0 3.5 da1
0 4 4 16 4.3 0 0 0.0 0 0
0.0 1.7 da2
0 3 3 12 31.8 0 0 0.0 0 0
0.0 9.5 da3
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da4
0 4 4 16 6.9 0 0 0.0 0 0
0.0 2.8 da5
0 2 2 8 13.6 0 0 0.0 0 0
0.0 2.7 da6
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da7
0 4 4 16 7.4 0 0 0.0 0 0
0.0 3.0 da8
0 1 1 4 12.6 0 0 0.0 0 0
0.0 1.3 da9
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da10
0 6 6 24 10.0 0 0 0.0 0 0
0.0 6.0 da11
0 1 1 4 21.6 0 0 0.0 0 0
0.0 2.2 da12
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da13
1 1 1 4 10.8 0 0 0.0 0 0
0.0 1.1 da14
0 1 1 4 18.3 0 0 0.0 0 0
0.0 1.8 da15
0 1 1 4 9.9 0 0 0.0 0 0
0.0 1.0 da16
0 1 1 4 21.6 0 0 0.0 0 0
0.0 2.2 da17
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da19
0 588 587 1130 0.2 1 2 0.6 0 0
0.0 10.5 da20
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada1
dT: 1.001s w: 1.000s filter: da[0-9]*$
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
ms/d %busy Name
23 1412 0 0 0.0 1412 24423 15.0 0 0
0.0 99.6 da0
23 4699 0 0 0.0 4699 72429 4.1 0 0
0.0 97.2 da1
24 4294 0 0 0.0 4294 67749 4.6 0 0
0.0 99.0 da2
23 4923 0 0 0.0 4923 75746 3.8 0 0
0.0 96.2 da3
24 4452 0 0 0.0 4452 67729 4.3 0 0
0.0 97.5 da4
23 4558 0 0 0.0 4558 68992 4.2 0 0
0.0 97.3 da5
24 813 0 0 0.0 813 10675 27.2 0 0
0.0 99.6 da6
24 866 0 0 0.0 866 11338 25.5 0 0
0.0 99.5 da7
24 845 0 0 0.0 845 11058 25.8 0 0
0.0 99.5 da8
24 866 0 0 0.0 866 11174 25.3 0 0
0.0 98.8 da9
24 928 0 0 0.0 928 12061 23.6 0 0
0.0 99.2 da10
24 864 0 0 0.0 864 11150 25.6 0 0
0.0 99.2 da11
24 674 0 0 0.0 674 14587 32.4 0 0
0.0 99.9 da12
24 705 0 0 0.0 705 15043 31.7 0 0 0.0
100.1 da13
24 670 0 0 0.0 670 14527 33.0 0 0
0.0 99.5 da14
24 648 0 0 0.0 648 13360 34.3 0 0
0.0 99.5 da15
24 716 0 0 0.0 716 14835 30.9 0 0
0.0 99.5 da16
24 730 0 0 0.0 730 15227 30.0 0 0
0.0 98.7 da17
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da19
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da20
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada1
dT: 1.001s w: 1.000s filter: da[0-9]*$
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
ms/d %busy Name
24 1169 0 0 0.0 1169 15954 20.6 0 0 0.0
100.3 da0
0 1811 0 0 0.0 1811 20961 3.9 0 0
0.0 29.6 da1
0 2392 0 0 0.0 2392 27444 3.7 0 0
0.0 37.1 da2
0 1199 0 0 0.0 1199 14571 3.7 0 0
0.0 18.5 da3
0 2305 0 0 0.0 2305 26924 4.0 0 0
0.0 38.1 da4
0 1820 0 0 0.0 1820 21573 3.8 0 0
0.0 28.8 da5
24 744 0 0 0.0 744 9248 32.6 0 0
0.0 99.5 da6
24 771 0 0 0.0 771 10722 31.1 0 0 0.0
100.0 da7
24 716 0 0 0.0 716 9268 33.5 0 0 0.0
100.2 da8
24 754 0 0 0.0 754 9280 31.8 0 0
0.0 99.9 da9
24 769 0 0 0.0 769 9384 31.4 0 0
0.0 99.9 da10
24 742 0 0 0.0 742 9508 32.1 0 0
0.0 98.5 da11
24 665 0 0 0.0 665 9967 36.5 0 0
0.0 99.3 da12
24 740 0 0 0.0 740 10958 32.2 0 0
0.0 98.7 da13
24 639 0 0 0.0 639 9619 37.4 0 0
0.0 99.7 da14
24 695 0 0 0.0 695 10367 35.1 0 0
0.0 99.8 da15
24 583 0 0 0.0 583 8429 41.5 0 0 0.0
100.2 da16
24 697 0 0 0.0 697 10287 34.3 0 0 0.0
100.0 da17
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da19
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da20
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada1
dT: 1.001s w: 1.000s filter: da[0-9]*$
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
ms/d %busy Name
24 895 0 0 0.0 895 14773 26.9 0 0 0.0
101.6 da0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da1
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da2
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da3
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da4
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da5
24 702 0 0 0.0 702 9327 33.8 0 0
0.0 97.6 da6
24 675 0 0 0.0 675 9191 35.3 0 0 0.0
100.1 da7
24 668 0 0 0.0 668 8575 36.1 0 0 0.0
100.7 da8
24 744 0 0 0.0 744 9179 31.4 0 0
0.0 97.1 da9
24 750 0 0 0.0 750 9003 32.3 0 0 0.0
100.1 da10
24 586 0 0 0.0 586 6909 40.9 0 0 0.0
100.0 da11
24 711 0 0 0.0 711 13990 33.6 0 0
0.0 99.4 da12
24 653 0 0 0.0 653 10074 36.5 0 0 0.0
100.0 da13
24 645 0 0 0.0 645 13111 37.0 0 0 0.0
100.0 da14
24 687 0 0 0.0 687 12795 34.8 0 0
0.0 99.9 da15
24 678 0 0 0.0 678 12855 35.6 0 0 0.0
100.6 da16
24 696 0 0 0.0 696 13091 34.4 0 0
0.0 99.2 da17
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da19
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da20
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada1
dT: 1.001s w: 1.000s filter: da[0-9]*$
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
ms/d %busy Name
24 1177 0 0 0.0 1177 18586 20.6 0 0 0.0
100.0 da0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da1
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da2
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da3
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da4
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da5
24 780 0 0 0.0 780 11733 30.6 0 0
0.0 99.3 da6
23 776 0 0 0.0 776 12636 31.1 0 0 0.0
100.2 da7
24 636 0 0 0.0 636 9387 37.8 0 0
0.0 99.7 da8
24 778 0 0 0.0 778 13343 31.2 0 0 0.0
100.6 da9
24 821 0 0 0.0 821 14202 29.3 0 0 0.0
101.0 da10
24 786 0 0 0.0 786 12024 30.6 0 0 0.0
100.0 da11
24 666 0 0 0.0 666 9351 35.9 0 0 0.0
100.0 da12
24 720 0 0 0.0 720 9954 33.1 0 0
0.0 99.7 da13
24 706 0 0 0.0 706 10014 34.0 0 0
0.0 99.8 da14
24 801 0 0 0.0 801 11105 30.3 0 0 0.0
101.1 da15
24 738 0 0 0.0 738 10126 32.5 0 0
0.0 99.9 da16
24 670 0 0 0.0 670 9203 36.5 0 0 0.0
101.4 da17
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da19
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da20
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada1
dT: 1.001s w: 1.000s filter: da[0-9]*$
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
ms/d %busy Name
23 1650 0 0 0.0 1650 18674 14.4 0 0 0.0
100.0 da0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da1
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da2
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da3
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da4
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da5
24 669 0 0 0.0 669 13234 35.9 0 0 0.0
100.1 da6
24 573 0 0 0.0 573 12223 42.4 0 0
0.0 99.7 da7
23 519 0 0 0.0 519 10584 46.3 0 0 0.0
100.1 da8
24 659 0 0 0.0 659 15552 36.9 0 0 0.0
100.5 da9
24 687 0 0 0.0 687 18410 35.1 0 0
0.0 99.3 da10
23 671 0 0 0.0 671 13190 36.0 0 0 0.0
100.2 da11
24 807 0 0 0.0 807 9425 29.7 0 0
0.0 99.9 da12
24 665 0 0 0.0 665 7902 36.4 0 0 0.0
100.3 da13
24 861 0 0 0.0 861 10140 28.0 0 0
0.0 99.6 da14
24 714 0 0 0.0 714 8797 33.4 0 0
0.0 99.7 da15
24 753 0 0 0.0 753 9225 32.5 0 0 0.0
100.0 da16
24 760 0 0 0.0 760 9337 31.5 0 0 0.0
100.0 da17
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da19
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da20
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada1
dT: 1.001s w: 1.000s filter: da[0-9]*$
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
ms/d %busy Name
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da1
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da2
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da3
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da4
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da5
24 648 0 0 0.0 648 9954 37.0 0 0 0.0
100.0 da6
24 664 0 0 0.0 664 10194 36.1 0 0 0.0
100.0 da7
24 653 0 0 0.0 653 13163 36.7 0 0
0.0 99.9 da8
24 749 0 0 0.0 749 12716 31.8 0 0 0.0
100.4 da9
24 769 0 0 0.0 769 13239 31.4 0 0 0.0
100.3 da10
24 742 0 0 0.0 742 12017 32.4 0 0 0.0
100.1 da11
24 584 0 0 0.0 584 7393 40.8 0 0 0.0
101.1 da12
24 648 0 0 0.0 648 8336 37.7 0 0
0.0 99.7 da13
24 679 0 0 0.0 679 8756 35.3 0 0 0.0
100.8 da14
24 646 0 0 0.0 646 8160 37.0 0 0 0.0
100.5 da15
24 692 0 0 0.0 692 8900 35.5 0 0
0.0 99.9 da16
24 684 0 0 0.0 684 8692 35.6 0 0 0.0
101.7 da17
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da19
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da20
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada1
dT: 1.001s w: 1.000s filter: da[0-9]*$
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
ms/d %busy Name
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da1
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da2
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da3
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da4
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da5
24 662 0 0 0.0 662 14641 35.8 0 0 0.0
100.0 da6
24 705 0 0 0.0 705 16535 34.2 0 0
0.0 99.1 da7
24 644 0 0 0.0 644 10120 37.4 0 0 0.0
101.8 da8
24 560 0 0 0.0 560 14721 43.4 0 0
0.0 99.5 da9
24 677 0 0 0.0 677 20864 35.2 0 0 0.0
100.0 da10
24 685 0 0 0.0 685 19009 35.6 0 0 0.0
100.3 da11
24 714 0 0 0.0 714 10328 34.0 0 0 0.0
101.7 da12
24 738 0 0 0.0 738 10476 32.2 0 0 0.0
100.0 da13
24 587 0 0 0.0 587 8465 40.6 0 0 0.0
100.1 da14
24 720 0 0 0.0 720 9888 33.8 0 0 0.0
100.1 da15
24 603 0 0 0.0 603 8258 39.1 0 0 0.0
100.1 da16
24 623 0 0 0.0 623 9025 38.3 0 0
0.0 99.9 da17
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da19
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da20
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada1
dT: 1.001s w: 1.000s filter: da[0-9]*$
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
ms/d %busy Name
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da1
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da2
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da3
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da4
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da5
0 76 0 0 0.0 76 1702 39.6 0 0
0.0 10.9 da6
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da7
0 298 0 0 0.0 298 8704 33.8 0 0
0.0 40.5 da8
0 112 0 0 0.0 112 3065 48.0 0 0
0.0 22.1 da9
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da10
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da11
24 645 0 0 0.0 645 9942 37.3 0 0 0.0
100.9 da12
24 581 0 0 0.0 581 8660 41.1 0 0
0.0 99.8 da13
24 717 0 0 0.0 717 10910 33.6 0 0 0.0
101.4 da14
24 737 0 0 0.0 737 11737 32.6 0 0
0.0 99.9 da15
24 556 0 0 0.0 556 8796 42.3 0 0
0.0 99.5 da16
24 538 0 0 0.0 538 8444 44.1 0 0 0.0
100.1 da17
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da19
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da20
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada1
dT: 1.001s w: 1.000s filter: da[0-9]*$
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
ms/d %busy Name
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da1
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da2
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da3
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da4
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da5
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da6
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da7
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da8
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da9
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da10
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da11
0 642 0 0 0.0 642 8875 37.2 0 0 0.0
100.0 da12
24 672 0 0 0.0 672 9323 36.0 0 0 0.0
100.0 da13
24 765 0 0 0.0 765 10602 31.5 0 0 0.0
101.6 da14
0 684 0 0 0.0 684 9902 31.0 0 0
0.0 87.1 da15
23 683 0 0 0.0 683 10210 35.6 0 0 0.0
100.4 da16
24 654 0 0 0.0 654 9603 37.1 0 0
0.0 99.9 da17
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da19
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da20
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada0
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 ada1
dT: 1.000s w: 1.000s filter: da[0-9]*$
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
ms/d %busy Name
0 114 109 3811 1.9 4 16 0.5 0 0
0.0 12.6 da0
0 349 343 12129 0.7 4 16 0.5 0 0
0.0 24.5 da1
0 232 226 8322 1.0 4 16 0.5 0 0
0.0 23.6 da2
0 230 224 8338 2.7 4 16 0.5 0 0
0.0 28.0 da3
0 349 343 12129 0.7 4 16 0.5 0 0
0.0 23.9 da4
0 122 116 3775 1.1 4 16 0.5 0 0
0.0 21.5 da5
0 5 0 0 0.0 4 16 0.4 0 0
0.0 4.8 da6
0 5 0 0 0.0 4 16 0.3 0 0
0.0 4.2 da7
0 356 351 12577 0.8 4 16 0.4 0 0
0.0 14.4 da8
0 349 344 12573 0.7 4 16 0.3 0 0
0.0 15.5 da9
0 359 354 12573 0.6 4 16 0.4 0 0
0.0 9.5 da10
0 360 355 12589 1.0 4 16 0.3 0 0
0.0 14.0 da11
0 6 0 0 0.0 4 16 0.3 0 0
0.0 16.9 da12
0 5 0 0 0.0 4 16 0.3 0 0
0.0 3.6 da13
0 5 0 0 0.0 4 16 0.3 0 0
0.0 5.3 da14
0 5 0 0 0.0 4 16 0.3 0 0
0.0 3.8 da15
0 5 0 0 0.0 4 16 0.3 0 0
0.0 4.9 da16
0 9 4 16 15.7 4 16 0.4 0 0
0.0 5.8 da17
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da19
0 0 0 0 0.0 0 0 0.0 0 0
0.0 0.0 da20
0 1 0 0 0.0 0 0 0.0 1 3839
0.4 0.0 ada0
0 2 0 0 0.0 0 0 0.0 2 3839
0.5 0.0 ada1
^C
There were some heavy writes to da20 (the L2ARC) but no delete requests.
While deletes on ada0 and ada1 do happen as expected. The L2ARC should
delete, as it is nearly full.
zpool iostat -v storage 10
capacity operations bandwidth
pool alloc free read write read write
-------------- ----- ----- ----- ----- ----- -----
storage 44.8T 3.95T 77 9.34K 5.03M 43.1M
raidz2 15.1T 1.13T 27 3.09K 2.37M 14.1M
gpt/disk00 - - 8 485 303K 6.66M
gpt/disk01 - - 16 477 603K 6.56M
gpt/disk02 - - 9 476 320K 6.56M
gpt/disk03 - - 8 476 322K 6.56M
gpt/disk04 - - 16 477 603K 6.56M
gpt/disk05 - - 8 476 301K 6.55M
raidz2 14.7T 1.53T 32 3.13K 2.57M 14.5M
gpt/disk06 - - 14 444 502K 6.74M
gpt/disk07 - - 11 444 385K 6.74M
gpt/disk08 - - 14 444 552K 6.74M
gpt/disk09 - - 15 445 582K 6.73M
gpt/disk10 - - 7 444 280K 6.72M
gpt/disk11 - - 9 444 369K 6.72M
raidz2 15.0T 1.29T 17 3.10K 87.0K 14.1M
gpt/disk12 - - 4 434 40.8K 6.65M
gpt/disk13 - - 2 434 16.3K 6.65M
gpt/disk14 - - 0 433 8.75K 6.62M
gpt/disk15 - - 4 435 40.6K 6.66M
gpt/disk16 - - 2 433 15.3K 6.61M
gpt/disk17 - - 0 434 8.54K 6.65M
logs - - - - - -
ada0p2 1.01M 7.94G 0 2 0 185K
ada1p2 1.01M 7.94G 0 2 0 185K
cache - - - - - -
da20p2 215G 80M 439 674 703K 2.84M
-------------- ----- ----- ----- ----- ----- -----
Daniel
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?51E539A6.9090109>