Date: Wed, 08 Jul 2015 11:37:09 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-fs@freebsd.org Subject: Re: Speeding up resilvering Message-ID: <559D51B5.3050709@denninger.net> In-Reply-To: <559D4DD3.10903@infracaninophile.co.uk> References: <559D2AB6.5070007@FreeBSD.org> <559D2C7C.6060201@denninger.net> <559D3167.1000705@infracaninophile.co.uk> <559D3236.1060102@denninger.net> <559D3380.8050703@sentex.net> <559D4DD3.10903@infracaninophile.co.uk>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
On 7/8/2015 11:20, Matthew Seaman wrote:
> On 07/08/15 15:28, Mike Tancsa wrote:
>>> Look at the IO saturation on the disk channel(s) involved with either
>>>> systat -vm or iostat. If the channel is saturated then there's nothing
>>>> you can do in terms of tuning; the question then turns to why actual I/O
>>>> performance is so poor and has to be addressed there.
>> I had one server that was taking ages, and it turned out to be the
>> controller, not the disk that was hosed. As Karl suggested, take a look
>> at the throughput on gstat. Is anyone disk or groups of disks lagging
>> far behind on write speeds ? In my case, it was a very obvious and
>> glaring outlier.
> Thanks for the suggestions. I've been playing with gstat et al, and as
> far as I can tell, all the drives are behaving reasonably well. I'm
> certainly getting 90-100% capacity (mostly reads) continually on the
> original drives whilst the new one (mostly writes) seems to go in
> bursts. Which is pretty much what I'd expect when resilvering a RAIDZ.
>
> So, having exhausted that, I actually sat down and timed what progress
> it was making rather more carefully. Turns out my impression that
> applying the sysctl tweaks I mentioned previously had little effect was
> wrong. Current projection is about 50h total to do the resilvering,
> which is much, much better than the approx 12days I was expecting
> previously. In fact, that's pretty much inline with what I'd expect
> from this hardware.
>
> Cheers,
>
> Matthew
>
>
>
Be aware that when a resilver /starts /it does a number of very small
and diverse I/O operations. Throughput during that time _*stinks*_, and
as a result in the first few minutes to half-hour or so (with a very
large vdev) you will see utterly ridiculous projections of completion time.
Once that part of the process completes (usually within a few minutes on
modest to moderate size vdevs, but it will be longer on very large ones)
the performance level of the resilver will go up a lot and the projected
completion time will become far more reasonable.
--
Karl Denninger
karl@denninger.net <mailto:karl@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
[-- Attachment #2 --]
0 *H
01 0 + 0 *H
_0[0C)0
*H
01 0 UUS10UFlorida10U Niceville10U
Cuda Systems LLC1 0UCuda Systems LLC CA1"0 *H
Cuda Systems LLC CA0
150421022159Z
200419022159Z0Z1 0 UUS10UFlorida10U
Cuda Systems LLC1 0 UKarl Denninger (OCSP)0"0
*H
0
X@vkY
Tq/vE]5#֯MX \8LJ/V?5Da+
sJc*/r{ȼnS+ w")ąZ^DtdCOZ ~7Q '@a#ijc۴oZdB&!Ӝ-< ? HN5y
5}F|e f"V لio74zn">a1qWuɖbFeGE&3(KhixG3!#e_XƬ Ϝ/,$+;4y 'Bz<qT9_?rRUpn5
Jn&Rx/p Jyel*pN8/#9u/YP EC)TY>~/˘N[vyiDKˉ ,^" ?$T8 v& K%z8C @?K{9f`+@,| Mbia 007++0)0'+0http://cudasystems.net:88880 U 0 0 `HB0 U 0, `HB
OpenSSL Generated Certificate0 U -h\Ff Y0 U #0$q}ݽʒm50 U 0karl@denninger.net0
*H
Owbabɺx&Uk[ (Oj!%p MQ0I!#QH}.>~2&D}<wm_>V6 v ]f>=Nn+8;q wfΰ/RLyUG#b}n!Dր _up|_ǰc/%ۥ
nN8:d;-UJd/m1~VނיnN I˾$tF1&}|?q?\đXԑ&\4V<lKۮ3%Am_(q -(cAeGX)f}-˥6cv~Kg8m~v ;|9:-iAPқ6ېn-.)<[$KJtt/L4ᖣ^Cmu4vb{+BG$M0c\[MR|0FԸP&78"4p# }D Z9;V9#>Sw"[UP710001 0 UUS10UFlorida10U Niceville10U
Cuda Systems LLC1 0UCuda Systems LLC CA1"0 *H
Cuda Systems LLC CA)0 + !0 *H
1 *H
0 *H
1
150708163709Z0# *H
1^TG%<NF Rqyu0l *H
1_0]0 `He*0 `He0
*H
0*H
0
*H
@0+0
*H
(0 +71001 0 UUS10UFlorida10U Niceville10U
Cuda Systems LLC1 0UCuda Systems LLC CA1"0 *H
Cuda Systems LLC CA)0 *H
101 0 UUS10UFlorida10U Niceville10U
Cuda Systems LLC1 0UCuda Systems LLC CA1"0 *H
Cuda Systems LLC CA)0
*H
8Tf$={l^O}|J7i eT iB@]l#+)xwx @N31I xzoa@f3GP=˼sXhd9F tgXl:BoT w MPY#P7KS \ ;:xKc}ֳ_X|CG
'7,4n (ӭp$?eD
!:ɬZS"r7[roXJWSrm{ iSJq32a CRIq,Fj#'`u,1F5_ ~ss ^l
W
0t{mvPsh S:+A ]M7>R D,N&ͩQs]Њ#DK+g蕸F;},+ᘛIX w/;e+YgKq~ȄhLOx[K!O{G#%>Nqꆿ:!P߽sKI&d d]fߊ
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?559D51B5.3050709>