Date: Thu, 14 Dec 2017 10:35:13 -0500 From: Allan Jude <allanjude@freebsd.org> To: freebsd-current@freebsd.org Subject: Re: ZFS RAIDZ1: resilvering at <17.3M/s => abyssal slow ... Message-ID: <b0eb09fc-d012-2b00-1c93-6840ddf8056f@freebsd.org> In-Reply-To: <20171214155254.4736ebe5@thor.intern.walstatt.dynvpn.de> References: <20171214124900.64211bd9@thor.intern.walstatt.dynvpn.de> <CA%2Bt49PJuznLxGnLERwAhVW2CjETJYWO6rKcWo3qOo5bZLQkqYA@mail.gmail.com> <20171214144351.24a81faa@thor.intern.walstatt.dynvpn.de> <B02D080F-96B1-4456-8D19-21F89821E2C0@FreeBSD.org> <20171214155254.4736ebe5@thor.intern.walstatt.dynvpn.de>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --UW0wOunc2jpg5Ln7xrlefCBUM5FSFqNnf Content-Type: multipart/mixed; boundary="ECtoMPAcH8o2t9S21tcRHMfvbn84vVfXx"; protected-headers="v1" From: Allan Jude <allanjude@freebsd.org> To: freebsd-current@freebsd.org Message-ID: <b0eb09fc-d012-2b00-1c93-6840ddf8056f@freebsd.org> Subject: Re: ZFS RAIDZ1: resilvering at <17.3M/s => abyssal slow ... References: <20171214124900.64211bd9@thor.intern.walstatt.dynvpn.de> <CA+t49PJuznLxGnLERwAhVW2CjETJYWO6rKcWo3qOo5bZLQkqYA@mail.gmail.com> <20171214144351.24a81faa@thor.intern.walstatt.dynvpn.de> <B02D080F-96B1-4456-8D19-21F89821E2C0@FreeBSD.org> <20171214155254.4736ebe5@thor.intern.walstatt.dynvpn.de> In-Reply-To: <20171214155254.4736ebe5@thor.intern.walstatt.dynvpn.de> --ECtoMPAcH8o2t9S21tcRHMfvbn84vVfXx Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2017-12-14 09:52, O. Hartmann wrote: > Am Thu, 14 Dec 2017 15:46:17 +0100 > Dimitry Andric <dim@FreeBSD.org> schrieb: >=20 >> On 14 Dec 2017, at 14:43, O. Hartmann <ohartmann@walstatt.org> wrote: >>> >>> Am Thu, 14 Dec 2017 14:09:39 +0100 >>> Daniel Nebdal <dnebdal@gmail.com> schrieb: =20 >>>> On Thu, Dec 14, 2017 at 12:48 PM, O. Hartmann <ohartmann@walstatt.or= g> wrote: =20 >>>>> I just started the rebuild/resilvering process and watch the pool c= rwaling at ~ 18 >>>>> MB/s. At the moment, there is no load on the array, the host is a I= vyBridge XEON >>>>> with 4 core/8 threads and 3,4 GHz and 16 GB of RAM. The HDDs are at= tached to a >>>>> on-board SATA II (300 MB/s max) Intel chip - this just for the reco= rd. >>>>> >>>>> Recently, I switch on the "sync" attribute on most of the defined p= ools's zfs >>>>> filesystems >>>>> - I also use a SSD for ZIL/L2ARC caching, but it seems to be unused= recently in >>>>> FreeBSD CURRENT's ZFS - this from a observers perspective only. >>>>> >>>>> When scrubbing, I see recently also reduced performance on the pool= , so I'm >>>>> wondering about the low throughput at the very moment when resilver= ing is in >>>>> progress. >>>>> >>>>> If the "perspective" of "zpool status" is correct, then I have to w= ait after two >>>>> hours for another 100 hours - ~ 4 days? Ups ... I think there is so= mething badly >>>>> misconfigured or missing. =20 >> ... >>>> This is kind of to be expected - for whatever reason, resilvers seem= >>>> to go super slow at first and then speed up significantly. Just don'= t >>>> ask me how long "at first" is - I'd give it several (more) hours. =20 >> >> Hopefully this will get better in the future, please read: >> >> http://open-zfs.org/wiki/Scrub/Resilver_Performance >> >> -Dimitry >> >=20 > It has already been started to become better ;-) >=20 > After a while now, the throughput is at 128 MBytes/s and the estimated = time decreased to > ~ 8 h now - that is much more appreciable than 4 days ;-) >=20 > Kind regards, > Oliver >=20 >=20 The time estimate is a pure average over the entire length of the scrub or resilver operation. The very start of the operation is quite slow, because it involves a lot of random seeks, and the read-ahead is not very smart (patches are in progress). And yes, after you adjust the resilver_delay, it will take time for its impact to be visible in 'zpool status'. At times, you are better off looking at 'gstat', to see how busy the disks actually are. You will likely notice the bottleneck is IOPS, you will be at the limit of at least one the entire duration of the resilver, unless your resilver_delay is high enough to leave some available IOPS. --=20 Allan Jude --ECtoMPAcH8o2t9S21tcRHMfvbn84vVfXx-- --UW0wOunc2jpg5Ln7xrlefCBUM5FSFqNnf Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJaMpo1AAoJEBmVNT4SmAt+pC0QAIiNDb3StSEJmZKi7e3bghQ1 BJfJCq/viywF5jQ4rggY4gvVEQhWPkejAff/yFxJRc6Nc1wQ35Rc/tosQW96QuSK Cg/8hOKsSaTmB+0l2dq5dnp77kQ4L54lepxaawvIaVFErJRjhYs7ZgVrNG0i3wYk pTjFGrdC+Wpg+X20yI555I+17ep9TRTY0HZEjUnGjZTElf1hrXfy8HiYnax51qk/ 6WFLa9AIK+ZhfRRKEi1gf2sS7k0U0WZiJJHf0ObiEt1jWybGk556YneD+w1SuGI5 kAQ33pL7+zYKn74DO9DiBZ9eUnfCqSuNJIj00ITneY+Aek6e/nJNAimzoVcIFW6j KCQWzZ8T33BQwSdCAdeJetPkDFWL/CHtz3phRMjls3/30o66xr5nNNLEuz3Sl7Rn 98ssr6vtvD2mKbFT/MjLd/Zlz6znM2Of6SMVsw+wLeULpjZmr4APRZFq61o8tTdF lbk67PJE5HO4mtldz/5plG+80vASJSjSu3Tidsgl3l4iPTebpYKQiDVks8T0BY+1 rbHPvm2JRH0mSrxLQlBLI+V66m9npWyBZkrV/YyzcXymZHLEp3b9MTCKesH+jVv/ uulfc84CzDajwaCn0ibLr6QrnA49me6Ri06PXh8rDGoECxGwDC4q1303k3XvIEvV ei3O3CinMTTUrOp/RESi =4yon -----END PGP SIGNATURE----- --UW0wOunc2jpg5Ln7xrlefCBUM5FSFqNnf--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?b0eb09fc-d012-2b00-1c93-6840ddf8056f>