From owner-freebsd-current@freebsd.org Thu Dec 14 15:35:19 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C350FE8441B for ; Thu, 14 Dec 2017 15:35:19 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9A4AB68DB1 for ; Thu, 14 Dec 2017 15:35:19 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id B6C6A14AC1 for ; Thu, 14 Dec 2017 15:35:17 +0000 (UTC) Subject: Re: ZFS RAIDZ1: resilvering at <17.3M/s => abyssal slow ... To: freebsd-current@freebsd.org References: <20171214124900.64211bd9@thor.intern.walstatt.dynvpn.de> <20171214144351.24a81faa@thor.intern.walstatt.dynvpn.de> <20171214155254.4736ebe5@thor.intern.walstatt.dynvpn.de> From: Allan Jude Message-ID: Date: Thu, 14 Dec 2017 10:35:13 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <20171214155254.4736ebe5@thor.intern.walstatt.dynvpn.de> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="UW0wOunc2jpg5Ln7xrlefCBUM5FSFqNnf" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Dec 2017 15:35:19 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --UW0wOunc2jpg5Ln7xrlefCBUM5FSFqNnf Content-Type: multipart/mixed; boundary="ECtoMPAcH8o2t9S21tcRHMfvbn84vVfXx"; protected-headers="v1" From: Allan Jude To: freebsd-current@freebsd.org Message-ID: Subject: Re: ZFS RAIDZ1: resilvering at <17.3M/s => abyssal slow ... References: <20171214124900.64211bd9@thor.intern.walstatt.dynvpn.de> <20171214144351.24a81faa@thor.intern.walstatt.dynvpn.de> <20171214155254.4736ebe5@thor.intern.walstatt.dynvpn.de> In-Reply-To: <20171214155254.4736ebe5@thor.intern.walstatt.dynvpn.de> --ECtoMPAcH8o2t9S21tcRHMfvbn84vVfXx Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2017-12-14 09:52, O. Hartmann wrote: > Am Thu, 14 Dec 2017 15:46:17 +0100 > Dimitry Andric schrieb: >=20 >> On 14 Dec 2017, at 14:43, O. Hartmann wrote: >>> >>> Am Thu, 14 Dec 2017 14:09:39 +0100 >>> Daniel Nebdal schrieb: =20 >>>> On Thu, Dec 14, 2017 at 12:48 PM, O. Hartmann wrote: =20 >>>>> I just started the rebuild/resilvering process and watch the pool c= rwaling at ~ 18 >>>>> MB/s. At the moment, there is no load on the array, the host is a I= vyBridge XEON >>>>> with 4 core/8 threads and 3,4 GHz and 16 GB of RAM. The HDDs are at= tached to a >>>>> on-board SATA II (300 MB/s max) Intel chip - this just for the reco= rd. >>>>> >>>>> Recently, I switch on the "sync" attribute on most of the defined p= ools's zfs >>>>> filesystems >>>>> - I also use a SSD for ZIL/L2ARC caching, but it seems to be unused= recently in >>>>> FreeBSD CURRENT's ZFS - this from a observers perspective only. >>>>> >>>>> When scrubbing, I see recently also reduced performance on the pool= , so I'm >>>>> wondering about the low throughput at the very moment when resilver= ing is in >>>>> progress. >>>>> >>>>> If the "perspective" of "zpool status" is correct, then I have to w= ait after two >>>>> hours for another 100 hours - ~ 4 days? Ups ... I think there is so= mething badly >>>>> misconfigured or missing. =20 >> ... >>>> This is kind of to be expected - for whatever reason, resilvers seem= >>>> to go super slow at first and then speed up significantly. Just don'= t >>>> ask me how long "at first" is - I'd give it several (more) hours. =20 >> >> Hopefully this will get better in the future, please read: >> >> http://open-zfs.org/wiki/Scrub/Resilver_Performance >> >> -Dimitry >> >=20 > It has already been started to become better ;-) >=20 > After a while now, the throughput is at 128 MBytes/s and the estimated = time decreased to > ~ 8 h now - that is much more appreciable than 4 days ;-) >=20 > Kind regards, > Oliver >=20 >=20 The time estimate is a pure average over the entire length of the scrub or resilver operation. The very start of the operation is quite slow, because it involves a lot of random seeks, and the read-ahead is not very smart (patches are in progress). And yes, after you adjust the resilver_delay, it will take time for its impact to be visible in 'zpool status'. At times, you are better off looking at 'gstat', to see how busy the disks actually are. You will likely notice the bottleneck is IOPS, you will be at the limit of at least one the entire duration of the resilver, unless your resilver_delay is high enough to leave some available IOPS. --=20 Allan Jude --ECtoMPAcH8o2t9S21tcRHMfvbn84vVfXx-- --UW0wOunc2jpg5Ln7xrlefCBUM5FSFqNnf Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJaMpo1AAoJEBmVNT4SmAt+pC0QAIiNDb3StSEJmZKi7e3bghQ1 BJfJCq/viywF5jQ4rggY4gvVEQhWPkejAff/yFxJRc6Nc1wQ35Rc/tosQW96QuSK Cg/8hOKsSaTmB+0l2dq5dnp77kQ4L54lepxaawvIaVFErJRjhYs7ZgVrNG0i3wYk pTjFGrdC+Wpg+X20yI555I+17ep9TRTY0HZEjUnGjZTElf1hrXfy8HiYnax51qk/ 6WFLa9AIK+ZhfRRKEi1gf2sS7k0U0WZiJJHf0ObiEt1jWybGk556YneD+w1SuGI5 kAQ33pL7+zYKn74DO9DiBZ9eUnfCqSuNJIj00ITneY+Aek6e/nJNAimzoVcIFW6j KCQWzZ8T33BQwSdCAdeJetPkDFWL/CHtz3phRMjls3/30o66xr5nNNLEuz3Sl7Rn 98ssr6vtvD2mKbFT/MjLd/Zlz6znM2Of6SMVsw+wLeULpjZmr4APRZFq61o8tTdF lbk67PJE5HO4mtldz/5plG+80vASJSjSu3Tidsgl3l4iPTebpYKQiDVks8T0BY+1 rbHPvm2JRH0mSrxLQlBLI+V66m9npWyBZkrV/YyzcXymZHLEp3b9MTCKesH+jVv/ uulfc84CzDajwaCn0ibLr6QrnA49me6Ri06PXh8rDGoECxGwDC4q1303k3XvIEvV ei3O3CinMTTUrOp/RESi =4yon -----END PGP SIGNATURE----- --UW0wOunc2jpg5Ln7xrlefCBUM5FSFqNnf--