From owner-freebsd-current@freebsd.org Thu Dec 14 13:44:01 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1EAAAE8081C for ; Thu, 14 Dec 2017 13:44:01 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9396764532 for ; Thu, 14 Dec 2017 13:44:00 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from thor.intern.walstatt.dynvpn.de ([78.48.220.8]) by mail.gmx.com (mrgmx101 [212.227.17.168]) with ESMTPSA (Nemesis) id 0ME33j-1eFNBq19LD-00HNiU; Thu, 14 Dec 2017 14:43:57 +0100 Date: Thu, 14 Dec 2017 14:43:24 +0100 From: "O. Hartmann" To: Daniel Nebdal Cc: "O. Hartmann" , FreeBSD CURRENT Subject: Re: ZFS RAIDZ1: resilvering at <17.3M/s => abyssal slow ... Message-ID: <20171214144351.24a81faa@thor.intern.walstatt.dynvpn.de> In-Reply-To: References: <20171214124900.64211bd9@thor.intern.walstatt.dynvpn.de> Organization: WALSTATT User-Agent: OutScare 3.1415926 X-Operating-System: ImNotAnOperatingSystem 3.141592527 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; boundary="Sig_/a4HkXQOB3xxRloK4i/oe82M"; protocol="application/pgp-signature" X-Provags-ID: V03:K0:v9bZkWyrPl196DjFcNxQOJT8ssGobQIK3/hVTIEG5GvIMzsq55g CpAcY98288F1gUpNGiA8r+abUMeIMGOti0WdWRZLtThrtbqYHDgykg15JJv1QpGEsXY5n+s KhjyVdRsiYLUhmKXic+8vTIGVAp7FZFzMqFL6D3xMhM75eXbtZ2jlQpxyZdwp4JIQN5YkV7 RfJvK/QgIbhZoZ8OO70JA== X-UI-Out-Filterresults: notjunk:1;V01:K0:qEmG3QqU6/U=:BRQ7v7GbEk3xJSZQ6/j6dR xczTKkxDOaMzDei9zIA3bBviW+pzm7ESwRWusOaN8F1AK9l3U/iwCX+uEKo+ov0MpJe7X+7aN RYNslULshJY2odUSL8b8LkAW9UfQu+RzxH2WTRUW1spbIVjC3GC0/Tyr1MSi12+IUcRhARrO+ jaWhdZUT9EFHJwa8O3z7+0pLo9Pqhe1rAAHc2G86YUDME0fZdEYqiG0KuYvvojzeZm5IURpmG lJHfn0zosajvZTCCNDBr3TZ+qkVWU6PpCKRrTLfeDiquoqaElMV/iGj5g3GunvaTDN2LuI8Jy pnSdF+aya3FfKB773Gtpi2CQh2LbNvbZGKl5Fg7e8+TO0AMIsKWqS9Ewh7XyNwIf5Vhz9VUKl A0dZY5BTLT05PZBPFYmbCQh0dG2Pj7ZYyvzzmLShrIfXisGsD7hVpkmuOjzXFf4JLxI5leFDm ztxXSbENAYWy+UlbZ5fFQDfIY6uOZcIa2UuNuczH5WDrv5TOUYFcG7RqlI04Tqc8zSU2nVFy2 lPUwSwh2F0VHl20KLOT5rdPGqj8hLeMm2MmxQthd6nMhTN6N1OrM5vO3r5HeAvDPqZZCQYqfC wzbokq7SUu8YwKwJa4HkYh4DO9OAXAhgx8yovwhg9ykts9OTPEF0Kh3rbbTaNIdDBnLw/e1TJ cKB1qkYBT5d5+r1noIb9lchu/8T6ixJtFb/Nuaqfsj9adrBz/fRdZnrX93W83vEX7wFFzZYAb SRL54/Ioys8055CmJthvmDSwB4GryexMSZKMDTo1Ufr1WE4nOx1runC58nlV2chH6YCqA4uL/ DduSlx29/FQ/8fmJ10jC6XCzSo/wA== X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Dec 2017 13:44:01 -0000 --Sig_/a4HkXQOB3xxRloK4i/oe82M Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Am Thu, 14 Dec 2017 14:09:39 +0100 Daniel Nebdal schrieb: > On Thu, Dec 14, 2017 at 12:48 PM, O. Hartmann wr= ote: > > Hello out there, > > > > I had to replace a HDD on a RAIDZ1 pool comprised from 4 HDDs, each 3 T= B (netto size > > of the pool is about 8-9 TB, the used amount is reported to be ~ 6TB). > > > > I just started the rebuild/resilvering process and watch the pool crwal= ing at ~ 18 > > MB/s. At the moment, there is no load on the array, the host is a IvyBr= idge XEON with > > 4 core/8 threads and 3,4 GHz and 16 GB of RAM. The HDDs are attached to= a on-board > > SATA II (300 MB/s max) Intel chip - this just for the record. > > > > Recently, I switch on the "sync" attribute on most of the defined pools= 's zfs > > filesystems > > - I also use a SSD for ZIL/L2ARC caching, but it seems to be unused rec= ently in > > FreeBSD CURRENT's ZFS - this from a observers perspective only. > > > > When scrubbing, I see recently also reduced performance on the pool, so= I'm wondering > > about the low throughput at the very moment when resilvering is in prog= ress. > > > > If the "perspective" of "zpool status" is correct, then I have to wait = after two hours > > for another 100 hours - ~ 4 days? Ups ... I think there is something ba= dly > > misconfigured or missing. > > > > The pool is not "tuned" in any very sophisticated way, since I trust th= e kernels > > ability to scale automatically - if I'm not wrong for amd64 ... > > > > Are there any aspects to look at? > > > > Thanks in advance, > > > > Oliver > > > > -- > > O. Hartmann > > > > Ich widerspreche der Nutzung oder =C3=9Cbermittlung meiner Daten f=C3= =BCr > > Werbezwecke oder f=C3=BCr die Markt- oder Meinungsforschung (=C2=A7 28 = Abs. 4 BDSG). =20 >=20 >=20 > This is kind of to be expected - for whatever reason, resilvers seem > to go super slow at first and then speed up significantly. Just don't > ask me how long "at first" is - I'd give it several (more) hours. > I don't _think_ sync affects resilver speed (since that's a filesystem > level setting and the resilver happens at the block level), but that's > pure speculation. > The Tuning guide at https://wiki.freebsd.org/ZFSTuningGuide also > suggests a list of sysctls you can tweak, which could be worth a try: >=20 > ### wiki > If you're getting horrible performance during a scrub or resilver, the > following sysctls can be set: >=20 > vfs.zfs.scrub_delay=3D0 > vfs.zfs.top_maxinflight=3D128 > vfs.zfs.resilver_min_time_ms=3D5000 > vfs.zfs.resilver_delay=3D0 >=20 > Setting those sysctls to those values increased my (Shawn Webb's) > resilver performance from 7MB/s to 230MB/s. > ### >=20 >=20 >=20 I had already set those tuning parameters, shortly after I sent the mail to= the list, but there wasn't any sign of increasing performance since then. Now, after roughly 4 hours of running resilvering, it seems that the speed/= throughput is increasing: from initially 12 - 17 MB/s its now at ~ 90 MBytes/s - sounds m= ore promising. I remember that I whitnessed this also when I checked the last time scrubbi= ng: a non linear gradient in throughput while time shifts forward ... So, everything seems to be all right ... Kind regards, Oliver --=20 O. Hartmann Ich widerspreche der Nutzung oder =C3=9Cbermittlung meiner Daten f=C3=BCr Werbezwecke oder f=C3=BCr die Markt- oder Meinungsforschung (=C2=A7 28 Abs.= 4 BDSG). --Sig_/a4HkXQOB3xxRloK4i/oe82M Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iLUEARMKAB0WIQQZVZMzAtwC2T/86TrS528fyFhYlAUCWjKAFwAKCRDS528fyFhY lKbnAf9kPSkE5QQPzBcIJIRC+CCQbf4Sggo8Te4DkufQR4z70UqaWlDGYZ+Rg1QE Pe9Xdfw2BUSuzYcKf2t1Kn8HjDDtAf9UubfzRat+MI96OqkwT4fmuDxfa3DIxURW xry6hFJ6/cKmSsrw0T//PYv9HoWofvVc/OUe+DplawUpwSjbdusB =6mYM -----END PGP SIGNATURE----- --Sig_/a4HkXQOB3xxRloK4i/oe82M--