From owner-freebsd-fs@freebsd.org Wed Jul 8 16:20:51 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72DA1996912 for ; Wed, 8 Jul 2015 16:20:51 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (smtp.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.infracaninophile.co.uk", Issuer "infracaninophile.co.uk" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 19EE914F4 for ; Wed, 8 Jul 2015 16:20:50 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from ox-dell39.ox.adestra.com (no-reverse-dns.metronet-uk.com [85.199.232.226] (may be forged)) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.15.1/8.15.1) with ESMTPSA id t68GKbtb065258 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 8 Jul 2015 17:20:39 +0100 (BST) (envelope-from m.seaman@infracaninophile.co.uk) Authentication-Results: smtp.infracaninophile.co.uk; dmarc=none header.from=infracaninophile.co.uk DKIM-Filter: OpenDKIM Filter v2.9.2 smtp.infracaninophile.co.uk t68GKbtb065258 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=infracaninophile.co.uk; s=201001-infracaninophile; t=1436372440; bh=+Ikz/eR9WdypCqz9yso8QSSQFMnPU4AbXmBUDhpA/lw=; h=Date:From:To:Subject:References:In-Reply-To; z=Date:=20Wed,=2008=20Jul=202015=2017:20:35=20+0100|From:=20Matthew =20Seaman=20|To:=20freebsd-fs@fre ebsd.org|Subject:=20Re:=20Speeding=20up=20resilvering|References:= 20<559D2AB6.5070007@FreeBSD.org>=20<559D2C7C.6060201@denninger.net >=20<559D3167.1000705@infracaninophile.co.uk>=20<559D3236.1060102@ denninger.net>=20<559D3380.8050703@sentex.net>|In-Reply-To:=20<559 D3380.8050703@sentex.net>; b=GjhnYvSX+WZqyB3xKJBifrZQBDOUIndavDzj1Te6M3R5ga1ozwUFT3t86SPBVl9DX kqcBkmcuJcCfLK1fJ8170ZnYVNkdvyDUmhWmPxME072bTCvitbT4hmY+R2THGDnLvb HG76zJ7JAbyDSP6PpImLjtvu2QVBaSrpAyn31xaY= X-Authentication-Warning: lucid-nonsense.infracaninophile.co.uk: Host no-reverse-dns.metronet-uk.com [85.199.232.226] (may be forged) claimed to be ox-dell39.ox.adestra.com Message-ID: <559D4DD3.10903@infracaninophile.co.uk> Date: Wed, 08 Jul 2015 17:20:35 +0100 From: Matthew Seaman User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: Speeding up resilvering References: <559D2AB6.5070007@FreeBSD.org> <559D2C7C.6060201@denninger.net> <559D3167.1000705@infracaninophile.co.uk> <559D3236.1060102@denninger.net> <559D3380.8050703@sentex.net> In-Reply-To: <559D3380.8050703@sentex.net> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="060uth7oEQeKXehX28voNVE36nwF4EXRs" X-Virus-Scanned: clamav-milter 0.98.7 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-1.5 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on lucid-nonsense.infracaninophile.co.uk X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2015 16:20:51 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --060uth7oEQeKXehX28voNVE36nwF4EXRs Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 07/08/15 15:28, Mike Tancsa wrote: >> Look at the IO saturation on the disk channel(s) involved with either >> > systat -vm or iostat. If the channel is saturated then there's noth= ing >> > you can do in terms of tuning; the question then turns to why actual= I/O >> > performance is so poor and has to be addressed there. > I had one server that was taking ages, and it turned out to be the > controller, not the disk that was hosed. As Karl suggested, take a look= > at the throughput on gstat. Is anyone disk or groups of disks lagging > far behind on write speeds ? In my case, it was a very obvious and > glaring outlier. Thanks for the suggestions. I've been playing with gstat et al, and as far as I can tell, all the drives are behaving reasonably well. I'm certainly getting 90-100% capacity (mostly reads) continually on the original drives whilst the new one (mostly writes) seems to go in bursts. Which is pretty much what I'd expect when resilvering a RAIDZ. So, having exhausted that, I actually sat down and timed what progress it was making rather more carefully. Turns out my impression that applying the sysctl tweaks I mentioned previously had little effect was wrong. Current projection is about 50h total to do the resilvering, which is much, much better than the approx 12days I was expecting previously. In fact, that's pretty much inline with what I'd expect from this hardware. Cheers, Matthew --060uth7oEQeKXehX28voNVE36nwF4EXRs Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVnU3TAAoJEABRPxDgqeTnv8AP/jYepeX0oqLGnpr8kCEbyJOM ZYyKNMqo8li86H7nyMC4gpOBd4nbrfZt7mr7phqujXTTT9oVRFL4aD6biFxmQWMJ oFmNr2btAJWKn8XneYUyJDFEPRb/4hBaLTshuDTLXygUCLf5lkjhMEcASU1EwJQf BVpAWf3T/nPT5ZhaWsL8H/mp+dgGb2Zn72SBjeXjBE7hQe02Ay4q/s9nZ9emhi1z WG8f0khUxHu45DP3coHnQpssTYTiNeT5X1S1h6dHXxDoY+3NrYFCE4MmP9/YWw7I pPtSqiXa0Val+0Tt0yVWuvswjrZxTPvrgBs+7zbclhmypfK1sK06d8rITh1xGkv8 7YDRl3o1g8a+Ef7TwFN7uhShREeJMbTlGRbi4L3IgYs/FGyla/qM2D2EhVy/X4wH DI+YI+HKlTl7peI8P69mFjMAvUQdHkdYLaumzZoJDfiuHXeCLqo81GdlDkXXsYA+ Rq0XIfkeAq7WOUmv7v/vXwooeHEIEVrO+dQJy6ytv9Z3Xr3YNs464sEq8lnjWehn ZUgfyJkgwJVXELoR3xMgZJoXNVEvHYeorIrPQqRW+mGb/gRcnPYN8Wh08QkG4z0u mwYbr5sd0fF+BqXkVIEH/6xrogW2Bo/0zyu55woiW2kRFFfYk2no0as2/5wHHIBA fowqAS61LzIy7MgpFd74 =72FI -----END PGP SIGNATURE----- --060uth7oEQeKXehX28voNVE36nwF4EXRs--