From owner-freebsd-stable@freebsd.org  Mon Jan 23 15:37:33 2017
Return-Path: <owner-freebsd-stable@freebsd.org>
Delivered-To: freebsd-stable@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 212F6CBEA94
 for <freebsd-stable@mailman.ysv.freebsd.org>;
 Mon, 23 Jan 2017 15:37:33 +0000 (UTC)
 (envelope-from petefrench@ingresso.co.uk)
Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3])
 by mx1.freebsd.org (Postfix) with ESMTP id 0EB06831
 for <freebsd-stable@freebsd.org>; Mon, 23 Jan 2017 15:37:33 +0000 (UTC)
 (envelope-from petefrench@ingresso.co.uk)
Received: by mailman.ysv.freebsd.org (Postfix)
 id 0E156CBEA93; Mon, 23 Jan 2017 15:37:33 +0000 (UTC)
Delivered-To: stable@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DCA1CBEA92
 for <stable@mailman.ysv.freebsd.org>; Mon, 23 Jan 2017 15:37:33 +0000 (UTC)
 (envelope-from petefrench@ingresso.co.uk)
Received: from constantine.ingresso.co.uk
 (ingresso-1-pt.tunnel.tserv1.lon2.ipv6.he.net [IPv6:2001:470:1f1c:411::2])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id D3710830
 for <stable@freebsd.org>; Mon, 23 Jan 2017 15:37:32 +0000 (UTC)
 (envelope-from petefrench@ingresso.co.uk)
Received: from dilbert.london-internal.ingresso.co.uk ([10.64.50.6]
 helo=dilbert.ingresso.co.uk)
 by constantine.ingresso.co.uk with esmtps
 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.87 (FreeBSD))
 (envelope-from <petefrench@ingresso.co.uk>)
 id 1cVggV-0003Iw-Hv; Mon, 23 Jan 2017 15:37:31 +0000
Received: from petefrench by dilbert.ingresso.co.uk with local (Exim 4.87_1
 (FreeBSD)) (envelope-from <petefrench@ingresso.co.uk>)
 id 1cVggV-0004e0-GK; Mon, 23 Jan 2017 15:37:31 +0000
To: petefrench@ingresso.co.uk, stable@freebsd.org
Subject: Re: Horrible zpool resilver perormance on local hast
In-Reply-To: <E1cVgJH-0004aA-Kk@dilbert.ingresso.co.uk>
Message-Id: <E1cVggV-0004e0-GK@dilbert.ingresso.co.uk>
From: Pete French <petefrench@ingresso.co.uk>
Date: Mon, 23 Jan 2017 15:37:31 +0000
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-stable>, 
 <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Jan 2017 15:37:33 -0000

Partially answering my own question here, as it occured to
me that the zpool is scattering writes across the disc in 4k blocks,
but hast has a minimum extent size of 2meg by default. Thats eems like
a likely culprit for the 'dirty blocks' multiplcation I am seeing.

I shrunk down the extent size to 16k to see if that helped, and I
now see a far smaller dirty block to written block ration, but
also the write speed has now shrunk down even more to around 50k/s :-(

Any advice on tuning this properly ?

-pete.