From owner-freebsd-stable@freebsd.org  Mon Jan 23 15:13:33 2017
Return-Path: <owner-freebsd-stable@freebsd.org>
Delivered-To: freebsd-stable@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77A62CBE48C
 for <freebsd-stable@mailman.ysv.freebsd.org>;
 Mon, 23 Jan 2017 15:13:33 +0000 (UTC)
 (envelope-from petefrench@ingresso.co.uk)
Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org
 [IPv6:2001:1900:2254:206a::50:5])
 by mx1.freebsd.org (Postfix) with ESMTP id 65279149C
 for <freebsd-stable@freebsd.org>; Mon, 23 Jan 2017 15:13:33 +0000 (UTC)
 (envelope-from petefrench@ingresso.co.uk)
Received: by mailman.ysv.freebsd.org (Postfix)
 id 61940CBE48B; Mon, 23 Jan 2017 15:13:33 +0000 (UTC)
Delivered-To: stable@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 613CACBE48A
 for <stable@mailman.ysv.freebsd.org>; Mon, 23 Jan 2017 15:13:33 +0000 (UTC)
 (envelope-from petefrench@ingresso.co.uk)
Received: from constantine.ingresso.co.uk
 (ingresso-1-pt.tunnel.tserv1.lon2.ipv6.he.net [IPv6:2001:470:1f1c:411::2])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 32D1B149B
 for <stable@freebsd.org>; Mon, 23 Jan 2017 15:13:33 +0000 (UTC)
 (envelope-from petefrench@ingresso.co.uk)
Received: from dilbert.london-internal.ingresso.co.uk ([10.64.50.6]
 helo=dilbert.ingresso.co.uk)
 by constantine.ingresso.co.uk with esmtps
 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.87 (FreeBSD))
 (envelope-from <petefrench@ingresso.co.uk>) id 1cVgJH-0002y5-MP
 for stable@freebsd.org; Mon, 23 Jan 2017 15:13:31 +0000
Received: from petefrench by dilbert.ingresso.co.uk with local (Exim 4.87_1
 (FreeBSD)) (envelope-from <petefrench@ingresso.co.uk>)
 id 1cVgJH-0004aA-Kk
 for stable@freebsd.org; Mon, 23 Jan 2017 15:13:31 +0000
To: stable@freebsd.org
Subject: Horrible zpool resilver perormance on local hast
Message-Id: <E1cVgJH-0004aA-Kk@dilbert.ingresso.co.uk>
From: Pete French <petefrench@ingresso.co.uk>
Date: Mon, 23 Jan 2017 15:13:31 +0000
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-stable>, 
 <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Jan 2017 15:13:33 -0000

I have a pair of machines running hast cross two pairs of
drives - i.e. 4 drives total, two in each box, and hence
two hast resources, cbert0 and cebrt1.

Its been running mysql fir a long time, but there only 6 gig
of data actually in use in the pool. I rebuilt the secondary
machine, and wanted to resync the hast devices, but that would involve
copying 450 gig for only 6 gig of actual data.

"easy" I though, "just rebuild each has device in the pool
so it only has the current data dirty"

seems reasonbale enough ?

so I disconnected one bit of the mirror, did a 'hast create' to
zero it, and reattached it to the mirror.

(note that the secondary side is entirely disconnected, this is local
to one machine).

what I am seeing is that the zpool is resilvering at a rate
of 170K/s - currently it says this:

        622M scanned out of 6.53G at 170K/s, 10h9m to go

But the hastctl status outout says this:

  status: degraded
  workerpid: 950
  dirty: 136363114496 (127GB)
  statistics:

I onyl expect dirty blocks when the ZFS has ritten something -0 127 gig
dirty seems excessive for 622 meg written!

anyone got any idea what might be happening ?

-pete.