From owner-freebsd-fs@freebsd.org  Tue Mar 23 00:27:04 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id 927155BF201
 for <freebsd-fs@mailman.nyi.freebsd.org>; Tue, 23 Mar 2021 00:27:04 +0000 (UTC)
 (envelope-from mchouque@thi.eu.com)
Received: from relay10.mail.gandi.net (relay10.mail.gandi.net [217.70.178.230])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 4F4ByH54n5z3Ff5
 for <freebsd-fs@freebsd.org>; Tue, 23 Mar 2021 00:27:02 +0000 (UTC)
 (envelope-from mchouque@thi.eu.com)
Received: from weirdfishes.localdomain (62-210-143-248.rev.poneytelecom.eu
 [62.210.143.248]) (Authenticated sender: m@thi.eu.com)
 by relay10.mail.gandi.net (Postfix) with ESMTPSA id 685F5240005;
 Tue, 23 Mar 2021 00:27:00 +0000 (UTC)
Received: by weirdfishes.localdomain (Postfix, from userid 1000)
 id 1AC327203B9D5; Tue, 23 Mar 2021 01:27:00 +0100 (CET)
Date: Tue, 23 Mar 2021 01:27:00 +0100
From: Mathieu Chouquet-Stringer <me+freebsd@mathieu.digital>
To: Martin Simmons <martin@lispworks.com>
Cc: freebsd-fs@freebsd.org
Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2)
Message-ID: <YFk11A/j7URClN/l@weirdfishes>
References: <YFhuxr0qRzchA7x8@weirdfishes>
 <202103221515.12MFFHRK015188@higson.cam.lispworks.com>
 <YFi6Lwh3ISn8UMvS@weirdfishes>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <YFi6Lwh3ISn8UMvS@weirdfishes>
X-Face: %JOeya=Dg!}[/#Go&*&cQ+)){p1c8}u\Fg2Q3&)kothIq|JnWoVzJtCFo~4X<uJ\9cHK'.w
 3:{EoxBR
X-Rspamd-Queue-Id: 4F4ByH54n5z3Ff5
X-Spamd-Bar: /
Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none;
 spf=none (mx1.freebsd.org: domain of mchouque@thi.eu.com has no SPF policy
 when checking 217.70.178.230) smtp.mailfrom=mchouque@thi.eu.com
X-Spamd-Result: default: False [0.57 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[];
 TO_DN_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2];
 FORGED_SENDER(0.30)[me@mathieu.digital,mchouque@thi.eu.com];
 RCVD_IN_DNSWL_LOW(-0.10)[217.70.178.230:from];
 MIME_TRACE(0.00)[0:+];
 RBL_DBL_DONT_QUERY_IPS(0.00)[217.70.178.230:from];
 ASN(0.00)[asn:29169, ipnet:217.70.176.0/20, country:FR];
 R_DKIM_NA(0.00)[]; TAGGED_FROM(0.00)[freebsd];
 RWL_MAILSPIKE_NEUTRAL(0.00)[217.70.178.230:from];
 ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000];
 FROM_NEQ_ENVFROM(0.00)[me@mathieu.digital,mchouque@thi.eu.com];
 FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.97)[0.969];
 NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain];
 DMARC_NA(0.00)[mathieu.digital]; AUTH_NA(1.00)[];
 SPAMHAUS_ZRD(0.00)[217.70.178.230:from:127.0.2.255];
 TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_NA(0.00)[no SPF record];
 MID_RHS_NOT_FQDN(0.50)[]; RCVD_COUNT_TWO(0.00)[2];
 RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs]
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 23 Mar 2021 00:27:04 -0000

On Mon, Mar 22, 2021 at 04:39:27PM +0100, Mathieu Chouquet-Stringer wrote:
> 	Hello Martin,
> 
> Thanks for taking the time to reply to me.
> 
> On Mon, Mar 22, 2021 at 03:15:17PM +0000, Martin Simmons wrote:
> > You could try:
> > 
> > procstat -kk 0
> > 
> > to get an overview of what the kernel threads are doing.

Exploring some more with top -SH, it seems that's my thread:
   31 root         -8    -     0B  3984K CPU7     7  38:14  99.92% zfskern{dsl_scan_iss_0}

That's the only busy thing on this OS and I know it's the one I see in
htop because I see it ping-ponging on different CPUs.

Seeing dsl_scan_iss led me to there:
https://github.com/openzfs/zfs/blob/64e0fe14fff465604952d576051ca80c2f1be898/module/zfs/dsl_scan.c

And that led me to reading about vfs.zfs.scan_legacy:
https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html?highlight=scan_mem_lim_fact#zfs-scan-legacy

I killed the current scrub, put vfs.zfs.scan_legacy=1 in sysctl.conf,
rebooted and restarted it, it's already telling me it'll finish quicker
than the one which was running and wich had already done 10%. It's been
running for 22 minutes now.

  scan: scrub in progress since Tue Mar 23 01:03:09 2021
        63.0G scanned at 87.5M/s, 63.0G issued at 87.5M/s, 24.5T total
        0B repaired, 0.25% done, 3 days 09:14:15 to go

I'll let it run overnight and I'll check back tomorrow: I mean you
can't trust the progress bar...

-- 
Mathieu Chouquet-Stringer
            The sun itself sees not till heaven clears.
	             -- William Shakespeare --