From owner-freebsd-fs@freebsd.org  Fri Mar 26 09:37:52 2021
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id B2B245B932E;
 Fri, 26 Mar 2021 09:37:52 +0000 (UTC)
 (envelope-from mchouque@thi.eu.com)
Received: from relay10.mail.gandi.net (relay10.mail.gandi.net [217.70.178.230])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 4F6H2R3yfyz4S1C;
 Fri, 26 Mar 2021 09:37:50 +0000 (UTC)
 (envelope-from mchouque@thi.eu.com)
Received: from weirdfishes.localdomain (62-210-143-248.rev.poneytelecom.eu
 [62.210.143.248]) (Authenticated sender: m@thi.eu.com)
 by relay10.mail.gandi.net (Postfix) with ESMTPSA id 2B85B240016;
 Fri, 26 Mar 2021 09:37:47 +0000 (UTC)
Received: by weirdfishes.localdomain (Postfix, from userid 1000)
 id D14847203B9D5; Fri, 26 Mar 2021 10:37:47 +0100 (CET)
Date: Fri, 26 Mar 2021 10:37:47 +0100
From: Mathieu Chouquet-Stringer <me+freebsd@mathieu.digital>
To: Matt Churchyard <matt.churchyard@userve.net>
Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>, current@freebsd.org
Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2)
Message-ID: <YF2raxOUeN8Y23eT@weirdfishes>
References: <YFhuxr0qRzchA7x8@weirdfishes>
 <202103221515.12MFFHRK015188@higson.cam.lispworks.com>
 <YFi6Lwh3ISn8UMvS@weirdfishes> <YFk11A/j7URClN/l@weirdfishes>
 <YFm3BTK/J9XY/mCN@weirdfishes>
 <202103241230.12OCUqur030001@higson.cam.lispworks.com>
 <YFs3jFT7sEaGeQCe@weirdfishes>
 <33eb78e2de404a77b271880dbee4c22e@SERVER.ad.usd-group.com>
MIME-Version: 1.0
In-Reply-To: <33eb78e2de404a77b271880dbee4c22e@SERVER.ad.usd-group.com>
X-Face: %JOeya=Dg!}[/#Go&*&cQ+)){p1c8}u\Fg2Q3&)kothIq|JnWoVzJtCFo~4X<uJ\9cHK'.w
 3:{EoxBR
X-Rspamd-Queue-Id: 4F6H2R3yfyz4S1C
X-Spamd-Bar: -
Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none;
 spf=none (mx1.freebsd.org: domain of mchouque@thi.eu.com has no SPF policy
 when checking 217.70.178.230) smtp.mailfrom=mchouque@thi.eu.com
X-Spamd-Result: default: False [-1.40 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[];
 RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[];
 RWL_MAILSPIKE_GOOD(0.00)[217.70.178.230:from];
 HAS_ATTACHMENT(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000];
 FORGED_SENDER(0.30)[me@mathieu.digital,mchouque@thi.eu.com];
 RCVD_IN_DNSWL_LOW(-0.10)[217.70.178.230:from];
 MIME_TRACE(0.00)[0:+,1:+,2:+,3:+];
 RBL_DBL_DONT_QUERY_IPS(0.00)[217.70.178.230:from];
 ASN(0.00)[asn:29169, ipnet:217.70.176.0/20, country:FR];
 FROM_NEQ_ENVFROM(0.00)[me@mathieu.digital,mchouque@thi.eu.com];
 TAGGED_FROM(0.00)[freebsd]; R_DKIM_NA(0.00)[]; ARC_NA(0.00)[];
 NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[];
 RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000];
 MIME_GOOD(-0.10)[multipart/mixed,text/plain,image/svg+xml];
 DMARC_NA(0.00)[mathieu.digital]; AUTH_NA(1.00)[];
 SPAMHAUS_ZRD(0.00)[217.70.178.230:from:127.0.2.255];
 TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_NA(0.00)[no SPF record];
 MID_RHS_NOT_FQDN(0.50)[]; RCVD_COUNT_TWO(0.00)[2];
 RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[current,freebsd-fs]
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
X-Content-Filtered-By: Mailman/MimeDel 2.1.34
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 26 Mar 2021 09:37:52 -0000

On Thu, Mar 25, 2021 at 08:55:12AM +0000, Matt Churchyard wrote:
> Just an a aside, I did post a message a few weeks ago with a similar
> problem on 13 (as well as snapshot issues). Scrub seemed ok for a
> short while, but then ground to a halt. It would take 10+ minutes to
> go 0.01%, with everything appearing fairly idle. I finally gave up and
> stopped it after about 20 hours. Moving to 12.2 and rebuilding the
> pool, the system scrubbed the same data in an hour, and I've just
> scrubbed the same system after a month of use with about 4 times the
> data in 3 hours 20. As far as I'm aware, both should be using
> effectively the same "new" scrub code.
>
> Will be interesting if you find a cause as I didn't get any response
> to what for me was a complete showstopper for moving to 13.

Bear with me, I'm slowly resilvering now... But same thing, it's not
even maxing out my slow drives... Looks like it'll take 2 days...

I did some flame graphs using dtrace. The first one is just the output
of that:
dtrace -x stackframes=100 -n 'profile-99 /arg0/ { @[stack()] = count(); } tick-60s { exit(0); }'

Clearly my machine is not busy at all.
And the second is the output of pretty much the same thing except I'm
only capturing pid 31 which is the one busy.
dtrace -x stackframes=100 -n 'profile-99 /arg0 && pid == 31/ { @[stack()] = count(); } tick-60s { exit(0); }'

One striking thing is how many times hpet_get_timecount is present...
-- 
Mathieu Chouquet-Stringer                           mchouque@free.fr
            The sun itself sees not till heaven clears.
	             -- William Shakespeare --