From owner-freebsd-current@freebsd.org Fri Mar 26 12:30:23 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A7E015BDCFD for ; Fri, 26 Mar 2021 12:30:23 +0000 (UTC) (envelope-from freebsd@grem.de) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4F6LsW2pRvz4cbq for ; Fri, 26 Mar 2021 12:30:23 +0000 (UTC) (envelope-from freebsd@grem.de) Received: by mailman.nyi.freebsd.org (Postfix) id 5E0615BDCFC; Fri, 26 Mar 2021 12:30:23 +0000 (UTC) Delivered-To: current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5DB3C5BDFD8; Fri, 26 Mar 2021 12:30:23 +0000 (UTC) (envelope-from freebsd@grem.de) Received: from mail.evolve.de (mail.evolve.de [213.239.217.29]) (using TLSv1.3 with cipher TLS_CHACHA20_POLY1305_SHA256 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.evolve.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F6LsT5lqTz4chd; Fri, 26 Mar 2021 12:30:21 +0000 (UTC) (envelope-from freebsd@grem.de) Received: by mail.evolve.de (OpenSMTPD) with ESMTP id b1a7a5c9; Fri, 26 Mar 2021 12:30:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=grem.de; h=date:from:to:cc :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=20180501; bh=kAm37C2Z dSf5xxuaEZyc71zmh18=; b=PP1N54l0qo2FcZKwQr3YGIi+i8ZASjezWu8Xdru2 oe6lYfeeiX41DxRLUZ9tul9iN8HIsHzlP2WYNF5XalRzZyvuCV5Nac+1r/WIh6yl Vdk643mu51KXX4TY5kmSNcnLnCgQD/M6W21nIBw6EoskoyWhU8nOxUg3yvE8Eev4 hHombsZ48t9BvDSR+hMs+itE0Gn39+oOS3xyRCiqgPqxk5ajl2X1vXY2+GVHJg3H w+S7GONkf4Qr+MJe3sDlFWhG3iiIFYyxZwK9VbU15fdrNI+nC8gDlzU6SkbPAWtL dJMGaTYsObe61N9xJAti9TrWL5ukWLkpWHEf7kSJxYoN3A== DomainKey-Signature: a=rsa-sha1; c=nofws; d=grem.de; h=date:from:to:cc :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=20180501; b=OO e8b6n30Zzzx3UpQm0CZKFhKsSYO7BCBcOW5xVxJHqmOR0a8Rv8DW99jYlPEpjVEA L13OKVP+HeKI/Aqr35aeAFNgsMtsaWwv43Q8yDAhR0cMCigObCG9WC4uY9o796FN yYlAiucF2nghs0loBhDEkrOobZZ8th3BG/AXf34UCQy8DdM7Z0D7RQPTgHPMDVhj 8uyVmGytxps9nIOD6EWuQEMl7En8+XragBPxM8QiI2LJozIbaXhCBgMao9CT2Shg KjBtYM0/9ReAvkohyqRMJfjeVvmmIW2PrjUa+2o8hOmc+P9EJW3j9f7vm3l9s4eP 1iVuGxHlfboZtwO/aFFQ== Received: by mail.evolve.de (OpenSMTPD) with ESMTPSA id c7d67f02 (TLSv1.3:AEAD-CHACHA20-POLY1305-SHA256:256:NO); Fri, 26 Mar 2021 12:30:09 +0000 (UTC) Date: Fri, 26 Mar 2021 13:29:45 +0100 From: Michael Gmelin To: Mathieu Chouquet-Stringer Cc: Matt Churchyard , "freebsd-fs@freebsd.org" , current@freebsd.org Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2) Message-ID: <20210326132945.3274687e@bsd64.grem.de> In-Reply-To: References: <202103221515.12MFFHRK015188@higson.cam.lispworks.com> <202103241230.12OCUqur030001@higson.cam.lispworks.com> <33eb78e2de404a77b271880dbee4c22e@SERVER.ad.usd-group.com> X-Face: $wrgCtfdVw_H9WAY?S&9+/F"!41z'L$uo*WzT8miX?kZ~W~Lr5W7v?j0Sde\mwB&/ypo^}> +a'4xMc^^KroE~+v^&^#[B">soBo1y6(TW6#UZiC]o>C6`ej+i Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWJBwe5BQDl LASZU0/LTEWEfHbyj0Txi32+sKrp1Mv944X8/fm1rS+cAAAACXBIWXMAAAsTAAAL EwEAmpwYAAAAB3RJTUUH3wESCxwC7OBhbgAAACFpVFh0Q29tbWVudAAAAAAAQ3Jl YXRlZCB3aXRoIFRoZSBHSU1QbbCXAAAAAghJREFUOMu11DFvEzEUAGCfEhBVFzuq AKkLd0O6VrIQsLXVSZXoWE5N1K3DobBBA9fQpRWc8OkWouaIjedWKiyREOKs+3PY fvalCNjgLVHeF7/3bMtBzV8C/VsQ8tecEgCcDgrzjekwKZ7TwsJZd/ywEKwwP+ZM 8P3drTsAwWn2mpWuDDuYiK1bFs6De0KUUFw0tWxm+D4AIhuuvZqtyWYeO7jQ4Aea 7jUqI+ixhQoHex4WshEvSXdood7stlv4oSuFOC4tqGcr0NjEqXgV4mMJO38nld4+ xKNxRDon7khyKVqY7YR4d+Cg0OMrkWXZOM7YDkEfKiilCn1qYv4mighZiynuHHOA Wq9QJq+BIES7lMFUtcikMnkDGHUoncA+uHgrP0ctIEqfwLHzeSo+eUA66AqzwN6n 2ZHJhw6Qh/PoyC/QENyEyC/AyNjq74Bs+3UH0xYwzDUC4B97HgLocg1QLYgDDO1v f3UX9Y307Ew4AHh67YAFFsxEpkXwpXY3eIgMhAAE3R19L919nNnuD2wlPcDE3UeT L2ytEICQib9BXgS2fU8PrD82ToYO1OEmMSnYTjSqSv9wdC0tPYC+rQRQD9ESnldF CyqfmiYW+tlALt8gH2xrMdC/youbjzPXEun+/ReXsMCDyve3dZc09fn2Oas8oXGc Jj6/fOeK5UmSMPmf/jL+GD8BEj0k/Fn6IO4AAAAASUVORK5CYII= MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4F6LsT5lqTz4chd X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=grem.de header.s=20180501 header.b=PP1N54l0; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd@grem.de designates 213.239.217.29 as permitted sender) smtp.mailfrom=freebsd@grem.de X-Spamd-Result: default: False [-3.50 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[grem.de:s=20180501]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:213.239.217.29/32]; TAGGED_RCPT(0.00)[freebsd]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grem.de]; SPAMHAUS_ZRD(0.00)[213.239.217.29:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[grem.de:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[213.239.217.29:from]; ASN(0.00)[asn:24940, ipnet:213.239.192.0/18, country:DE]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs,current] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Mar 2021 12:30:23 -0000 On Fri, 26 Mar 2021 10:37:47 +0100 Mathieu Chouquet-Stringer wrote: > On Thu, Mar 25, 2021 at 08:55:12AM +0000, Matt Churchyard wrote: > > Just an a aside, I did post a message a few weeks ago with a similar > > problem on 13 (as well as snapshot issues). Scrub seemed ok for a > > short while, but then ground to a halt. It would take 10+ minutes to > > go 0.01%, with everything appearing fairly idle. I finally gave up > > and stopped it after about 20 hours. Moving to 12.2 and rebuilding > > the pool, the system scrubbed the same data in an hour, and I've > > just scrubbed the same system after a month of use with about 4 > > times the data in 3 hours 20. As far as I'm aware, both should be > > using effectively the same "new" scrub code. > > > > Will be interesting if you find a cause as I didn't get any response > > to what for me was a complete showstopper for moving to 13. > > Bear with me, I'm slowly resilvering now... But same thing, it's not > even maxing out my slow drives... Looks like it'll take 2 days... > > I did some flame graphs using dtrace. The first one is just the output > of that: > dtrace -x stackframes=100 -n 'profile-99 /arg0/ { @[stack()] = > count(); } tick-60s { exit(0); }' > > Clearly my machine is not busy at all. > And the second is the output of pretty much the same thing except I'm > only capturing pid 31 which is the one busy. > dtrace -x stackframes=100 -n 'profile-99 /arg0 && pid == 31/ { > @[stack()] = count(); } tick-60s { exit(0); }' > > One striking thing is how many times hpet_get_timecount is present... Does tuning of - vfs.zfs.scrub_delay - vfs.zfs.resilver_min_time_ms - vfs.zfs.resilver_delay make a difference? Best, Michael -- Michael Gmelin