From nobody Sun Oct 16 16:07:48 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mr4m25p57z4fWyf for ; Sun, 16 Oct 2022 16:08:02 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-vs1-f49.google.com (mail-vs1-f49.google.com [209.85.217.49]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Mr4m200LRz3g64 for ; Sun, 16 Oct 2022 16:08:01 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-vs1-f49.google.com with SMTP id d187so9369461vsd.6 for ; Sun, 16 Oct 2022 09:08:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=mHH3NcZsw5Bsbv1bWM2pUwacaXrtHErfZN70atnR2H8=; b=NZCH3pLmE9A57U5LtPX/zHa+35048cV3W1W51rnBwMZ7gEGd7fM8PPc8NQ8LN/BQzb 4WlrepRHVqEJqhxJRYmg5LMixIxCyjhgE9aAigFXQ0Ae7M+tW0v/7Wd2Y2W86TubDZY+ 4SM1wsWAdgkBo2OWlMjCeCTo0ce3rNIEZcMKfrVr3thWEsGvVNDWiRJ9xGQpcizjoaJw 7tNl9zDVU14KHvS1/ShF6fpwiPQqyQ0+/dWF9+8b4avIW6Cqq6Zo2AQorlu3ZZ8iMivq z3T0eSMBseu5FGQfNH6MRgacG9tbQ3cccXzGSoIMOcD/jsoBfsKop0F6N0zSMIWXZt8i LNdQ== X-Gm-Message-State: ACrzQf19zlqIhVoLUJNZ2KEZcebVFYQbl/eQyfBaiHClHj+E+i/Slv3t 6+P2r8bNBTNCgwIJbAXqNSQqoifaSTXHInmYXH5v3xle9CE= X-Google-Smtp-Source: AMsMyM4bf0ipFNQM5CTOT2xYh4AzKgR6X7lzewgnlxsBWPm7FtvHM7+qqrNFiJh3cfyczT3IT/KaykxxPVcHyhvUjMs= X-Received: by 2002:a67:f699:0:b0:3a7:c31b:c8e2 with SMTP id n25-20020a67f699000000b003a7c31bc8e2mr2511094vso.11.1665936480574; Sun, 16 Oct 2022 09:08:00 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <1625dc7b-81c4-4350-8f86-1b65f5a860d9@app.fastmail.com> In-Reply-To: <1625dc7b-81c4-4350-8f86-1b65f5a860d9@app.fastmail.com> From: Alan Somers Date: Sun, 16 Oct 2022 10:07:48 -0600 Message-ID: Subject: Re: zfs with operations like rm -rf takes a very long time recently To: void Cc: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4Mr4m200LRz3g64 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.217.49 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; FREEMAIL_TO(0.00)[f-m.fm]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; R_DKIM_NA(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.217.49:from]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[asomers]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.217.49:from]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; TO_DN_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On Sun, Oct 16, 2022 at 9:13 AM void wrote: > > On Sun, 16 Oct 2022, at 14:15, Alan Somers wrote: > > The usual reason why rm gets slow is because your pool is nearly full > > and there's a snapshot. A snapshot means that rm doesn't actually > > free space; it just rewrites metadata, which requires even more space. > > And when a zpool is nearly full, writes always slow way down. > > But if that's not it, then you should also check gstat to see if the > > disk itself is slow. > > Hi, > > The disk in question is one of these: > https://skinflint.co.uk/toshiba-mobile-hdd-mq01-series-1tb-mq01abd100m-a1820027.html > > It's a CMR disk. Power-on hrs = 39253 > > da0: 400.000MB/s transfers > da0: 953869MB (1953525168 512 byte sectors) > da0: quirks=0x2 > > Filesystem Size Used Avail Capacity Mounted on > zroot/ROOT/default 863G 146G 717G 17% / > > there are no snapshots in "zfs list -t snapshot" > > rm -rf /var/cache/ccache/* has been running for 53 mins so far (it's still running) > > # gstat -dopC > timestamp,name,q-depth,total_ops/s,read/s,read-KiB/s,ms/read,write/s,write-KiB/s,ms/write,delete/s,delete-KiB/s,ms/delete,other/s,ms/other,%busy > 2022-10-16 16:01:24.761122175,da0,5,0,0,0,0.0,0,0,0.0,0,0,0.0,0,0.0,0.0 > 2022-10-16 16:01:25.762821956,da0,4,40,34,136,107.8,4,16,57.1,0,0,0.0,2,293.9,111.1 > 2022-10-16 16:01:26.763826447,da0,3,52,50,200,37.6,1,60,34.1,0,0,0.0,1,152.0,95.0 > 2022-10-16 16:01:27.767079004,da0,3,108,40,159,33.6,67,2173,10.3,0,0,0.0,1,12.2,85.1 > 2022-10-16 16:01:28.813081219,da0,1,57,17,69,62.8,38,467,31.0,0,0,0.0,2,156.9,108.7 > 2022-10-16 16:01:29.818093791,da0,1,56,56,223,17.2,0,0,0.0,0,0,0.0,0,0.0,95.9 > 2022-10-16 16:01:30.825841923,da0,1,57,57,230,16.9,0,0,0.0,0,0,0.0,0,0.0,95.3 > 2022-10-16 16:01:31.828940957,da0,1,58,58,231,16.7,0,0,0.0,0,0,0.0,0,0.0,96.7 > 2022-10-16 16:01:32.830822873,da0,3,150,29,116,33.9,120,8460,9.4,0,0,0.0,1,125.4,99.7 > 2022-10-16 16:01:33.877434519,da0,3,115,6,23,128.4,109,8607,17.1,0,0,0.0,0,0.0,93.4 > ^C > > I'm not sure what, of the data above, what would be considered "slow", or why it should be like this now. > > There's lots of things in sysctl -a concerning zfs. Can you suggest anything to > look out for? > > thanks, Gstat is showing that your disk is fully busy. It's also showing read latency as high as 128 ms, which is extremely slow. I suspect a problem with your disk. FYI, ZFS naturally has a 5-second rhythm (unless you changed vfs.zfs.txg.timeout), so gstat's output is sometimes more consistent if you use "-I 5s". You can also omit "-d" for magnetic HDDs, since they don't have anything like TRIM. I suggest checking dmesg to see if there are any messages about errors from da0. It would also be worth running "smartctl -a /dev/da0", from sysutils/smartmontools. -Alan