From eugen@grosbein.net Wed Nov 22 07:04:39 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SZsh55htLz520qb for ; Wed, 22 Nov 2023 07:05:09 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SZsh369czz4XRf for ; Wed, 22 Nov 2023 07:05:07 +0000 (UTC) (envelope-from eugen@grosbein.net) Authentication-Results: mx1.freebsd.org; none Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.17.1/8.17.1) with ESMTPS id 3AM74nCp075934 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Wed, 22 Nov 2023 07:04:49 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: jonc@chen.org.nz Received: from [10.58.0.11] (dadvw [10.58.0.11] (may be forged)) by eg.sd.rdtc.ru (8.17.1/8.17.1) with ESMTPS id 3AM74lfS095191 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Wed, 22 Nov 2023 14:04:48 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Unusual ZFS behaviour To: Jonathan Chen , freebsd-stable@freebsd.org References: From: Eugene Grosbein Message-ID: Date: Wed, 22 Nov 2023 14:04:39 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,SHORTCIRCUIT autolearn=disabled version=3.4.6 X-Spam-Report: * -0.0 SHORTCIRCUIT No description available. * -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on hz.grosbein.net X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/32, country:DE] X-Rspamd-Queue-Id: 4SZsh369czz4XRf 22.11.2023 13:49, Jonathan Chen wrote: > Hi, > > I'm running a somewhat recent version of STABLE-13/amd64: stable/13-n256681-0b7939d725ba: Fri Nov 10 08:48:36 NZDT 2023, and I'm seeing some unusual behaviour with ZFS. > > To reproduce: > 1. one big empty disk, GPT scheme, 1 freebsd-zfs partition. > 2. create a zpool, eg: tank > 3. create 2 sub-filesystems, eg: tank/one, tank/two > 4. fill each sub-filesystem with large files until the pool is ~80% full. In my case I had 200 10Gb files in each. > 5. in one session run 'md5 tank/one/*' > 6. in another session run 'md5 tank/two/*' > > For most of my runs, one of the sessions against a sub-filesystem will be starved of I/O, while the other one is performant. > > Is anyone else seeing this? Please try repeating the test with atime updates disabled: zfs set atime=off tank/one zfs set atime=off tank/two Does it make any difference? Does it make any difference, if you import the pool with readonly=on instead? Writing to ~80% pool is almost always slow for ZFS.