From nobody Wed May  1 19:38:24 2024
X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VV6n42cZ0z5JwgR
	for <freebsd-stable@mlmmj.nyi.freebsd.org>; Wed,  1 May 2024 19:38:32 +0000 (UTC)
	(envelope-from mike@sentex.net)
Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VV6n30Dszz4fNb
	for <freebsd-stable@freebsd.org>; Wed,  1 May 2024 19:38:30 +0000 (UTC)
	(envelope-from mike@sentex.net)
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net
Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19])
	by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 441JcPgu079926
	(version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL)
	for <freebsd-stable@freebsd.org>; Wed, 1 May 2024 15:38:25 -0400 (EDT)
	(envelope-from mike@sentex.net)
Received: from [IPV6:2607:f3e0:0:4:bd4c:f42f:eabc:4f29] ([IPv6:2607:f3e0:0:4:bd4c:f42f:eabc:4f29])
	by pyroxene2a.sentex.ca (8.18.1/8.15.2) with ESMTPS id 441JcOrF088111
	(version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO)
	for <freebsd-stable@freebsd.org>; Wed, 1 May 2024 15:38:24 -0400 (EDT)
	(envelope-from mike@sentex.net)
Message-ID: <5e1b5097-c1c0-4740-a491-63c709d01c25@sentex.net>
Date: Wed, 1 May 2024 15:38:24 -0400
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Content-Language: en-US
To: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org>
From: mike tancsa <mike@sentex.net>
Subject: how to tell if TRIM is working
Autocrypt: addr=mike@sentex.net; keydata=
 xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP
 xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s
 +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW
 yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH
 VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB
 AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs
 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W
 rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS
 j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ
 cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R
 yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw
 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk
 AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1
 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt
 RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO
 l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7
 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB
 CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8
 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM
 aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X
 rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB
 Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0
 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg==
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Scanned-By: MIMEDefang 2.86
X-Spamd-Bar: ---
X-Spamd-Result: default: False [-3.34 / 15.00];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-0.95)[-0.950];
	R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32];
	MIME_GOOD(-0.10)[text/plain];
	RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received];
	XM_UA_NO_VERSION(0.01)[];
	TO_DN_ALL(0.00)[];
	RCPT_COUNT_ONE(0.00)[1];
	FREEFALL_USER(0.00)[mike];
	ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA];
	MIME_TRACE(0.00)[0:+];
	MID_RHS_MATCH_FROM(0.00)[];
	R_DKIM_NA(0.00)[];
	MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org];
	FROM_EQ_ENVFROM(0.00)[];
	FROM_HAS_DN(0.00)[];
	ARC_NA(0.00)[];
	RCVD_COUNT_TWO(0.00)[2];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org];
	TO_MATCH_ENVRCPT_ALL(0.00)[];
	DMARC_NA(0.00)[sentex.net];
	RCVD_TLS_ALL(0.00)[]
X-Rspamd-Queue-Id: 4VV6n30Dszz4fNb

Kind of struggling to check if TRIM is actually working or not with my 
SSDs on RELENG_14 in ZFS.

On a pool that has almost no files on it (capacity at 0% out of 3TB), 
should not

zpool -w trim <pool> be almost instant after a couple of runs ? Instead 
it seems to always take about 10min to complete.

Looking at the stats,

kstat.zfs.tortank1.misc.iostats.trim_bytes_failed: 0
kstat.zfs.tortank1.misc.iostats.trim_extents_failed: 0
kstat.zfs.tortank1.misc.iostats.trim_bytes_skipped: 2743435264
kstat.zfs.tortank1.misc.iostats.trim_extents_skipped: 253898
kstat.zfs.tortank1.misc.iostats.trim_bytes_written: 14835526799360
kstat.zfs.tortank1.misc.iostats.trim_extents_written: 1169158

what and why are bytes being skipped ?

One of the drives for example

 Â sysctl -a kern.cam.ada.0
kern.cam.ada.0.trim_ticks: 0
kern.cam.ada.0.trim_goal: 0
kern.cam.ada.0.sort_io_queue: 0
kern.cam.ada.0.rotating: 0
kern.cam.ada.0.unmapped_io: 1
kern.cam.ada.0.flags: 
0x1be3bde<CAN_48BIT,CAN_FLUSHCACHE,CAN_NCQ,CAN_DMA,WAS_OTAG,CAN_TRIM,OPEN,SCTX_INIT,CAN_POWERMGT,CAN_DMA48,CAN_LOG,CAN_WCACHE,CAN_RAHEAD,PROBED,ANNOUNCED,DIRTY,PIM_ATA_EXT,UNMAPPEDIO>
kern.cam.ada.0.max_seq_zones: 0
kern.cam.ada.0.optimal_nonseq_zones: 0
kern.cam.ada.0.optimal_seq_zones: 0
kern.cam.ada.0.zone_support: None
kern.cam.ada.0.zone_mode: Not Zoned
kern.cam.ada.0.write_cache: -1
kern.cam.ada.0.read_ahead: -1
kern.cam.ada.0.trim_lbas: 7771432624
kern.cam.ada.0.trim_ranges: 371381
kern.cam.ada.0.trim_count: 310842
kern.cam.ada.0.delete_method: DSM_TRIM

If I take one of the disks out of the pool and replace it with a spare, 
and do a manual trim it seems to work

e.g. here was one disk in the pool that was taking a long time for each 
zpool trim

# time trim -f /dev/ada1
trim /dev/ada1 offset 0 length 1000204886016
0.000u 0.057s 1:29.33 0.0%Â Â Â Â Â  5+184k 0+0io 0pf+0w
and then if I re-run it
#Â  time trim -f /dev/ada1
trim /dev/ada1 offset 0 length 1000204886016
0.000u 0.052s 0:04.15 1.2%Â Â Â Â Â  1+52k 0+0io 0pf+0w

90 seconds and then 4 seconds after that.


 Â Â Â  ---Mike


From nobody Wed May  1 20:24:53 2024
X-Original-To: stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VV7pk59D3z5K0nX
	for <stable@mlmmj.nyi.freebsd.org>; Wed,  1 May 2024 20:25:02 +0000 (UTC)
	(envelope-from mgrooms@shrew.net)
Received: from mx1.shrew.net (mx1.shrew.net [204.27.62.57])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VV7pj6dDQz4lD4
	for <stable@freebsd.org>; Wed,  1 May 2024 20:25:01 +0000 (UTC)
	(envelope-from mgrooms@shrew.net)
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=shrew.net header.s=default header.b="k84/APl6";
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.57 as permitted sender) smtp.mailfrom=mgrooms@shrew.net
Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18])
	by mx1.shrew.net (8.17.1/8.17.1) with ESMTP id 441KOsPm043904
	for <stable@freebsd.org>; Wed, 1 May 2024 15:24:54 -0500 (CDT)
	(envelope-from mgrooms@shrew.net)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default;
	t=1714595094;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=jfjdBsXmJkF8YaXxk9WmJeugZf+RdVZ/jxvWicSZvPk=;
	b=k84/APl6d4vE9SnO5FOGGA9ZEFdRF2jh7Bk8dtCyQRLYdoLWEpHXyeDp3PyN8tDZSSl242
	7kSmnQLUE2FVMhsaWSOlfaPEqCtzeU0esWB2AN42Bi6IXj7dgoFl88cM3qAwCfqCjaQfBK
	+iRj6yA4VNW8hCT6DXFgCzMRFE1QnQ4EnUA2xfedE1cDjYBBEKt6PRmERnNsD7s58N+MhY
	2/fGwOS3yNiM4RJELGBR/D2BRMRsAmFX476kq3nIVkZE3gXJ1a0Q53WONtG9X/xuJQPHlJ
	FPlkeyj8Sk0NCC5Dlh4PZ9AWEf+9CaMnHiE1nFNf6NzlH/ad/nH/11HXdmEpSA==
Received: from [10.22.200.32] (unknown [136.62.156.42])
	by mail.shrew.net (Postfix) with ESMTPSA id 7416C3AB37
	for <stable@freebsd.org>; Wed,  1 May 2024 15:24:54 -0500 (CDT)
Message-ID: <67721332-fa1d-4b3c-aa57-64594ad5d77a@shrew.net>
Date: Wed, 1 May 2024 15:24:53 -0500
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: how to tell if TRIM is working
To: stable@freebsd.org
References: <5e1b5097-c1c0-4740-a491-63c709d01c25@sentex.net>
Content-Language: en-US
From: Matthew Grooms <mgrooms@shrew.net>
In-Reply-To: <5e1b5097-c1c0-4740-a491-63c709d01c25@sentex.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Spamd-Bar: ---
X-Spamd-Result: default: False [-3.49 / 15.00];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-1.00)[-1.000];
	R_SPF_ALLOW(-0.20)[+mx];
	R_DKIM_ALLOW(-0.20)[shrew.net:s=default];
	MIME_GOOD(-0.10)[text/plain];
	XM_UA_NO_VERSION(0.01)[];
	RCPT_COUNT_ONE(0.00)[1];
	DKIM_TRACE(0.00)[shrew.net:+];
	DMARC_NA(0.00)[shrew.net];
	RCVD_TLS_LAST(0.00)[];
	TO_MATCH_ENVRCPT_ALL(0.00)[];
	FROM_HAS_DN(0.00)[];
	RCVD_VIA_SMTP_AUTH(0.00)[];
	ARC_NA(0.00)[];
	RCVD_COUNT_TWO(0.00)[2];
	FROM_EQ_ENVFROM(0.00)[];
	ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US];
	TO_DN_NONE(0.00)[];
	MLMMJ_DEST(0.00)[stable@freebsd.org];
	PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org];
	MID_RHS_MATCH_FROM(0.00)[];
	MIME_TRACE(0.00)[0:+]
X-Rspamd-Queue-Id: 4VV7pj6dDQz4lD4

On 5/1/24 14:38, mike tancsa wrote:
> Kind of struggling to check if TRIM is actually working or not with my 
> SSDs on RELENG_14 in ZFS.
>
> On a pool that has almost no files on it (capacity at 0% out of 3TB), 
> should not
>
> zpool -w trim <pool> be almost instant after a couple of runs ? 
> Instead it seems to always take about 10min to complete.
>
> Looking at the stats,
>
> kstat.zfs.tortank1.misc.iostats.trim_bytes_failed: 0
> kstat.zfs.tortank1.misc.iostats.trim_extents_failed: 0
> kstat.zfs.tortank1.misc.iostats.trim_bytes_skipped: 2743435264
> kstat.zfs.tortank1.misc.iostats.trim_extents_skipped: 253898
> kstat.zfs.tortank1.misc.iostats.trim_bytes_written: 14835526799360
> kstat.zfs.tortank1.misc.iostats.trim_extents_written: 1169158
>
> what and why are bytes being skipped ?
>
> One of the drives for example
>
> Â sysctl -a kern.cam.ada.0
> kern.cam.ada.0.trim_ticks: 0
> kern.cam.ada.0.trim_goal: 0
> kern.cam.ada.0.sort_io_queue: 0
> kern.cam.ada.0.rotating: 0
> kern.cam.ada.0.unmapped_io: 1
> kern.cam.ada.0.flags: 
> 0x1be3bde<CAN_48BIT,CAN_FLUSHCACHE,CAN_NCQ,CAN_DMA,WAS_OTAG,CAN_TRIM,OPEN,SCTX_INIT,CAN_POWERMGT,CAN_DMA48,CAN_LOG,CAN_WCACHE,CAN_RAHEAD,PROBED,ANNOUNCED,DIRTY,PIM_ATA_EXT,UNMAPPEDIO>
> kern.cam.ada.0.max_seq_zones: 0
> kern.cam.ada.0.optimal_nonseq_zones: 0
> kern.cam.ada.0.optimal_seq_zones: 0
> kern.cam.ada.0.zone_support: None
> kern.cam.ada.0.zone_mode: Not Zoned
> kern.cam.ada.0.write_cache: -1
> kern.cam.ada.0.read_ahead: -1
> kern.cam.ada.0.trim_lbas: 7771432624
> kern.cam.ada.0.trim_ranges: 371381
> kern.cam.ada.0.trim_count: 310842
> kern.cam.ada.0.delete_method: DSM_TRIM
>
> If I take one of the disks out of the pool and replace it with a 
> spare, and do a manual trim it seems to work
>
I had a hard time seeing evidence of this at the disk level while 
fiddling with TRIM recently. It appeared that at least some counters are 
driver and operation specific. For example, the da driver appears to 
update counters in some paths but not others. I assume that ada is 
different. There is a bug report for da, but haven't seen any feedback ...

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277673

You could try to run gstat with the -d flag during the time period when 
the delete operations are expected to occur. That should give you an 
idea of what's happening at the disk level in real time but may not 
offer more info than you're already seeing.

> e.g. here was one disk in the pool that was taking a long time for 
> each zpool trim
>
> # time trim -f /dev/ada1
> trim /dev/ada1 offset 0 length 1000204886016
> 0.000u 0.057s 1:29.33 0.0%Â Â Â Â Â  5+184k 0+0io 0pf+0w
> and then if I re-run it
> #Â  time trim -f /dev/ada1
> trim /dev/ada1 offset 0 length 1000204886016
> 0.000u 0.052s 0:04.15 1.2%Â Â Â Â Â  1+52k 0+0io 0pf+0w
>
> 90 seconds and then 4 seconds after that.
>

-Matthew

From nobody Wed May  1 22:32:33 2024
X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VVBfB6wkkz5K9kt
	for <freebsd-stable@mlmmj.nyi.freebsd.org>; Wed,  1 May 2024 22:32:50 +0000 (UTC)
	(envelope-from alandaluz@gmail.com)
Received: from mail-lj1-x236.google.com (mail-lj1-x236.google.com [IPv6:2a00:1450:4864:20::236])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VVBf960Jrz4vtR
	for <freebsd-stable@freebsd.org>; Wed,  1 May 2024 22:32:49 +0000 (UTC)
	(envelope-from alandaluz@gmail.com)
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=gmail.com header.s=20230601 header.b=jUc2x5kC;
	dmarc=pass (policy=none) header.from=gmail.com;
	spf=pass (mx1.freebsd.org: domain of alandaluz@gmail.com designates 2a00:1450:4864:20::236 as permitted sender) smtp.mailfrom=alandaluz@gmail.com
Received: by mail-lj1-x236.google.com with SMTP id 38308e7fff4ca-2dd6a7ae2dcso118162011fa.1
        for <freebsd-stable@freebsd.org>; Wed, 01 May 2024 15:32:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1714602765; x=1715207565; darn=freebsd.org;
        h=to:subject:message-id:date:from:reply-to:mime-version:from:to:cc
         :subject:date:message-id:reply-to;
        bh=voDBQ8ta1NMFSBVu3TFwP0nfYKX+IsJ9Y4KyVqpTAlk=;
        b=jUc2x5kC5vj9UmFjXiB/n6CQq3yTap6hrJD5g3l+ZfzGW4h18cPimUAa0c/2bTDkHS
         +VFnN71aBwK1WYyUT3LMizYJCtN4Cjlpnf6VuDakwBPyvQ3aWA4JpHQUs8fxFV3l94wG
         LGxANH72i6SA4IsFmX6esXivyDGjvYeptVZ8UYs/lTsDGcOE3/Ih7ZWx/VRuAyrpxbAo
         kAOwkRM/9ssjiTM2pnnkJrLRz9pdK3TyDDJxNwNljnOWOeKe4hiOSQmvY1d5dtOCevcO
         WLwF5fSOz7yaGUQUiPobJvGD4XW0CDtN2lyz0TfeGzkRHz+eHh/ABkQba57nXrb0ZWSI
         +mBA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1714602765; x=1715207565;
        h=to:subject:message-id:date:from:reply-to:mime-version
         :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
        bh=voDBQ8ta1NMFSBVu3TFwP0nfYKX+IsJ9Y4KyVqpTAlk=;
        b=ewel0bgEA0AJFZtr4m3PunNj6zkeBX5+yZO3BroD2AvdVYoQ0tLOKmEXOL1xXHOawe
         ntV5/PbbCenqsY6Qiaudxu3A9J+VeeTB5q+oExdDwdVCnErcogsWaSBCyUdxnIg8+BLF
         Fk6HBI2ACS6AiVPKm4QGHo+l7NzSdGls/JilkP0xaW18Ct8OG9bhL9kciz7nIBZt54MV
         4XGDAnIAIls2lLKDdUIcpL7voajfhbhbCbwXrdabXLdCQlIe50boVB6e+4rmkNrY43Ky
         uQN1lzD8B+UNArNdtVvwKNo5a2cyYFsPT7bItT0H4bxTKmg7D80H5irBeyVisjZkprbn
         qxkg==
X-Gm-Message-State: AOJu0Yy1AG3eEhEJ35FXVEgYgL2kF4DMI28aiwK6Kcg5DdeDtEK7kx8D
	6NWw3Hq9dZMvIkNOnifgrqZKC8424xwzyxGC6hlK0u48rdZaZcdpeATjBfF0pCn2O2Of0JDlhnd
	XAZU0ipUUDJ3eF/Lr0X8nj2w43CNQu0Xa
X-Google-Smtp-Source: AGHT+IG9xLCpY6zW70e0AM4mFhxztQTwvkIyN/ILd0u0f8WLVs4ghNPDvbhsLojaMar1uMyfoog1OcwlozKmMPXtRu8=
X-Received: by 2002:a2e:b8d2:0:b0:2e1:c448:d61e with SMTP id
 s18-20020a2eb8d2000000b002e1c448d61emr461426ljp.15.1714602764549; Wed, 01 May
 2024 15:32:44 -0700 (PDT)
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
Reply-To: cbl@infowest.com
From: "Cassidy B. Larson" <alandaluz@gmail.com>
Date: Wed, 1 May 2024 16:32:33 -0600
Message-ID: <CAPjUTPMB0fUyuc0qOgYGanKVnaTVGx_MnEfhSjtv_9roDt214w@mail.gmail.com>
Subject: Forking on 13.3 restricting to same CPU core
To: FreeBSD <freebsd-stable@freebsd.org>
Content-Type: multipart/alternative; boundary="000000000000b645e406176c10d0"
X-Spamd-Bar: ---
X-Spamd-Result: default: False [-4.00 / 15.00];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-1.00)[-1.000];
	NEURAL_HAM_LONG(-1.00)[-0.998];
	DMARC_POLICY_ALLOW(-0.50)[gmail.com,none];
	R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36];
	R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601];
	MIME_GOOD(-0.10)[multipart/alternative,text/plain];
	RCVD_TLS_LAST(0.00)[];
	ARC_NA(0.00)[];
	RCPT_COUNT_ONE(0.00)[1];
	FROM_HAS_DN(0.00)[];
	MIME_TRACE(0.00)[0:+,1:+,2:~];
	RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::236:from];
	TO_DN_ALL(0.00)[];
	FREEMAIL_ENVFROM(0.00)[gmail.com];
	FREEMAIL_FROM(0.00)[gmail.com];
	TO_MATCH_ENVRCPT_ALL(0.00)[];
	DKIM_TRACE(0.00)[gmail.com:+];
	HAS_REPLYTO(0.00)[cbl@infowest.com];
	MID_RHS_MATCH_FROMTLD(0.00)[];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org];
	FROM_EQ_ENVFROM(0.00)[];
	REPLYTO_DOM_NEQ_FROM_DOM(0.00)[];
	MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org];
	RCVD_COUNT_ONE(0.00)[1];
	ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	MISSING_XM_UA(0.00)[];
	DWL_DNSWL_NONE(0.00)[gmail.com:dkim]
X-Rspamd-Queue-Id: 4VVBf960Jrz4vtR

--000000000000b645e406176c10d0
Content-Type: text/plain; charset="UTF-8"

Hi all,

We have an in-house php script that processes images. Been running it for
years. It forks itself a number of times to increase the amount of images
we can process per second.   13.2 has been running great. When upgrading to
13.3 or 14.0, the php forked processes all end up leveraging the came cpu
core, and that slows us waaay down.

I had a 13.2 userland, and 13.3 kernel and the issue doesnt occur.
Using a 13.3 userland and 13.3 kernel the issues does occur.  Same with a
fresh install of 14.0 userland/kernel.

Leveraging the pkg php82 from 13.2 land on 13.3 or the 13.3 pkg php82
doesnt' matter, both show the same issue.

Something in the userland/libraries in 13.3 apparently restricts the forked
php processes from leveraging any additional cpu cores.  Any ideas
where/what/how/why?

Thanks,

-c

--000000000000b645e406176c10d0
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi all,<div><br></div><div>We have an in-house php script =
that processes images. Been running it for years. It forks itself a number =
of times to increase the amount of images we can process per second. =C2=A0=
 13.2 has been running great. When upgrading to 13.3 or 14.0, the php forke=
d processes all end up leveraging the came cpu core, and that slows us waaa=
y down.=C2=A0</div><div><br></div><div>I had a 13.2 userland, and 13.3 kern=
el and the issue doesnt=C2=A0occur.<br></div><div>Using a 13.3 userland and=
 13.3 kernel the issues does occur.=C2=A0 Same with=C2=A0a fresh install of=
 14.0 userland/kernel.=C2=A0</div><div><br></div><div>Leveraging the pkg ph=
p82 from 13.2 land on 13.3 or the 13.3 pkg php82 doesnt&#39; matter, both s=
how the same issue.=C2=A0</div><div><br></div><div>Something in the userlan=
d/libraries in 13.3 apparently restricts the forked php processes from leve=
raging any additional cpu cores.=C2=A0 Any ideas where/what/how/why?=C2=A0<=
/div><div><br></div><div>Thanks,</div><div><br></div><div>-c</div></div>

--000000000000b645e406176c10d0--

From nobody Thu May  2 12:29:56 2024
X-Original-To: stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VVYD90nSJz5JxcJ
	for <stable@mlmmj.nyi.freebsd.org>; Thu,  2 May 2024 12:30:01 +0000 (UTC)
	(envelope-from mike@sentex.net)
Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VVYD84bntz44V6
	for <stable@freebsd.org>; Thu,  2 May 2024 12:30:00 +0000 (UTC)
	(envelope-from mike@sentex.net)
Authentication-Results: mx1.freebsd.org;
	none
Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19])
	by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 442CTxu8030979
	(version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL);
	Thu, 2 May 2024 08:29:59 -0400 (EDT)
	(envelope-from mike@sentex.net)
Received: from [IPV6:2607:f3e0:0:4:bd4c:f42f:eabc:4f29] ([IPv6:2607:f3e0:0:4:bd4c:f42f:eabc:4f29])
	by pyroxene2a.sentex.ca (8.18.1/8.15.2) with ESMTPS id 442CTv2N004147
	(version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO);
	Thu, 2 May 2024 08:29:58 -0400 (EDT)
	(envelope-from mike@sentex.net)
Message-ID: <77e203b3-c555-408b-9634-c452cb3a57ac@sentex.net>
Date: Thu, 2 May 2024 08:29:56 -0400
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: how to tell if TRIM is working
To: Matthew Grooms <mgrooms@shrew.net>, stable@freebsd.org
References: <5e1b5097-c1c0-4740-a491-63c709d01c25@sentex.net>
 <67721332-fa1d-4b3c-aa57-64594ad5d77a@shrew.net>
Content-Language: en-US
From: mike tancsa <mike@sentex.net>
Autocrypt: addr=mike@sentex.net; keydata=
 xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP
 xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s
 +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW
 yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH
 VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB
 AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs
 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W
 rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS
 j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ
 cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R
 yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw
 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk
 AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1
 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt
 RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO
 l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7
 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB
 CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8
 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM
 aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X
 rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB
 Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0
 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg==
In-Reply-To: <67721332-fa1d-4b3c-aa57-64594ad5d77a@shrew.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Scanned-By: MIMEDefang 2.86
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]
X-Rspamd-Queue-Id: 4VVYD84bntz44V6

On 5/1/2024 4:24 PM, Matthew Grooms wrote:
> On 5/1/24 14:38, mike tancsa wrote:
>> Kind of struggling to check if TRIM is actually working or not with 
>> my SSDs on RELENG_14 in ZFS.
>>
>> On a pool that has almost no files on it (capacity at 0% out of 3TB), 
>> should not
>>
>> zpool -w trim <pool> be almost instant after a couple of runs ? 
>> Instead it seems to always take about 10min to complete.
>>
>> Looking at the stats,
>>
>> kstat.zfs.tortank1.misc.iostats.trim_bytes_failed: 0
>> kstat.zfs.tortank1.misc.iostats.trim_extents_failed: 0
>> kstat.zfs.tortank1.misc.iostats.trim_bytes_skipped: 2743435264
>> kstat.zfs.tortank1.misc.iostats.trim_extents_skipped: 253898
>> kstat.zfs.tortank1.misc.iostats.trim_bytes_written: 14835526799360
>> kstat.zfs.tortank1.misc.iostats.trim_extents_written: 1169158
>>
>> what and why are bytes being skipped ?
>>
>> One of the drives for example I had a hard time seeing evidence of 
>> this at the disk level while fiddling with TRIM recently. It appeared 
>> that at least some counters are driver and operation specific. For 
>> example, the da driver appears to update counters in some paths but 
>> not others. I assume that ada is different. There is a bug report for 
>> da, but haven't seen any feedback ... 
>
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277673
>
> You could try to run gstat with the -d flag during the time period 
> when the delete operations are expected to occur. That should give you 
> an idea of what's happening at the disk level in real time but may not 
> offer more info than you're already seeing.
>

It *seems* to be doing something.Â  What I dont understand is why if I 
run it once, do nothing (no writes / snapshots etc), and then run trim 
again, it seems to be doing something with gstat even though there 
should not be anything to mark as being trimmed ?

dT: 1.002sÂ  w: 1.000s
 Â L(q)Â  ops/sÂ Â Â  r/sÂ Â  kBpsÂ Â  ms/rÂ Â Â  w/sÂ Â  kBpsÂ Â  ms/wÂ Â Â  d/s kBpsÂ Â  
ms/dÂ Â  %busy Name
 Â Â Â  0Â Â  1254Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â Â  986Â Â  5202Â Â Â  2.0Â Â Â  244 
8362733Â Â Â  4.5Â Â  55.6Â  ada0
 Â Â  12Â Â  1242Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1012Â Â  5218Â Â Â  1.9Â Â Â  206 
4972041Â Â Â  6.0Â Â  63.3Â  ada2
 Â Â  12Â Â  1242Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1012Â Â  5218Â Â Â  1.9Â Â Â  206 
4972041Â Â Â  6.0Â Â  63.3Â  ada2p1
 Â Â Â  0Â Â  4313Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1024Â Â  5190Â Â Â  0.8Â Â  3266 
6463815Â Â Â  0.4Â Â  62.8Â  ada3
 Â Â Â  0Â Â  1254Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â Â  986Â Â  5202Â Â Â  2.0Â Â Â  244 
8362733Â Â Â  4.5Â Â  55.6Â  ada0p1
 Â Â Â  0Â Â  4238Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â Â  960Â Â  4874Â Â Â  0.7Â Â  3254 
6280362Â Â Â  0.4Â Â  59.8Â  ada5
 Â Â Â  0Â Â  4313Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1024Â Â  5190Â Â Â  0.8Â Â  3266 
6463815Â Â Â  0.4Â Â  62.8Â  ada3p1
 Â Â Â  0Â Â  4238Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â Â  960Â Â  4874Â Â Â  0.7Â Â  3254 
6280362Â Â Â  0.4Â Â  59.8Â  ada5p1
dT: 1.001sÂ  w: 1.000s
 Â L(q)Â  ops/sÂ Â Â  r/sÂ Â  kBpsÂ Â  ms/rÂ Â Â  w/sÂ Â  kBpsÂ Â  ms/wÂ Â Â  d/s kBpsÂ Â  
ms/dÂ Â  %busy Name
 Â Â Â  2Â Â  2381Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1580Â Â  9946Â Â Â  0.9Â Â Â  767 
5990286Â Â Â  1.8Â Â  70.0Â  ada0
 Â Â Â  2Â Â  2801Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1540Â Â  9782Â Â Â  0.9Â Â  1227 
11936510Â Â Â  1.0Â Â  65.2Â  ada2
 Â Â Â  2Â Â  2801Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1540Â Â  9782Â Â Â  0.9Â Â  1227 
11936510Â Â Â  1.0Â Â  65.2Â  ada2p1
 Â Â Â  0Â Â  2072Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1529Â Â  9566Â Â Â  0.8Â Â Â  509 
12549587Â Â Â  2.1Â Â  57.0Â  ada3
 Â Â Â  2Â Â  2381Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1580Â Â  9946Â Â Â  0.9Â Â Â  767 
5990286Â Â Â  1.8Â Â  70.0Â  ada0p1
 Â Â Â  0Â Â  2042Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1517Â Â  9427Â Â Â  0.6Â Â Â  491 
12549535Â Â Â  1.9Â Â  52.4Â  ada5
 Â Â Â  0Â Â  2072Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1529Â Â  9566Â Â Â  0.8Â Â Â  509 
12549587Â Â Â  2.1Â Â  57.0Â  ada3p1
 Â Â Â  0Â Â  2042Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1517Â Â  9427Â Â Â  0.6Â Â Â  491 
12549535Â Â Â  1.9Â Â  52.4Â  ada5p1
dT: 1.002sÂ  w: 1.000s
 Â L(q)Â  ops/sÂ Â Â  r/sÂ Â  kBpsÂ Â  ms/rÂ Â Â  w/sÂ Â  kBpsÂ Â  ms/wÂ Â Â  d/s kBpsÂ Â  
ms/dÂ Â  %busy Name
 Â Â Â  2Â Â  1949Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1094Â Â  5926Â Â Â  1.2Â Â Â  827 
11267200Â Â Â  1.8Â Â  78.8Â  ada0
 Â Â Â  0Â Â  2083Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1115Â Â  6034Â Â Â  0.7Â Â Â  939 
16537981Â Â Â  1.4Â Â  67.2Â  ada2
 Â Â Â  0Â Â  2083Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1115Â Â  6034Â Â Â  0.7Â Â Â  939 
16537981Â Â Â  1.4Â Â  67.2Â  ada2p1
 Â Â Â  2Â Â  2525Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1098Â Â  5914Â Â Â  0.8Â Â  1399 
16021615Â Â Â  1.1Â Â  79.3Â  ada3
 Â Â Â  2Â Â  1949Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1094Â Â  5926Â Â Â  1.2Â Â Â  827 
11267200Â Â Â  1.8Â Â  78.8Â  ada0p1
 Â Â  12Â Â  2471Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1018Â Â  5399Â Â Â  1.0Â Â  1425 
15395566Â Â Â  1.1Â Â  80.5Â  ada5
 Â Â Â  2Â Â  2525Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1098Â Â  5914Â Â Â  0.8Â Â  1399 
16021615Â Â Â  1.1Â Â  79.3Â  ada3p1
 Â Â  12Â Â  2471Â Â Â Â Â  0Â Â Â Â Â  0Â Â Â  0.0Â Â  1018Â Â  5399Â Â Â  1.0Â Â  1425 
15395566Â Â Â  1.1Â Â  80.5Â  ada5p1

The ultimate problem is that after a while with a lot of writes, the 
disk performance will be toast until I do a manual trim -f of the disk 
:(Â Â  this is most notable on consumer WD SSDs.Â  I havent done any 
extensive tests with Samsung SSDs to see if there are performance 
penalties or not. It might be that they are just better at masking the 
problem.Â  I dont see the same issue with ZFS on Linux with the same 
disks / hardware

I have an open PR in 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277992 that I think 
might actually have 2 separate problems.

 Â Â Â  ---Mike


>> e.g. here was one disk in the pool that was taking a long time for 
>> each zpool trim
>>
>> # time trim -f /dev/ada1
>> trim /dev/ada1 offset 0 length 1000204886016
>> 0.000u 0.057s 1:29.33 0.0%Â Â Â Â Â  5+184k 0+0io 0pf+0w
>> and then if I re-run it
>> #Â  time trim -f /dev/ada1
>> trim /dev/ada1 offset 0 length 1000204886016
>> 0.000u 0.052s 0:04.15 1.2%Â Â Â Â Â  1+52k 0+0io 0pf+0w
>>
>> 90 seconds and then 4 seconds after that.
>>
>
> -Matthew
>

From nobody Thu May  2 14:16:04 2024
X-Original-To: stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VVbZq1TTyz5JNnw
	for <stable@mlmmj.nyi.freebsd.org>; Thu,  2 May 2024 14:16:19 +0000 (UTC)
	(envelope-from wlosh@bsdimp.com)
Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VVbZp5nf5z4HcV
	for <stable@freebsd.org>; Thu,  2 May 2024 14:16:18 +0000 (UTC)
	(envelope-from wlosh@bsdimp.com)
Authentication-Results: mx1.freebsd.org;
	none
Received: by mail-ej1-x633.google.com with SMTP id a640c23a62f3a-a51a7d4466bso894379966b.2
        for <stable@freebsd.org>; Thu, 02 May 2024 07:16:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1714659377; x=1715264177; darn=freebsd.org;
        h=cc:to:subject:message-id:date:from:in-reply-to:references
         :mime-version:from:to:cc:subject:date:message-id:reply-to;
        bh=dbKG79yBL61M1c4W9C58n6X/iKzlcupyjuuCir/paIw=;
        b=WSnAlcCcehpR4H2REDO2wAN+g0lZizNZmaerFq5Bc+9zcRuOBe+NGRRaGqzcXI6s/P
         xPJaJ66jaCsSR6sEBHJcaECeXxBEd97LFKiq4gnPeg6K6jRl3x/kJsFPeLkCaR6oKThP
         Mm36OvhMPgCiybN3F9i6SSTBvo4C95fmP5bsxa53dDVgwjms7sCbVhnXllwYmekWiwqy
         dbEcUn9TYcFKEw7aONTI6xHMagPGuG5IdFEeBnQLyzVgu0uNsrtmLnrsO/9hQO3IBgiF
         8mQOH09XLBMhaF2uAajTOl2IbwAy5AaHp315VWZvBGXG0ZgbUVvBrp7e8HtmDXDsKyCP
         svKw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1714659377; x=1715264177;
        h=cc:to:subject:message-id:date:from:in-reply-to:references
         :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
         :reply-to;
        bh=dbKG79yBL61M1c4W9C58n6X/iKzlcupyjuuCir/paIw=;
        b=Sxy0jOa5EFKsNTa1DQGyLU5cLMYOQtVZNLTEpPjzTcGn79SP2d+oztOYiC2UAk48Nn
         tQK8eATYhfamLEyULsxQDIZEjGpAaDkf2673Y6u2vZIAwc0aMqtmTF6NgKFQ6xFiN+W/
         VUkrxrlSu6s+VbeYXSsl9srLvHZG5u+l2BpnFyeuiQwA1FvzGG+fmazPrY3WqF1rY2h1
         WfHlwzRuKjBi9o5HmQdkQi5Zm+PGg1PB9ndxEMlGN3tz2CEZi0EI9rSCnM9t0/Q3lVOI
         fBiS02Y07llhm+boL55wAUvwQ80X3fBp+IGEAA3bxuuVQEPkP3Mc084hd+1Nx0L9vDQA
         Kpkw==
X-Forwarded-Encrypted: i=1; AJvYcCX1MshqOi9NDB73u3GoqtNVdF29z6eL0c4tdccf9FaKTta0RpSv9KweTtKshJBUD1BISgfMcVkG0AD7vO9PnzoX5+g=
X-Gm-Message-State: AOJu0YypqGJ89CBT6eIdUpDtUlSEdCixgF42tgZ4h3MpjeCv493U67KV
	QNqGojLuwq5U0h0/s5Va+aVcruHjePRh5AKLzsafu9boXno7uWuy8eANM8iyXTxddlimfjtE7Zo
	Jtmvg8kHMFp4MJaVbQJl6Icpw2DEKKT559HQTu/XIsCA9OwqeLHQ=
X-Google-Smtp-Source: AGHT+IFaQPTsHkRd5MmEYrb+r7V6cl/UrAzk0nGWN70JBjO+xTL3/Azuq62q8MlH8greDVbVtRU6+XOKUsC3keMJzlM=
X-Received: by 2002:a17:906:b883:b0:a52:645a:8ebd with SMTP id
 hb3-20020a170906b88300b00a52645a8ebdmr4129591ejb.14.1714659377167; Thu, 02
 May 2024 07:16:17 -0700 (PDT)
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
References: <5e1b5097-c1c0-4740-a491-63c709d01c25@sentex.net>
 <67721332-fa1d-4b3c-aa57-64594ad5d77a@shrew.net> <77e203b3-c555-408b-9634-c452cb3a57ac@sentex.net>
In-Reply-To: <77e203b3-c555-408b-9634-c452cb3a57ac@sentex.net>
From: Warner Losh <imp@bsdimp.com>
Date: Thu, 2 May 2024 08:16:04 -0600
Message-ID: <CANCZdfqx_vhNb2BukbM0bxrf8NH_9sXPKW+Uf=LdoXjw_2w=Dg@mail.gmail.com>
Subject: Re: how to tell if TRIM is working
To: mike tancsa <mike@sentex.net>
Cc: Matthew Grooms <mgrooms@shrew.net>, stable@freebsd.org
Content-Type: multipart/alternative; boundary="000000000000165b720617793f29"
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]
X-Rspamd-Queue-Id: 4VVbZp5nf5z4HcV

--000000000000165b720617793f29
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Thu, May 2, 2024 at 6:30=E2=80=AFAM mike tancsa <mike@sentex.net> wrote:

> On 5/1/2024 4:24 PM, Matthew Grooms wrote:
> > On 5/1/24 14:38, mike tancsa wrote:
> >> Kind of struggling to check if TRIM is actually working or not with
> >> my SSDs on RELENG_14 in ZFS.
> >>
> >> On a pool that has almost no files on it (capacity at 0% out of 3TB),
> >> should not
> >>
> >> zpool -w trim <pool> be almost instant after a couple of runs ?
> >> Instead it seems to always take about 10min to complete.
> >>
> >> Looking at the stats,
> >>
> >> kstat.zfs.tortank1.misc.iostats.trim_bytes_failed: 0
> >> kstat.zfs.tortank1.misc.iostats.trim_extents_failed: 0
> >> kstat.zfs.tortank1.misc.iostats.trim_bytes_skipped: 2743435264
> >> kstat.zfs.tortank1.misc.iostats.trim_extents_skipped: 253898
> >> kstat.zfs.tortank1.misc.iostats.trim_bytes_written: 14835526799360
> >> kstat.zfs.tortank1.misc.iostats.trim_extents_written: 1169158
> >>
> >> what and why are bytes being skipped ?
> >>
> >> One of the drives for example I had a hard time seeing evidence of
> >> this at the disk level while fiddling with TRIM recently. It appeared
> >> that at least some counters are driver and operation specific. For
> >> example, the da driver appears to update counters in some paths but
> >> not others. I assume that ada is different. There is a bug report for
> >> da, but haven't seen any feedback ...
> >
> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277673
> >
> > You could try to run gstat with the -d flag during the time period
> > when the delete operations are expected to occur. That should give you
> > an idea of what's happening at the disk level in real time but may not
> > offer more info than you're already seeing.
> >
>
> It *seems* to be doing something.  What I dont understand is why if I
> run it once, do nothing (no writes / snapshots etc), and then run trim
> again, it seems to be doing something with gstat even though there
> should not be anything to mark as being trimmed ?
>
> dT: 1.002s  w: 1.000s
>   L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps
> ms/d   %busy Name
>      0   1254      0      0    0.0    986   5202    2.0    244
> 8362733    4.5   55.6  ada0
>     12   1242      0      0    0.0   1012   5218    1.9    206
> 4972041    6.0   63.3  ada2
>     12   1242      0      0    0.0   1012   5218    1.9    206
> 4972041    6.0   63.3  ada2p1
>      0   4313      0      0    0.0   1024   5190    0.8   3266
> 6463815    0.4   62.8  ada3
>      0   1254      0      0    0.0    986   5202    2.0    244
> 8362733    4.5   55.6  ada0p1
>      0   4238      0      0    0.0    960   4874    0.7   3254
> 6280362    0.4   59.8  ada5
>      0   4313      0      0    0.0   1024   5190    0.8   3266
> 6463815    0.4   62.8  ada3p1
>      0   4238      0      0    0.0    960   4874    0.7   3254
> 6280362    0.4   59.8  ada5p1
> dT: 1.001s  w: 1.000s
>   L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps
> ms/d   %busy Name
>      2   2381      0      0    0.0   1580   9946    0.9    767
> 5990286    1.8   70.0  ada0
>      2   2801      0      0    0.0   1540   9782    0.9   1227
> 11936510    1.0   65.2  ada2
>      2   2801      0      0    0.0   1540   9782    0.9   1227
> 11936510    1.0   65.2  ada2p1
>      0   2072      0      0    0.0   1529   9566    0.8    509
> 12549587    2.1   57.0  ada3
>      2   2381      0      0    0.0   1580   9946    0.9    767
> 5990286    1.8   70.0  ada0p1
>      0   2042      0      0    0.0   1517   9427    0.6    491
> 12549535    1.9   52.4  ada5
>      0   2072      0      0    0.0   1529   9566    0.8    509
> 12549587    2.1   57.0  ada3p1
>      0   2042      0      0    0.0   1517   9427    0.6    491
> 12549535    1.9   52.4  ada5p1
> dT: 1.002s  w: 1.000s
>   L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps
> ms/d   %busy Name
>      2   1949      0      0    0.0   1094   5926    1.2    827
> 11267200    1.8   78.8  ada0
>      0   2083      0      0    0.0   1115   6034    0.7    939
> 16537981    1.4   67.2  ada2
>      0   2083      0      0    0.0   1115   6034    0.7    939
> 16537981    1.4   67.2  ada2p1
>      2   2525      0      0    0.0   1098   5914    0.8   1399
> 16021615    1.1   79.3  ada3
>      2   1949      0      0    0.0   1094   5926    1.2    827
> 11267200    1.8   78.8  ada0p1
>     12   2471      0      0    0.0   1018   5399    1.0   1425
> 15395566    1.1   80.5  ada5
>      2   2525      0      0    0.0   1098   5914    0.8   1399
> 16021615    1.1   79.3  ada3p1
>     12   2471      0      0    0.0   1018   5399    1.0   1425
> 15395566    1.1   80.5  ada5p1
>
> The ultimate problem is that after a while with a lot of writes, the
> disk performance will be toast until I do a manual trim -f of the disk
> :(   this is most notable on consumer WD SSDs.  I havent done any
> extensive tests with Samsung SSDs to see if there are performance
> penalties or not. It might be that they are just better at masking the
> problem.  I dont see the same issue with ZFS on Linux with the same
> disks / hardware
>

When trims are fast, you want to send them to the drive as soon as you
know the blocks are freed. UFS always does this (if trim is enabled at all)=
.
ZFS has a lot of knobs to control when / how / if this is done.

vfs.zfs.vdev.trim_min_active: 1
vfs.zfs.vdev.trim_max_active: 2
vfs.zfs.trim.queue_limit: 10
vfs.zfs.trim.txg_batch: 32
vfs.zfs.trim.metaslab_skip: 0
vfs.zfs.trim.extent_bytes_min: 32768
vfs.zfs.trim.extent_bytes_max: 134217728
vfs.zfs.l2arc.trim_ahead: 0

I've not tried to tune these in the past, but you can see how they
affect things.


Warner



> I have an open PR in
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277992 that I think
> might actually have 2 separate problems.
>
>      ---Mike
>
>
> >> e.g. here was one disk in the pool that was taking a long time for
> >> each zpool trim
> >>
> >> # time trim -f /dev/ada1
> >> trim /dev/ada1 offset 0 length 1000204886016
> >> 0.000u 0.057s 1:29.33 0.0%      5+184k 0+0io 0pf+0w
> >> and then if I re-run it
> >> #  time trim -f /dev/ada1
> >> trim /dev/ada1 offset 0 length 1000204886016
> >> 0.000u 0.052s 0:04.15 1.2%      1+52k 0+0io 0pf+0w
> >>
> >> 90 seconds and then 4 seconds after that.
> >>
> >
> > -Matthew
> >
>
>

--000000000000165b720617793f29
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: base64

PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0ciI+PGJyPjwvZGl2Pjxicj48ZGl2IGNsYXNzPSJn
bWFpbF9xdW90ZSI+PGRpdiBkaXI9Imx0ciIgY2xhc3M9ImdtYWlsX2F0dHIiPk9uIFRodSwgTWF5
IDIsIDIwMjQgYXQgNjozMOKAr0FNIG1pa2UgdGFuY3NhICZsdDs8YSBocmVmPSJtYWlsdG86bWlr
ZUBzZW50ZXgubmV0Ij5taWtlQHNlbnRleC5uZXQ8L2E+Jmd0OyB3cm90ZTo8YnI+PC9kaXY+PGJs
b2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAu
OGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDox
ZXgiPk9uIDUvMS8yMDI0IDQ6MjQgUE0sIE1hdHRoZXcgR3Jvb21zIHdyb3RlOjxicj4NCiZndDsg
T24gNS8xLzI0IDE0OjM4LCBtaWtlIHRhbmNzYSB3cm90ZTo8YnI+DQomZ3Q7Jmd0OyBLaW5kIG9m
IHN0cnVnZ2xpbmcgdG8gY2hlY2sgaWYgVFJJTSBpcyBhY3R1YWxseSB3b3JraW5nIG9yIG5vdCB3
aXRoIDxicj4NCiZndDsmZ3Q7IG15IFNTRHMgb24gUkVMRU5HXzE0IGluIFpGUy48YnI+DQomZ3Q7
Jmd0Ozxicj4NCiZndDsmZ3Q7IE9uIGEgcG9vbCB0aGF0IGhhcyBhbG1vc3Qgbm8gZmlsZXMgb24g
aXQgKGNhcGFjaXR5IGF0IDAlIG91dCBvZiAzVEIpLCA8YnI+DQomZ3Q7Jmd0OyBzaG91bGQgbm90
PGJyPg0KJmd0OyZndDs8YnI+DQomZ3Q7Jmd0OyB6cG9vbCAtdyB0cmltICZsdDtwb29sJmd0OyBi
ZSBhbG1vc3QgaW5zdGFudCBhZnRlciBhIGNvdXBsZSBvZiBydW5zID8gPGJyPg0KJmd0OyZndDsg
SW5zdGVhZCBpdCBzZWVtcyB0byBhbHdheXMgdGFrZSBhYm91dCAxMG1pbiB0byBjb21wbGV0ZS48
YnI+DQomZ3Q7Jmd0Ozxicj4NCiZndDsmZ3Q7IExvb2tpbmcgYXQgdGhlIHN0YXRzLDxicj4NCiZn
dDsmZ3Q7PGJyPg0KJmd0OyZndDsga3N0YXQuemZzLnRvcnRhbmsxLm1pc2MuaW9zdGF0cy50cmlt
X2J5dGVzX2ZhaWxlZDogMDxicj4NCiZndDsmZ3Q7IGtzdGF0Lnpmcy50b3J0YW5rMS5taXNjLmlv
c3RhdHMudHJpbV9leHRlbnRzX2ZhaWxlZDogMDxicj4NCiZndDsmZ3Q7IGtzdGF0Lnpmcy50b3J0
YW5rMS5taXNjLmlvc3RhdHMudHJpbV9ieXRlc19za2lwcGVkOiAyNzQzNDM1MjY0PGJyPg0KJmd0
OyZndDsga3N0YXQuemZzLnRvcnRhbmsxLm1pc2MuaW9zdGF0cy50cmltX2V4dGVudHNfc2tpcHBl
ZDogMjUzODk4PGJyPg0KJmd0OyZndDsga3N0YXQuemZzLnRvcnRhbmsxLm1pc2MuaW9zdGF0cy50
cmltX2J5dGVzX3dyaXR0ZW46IDE0ODM1NTI2Nzk5MzYwPGJyPg0KJmd0OyZndDsga3N0YXQuemZz
LnRvcnRhbmsxLm1pc2MuaW9zdGF0cy50cmltX2V4dGVudHNfd3JpdHRlbjogMTE2OTE1ODxicj4N
CiZndDsmZ3Q7PGJyPg0KJmd0OyZndDsgd2hhdCBhbmQgd2h5IGFyZSBieXRlcyBiZWluZyBza2lw
cGVkID88YnI+DQomZ3Q7Jmd0Ozxicj4NCiZndDsmZ3Q7IE9uZSBvZiB0aGUgZHJpdmVzIGZvciBl
eGFtcGxlIEkgaGFkIGEgaGFyZCB0aW1lIHNlZWluZyBldmlkZW5jZSBvZiA8YnI+DQomZ3Q7Jmd0
OyB0aGlzIGF0IHRoZSBkaXNrIGxldmVsIHdoaWxlIGZpZGRsaW5nIHdpdGggVFJJTSByZWNlbnRs
eS4gSXQgYXBwZWFyZWQgPGJyPg0KJmd0OyZndDsgdGhhdCBhdCBsZWFzdCBzb21lIGNvdW50ZXJz
IGFyZSBkcml2ZXIgYW5kIG9wZXJhdGlvbiBzcGVjaWZpYy4gRm9yIDxicj4NCiZndDsmZ3Q7IGV4
YW1wbGUsIHRoZSBkYSBkcml2ZXIgYXBwZWFycyB0byB1cGRhdGUgY291bnRlcnMgaW4gc29tZSBw
YXRocyBidXQgPGJyPg0KJmd0OyZndDsgbm90IG90aGVycy4gSSBhc3N1bWUgdGhhdCBhZGEgaXMg
ZGlmZmVyZW50LiBUaGVyZSBpcyBhIGJ1ZyByZXBvcnQgZm9yIDxicj4NCiZndDsmZ3Q7IGRhLCBi
dXQgaGF2ZW4mIzM5O3Qgc2VlbiBhbnkgZmVlZGJhY2sgLi4uIDxicj4NCiZndDs8YnI+DQomZ3Q7
IDxhIGhyZWY9Imh0dHBzOi8vYnVncy5mcmVlYnNkLm9yZy9idWd6aWxsYS9zaG93X2J1Zy5jZ2k/
aWQ9Mjc3NjczIiByZWw9Im5vcmVmZXJyZXIiIHRhcmdldD0iX2JsYW5rIj5odHRwczovL2J1Z3Mu
ZnJlZWJzZC5vcmcvYnVnemlsbGEvc2hvd19idWcuY2dpP2lkPTI3NzY3MzwvYT48YnI+DQomZ3Q7
PGJyPg0KJmd0OyBZb3UgY291bGQgdHJ5IHRvIHJ1biBnc3RhdCB3aXRoIHRoZSAtZCBmbGFnIGR1
cmluZyB0aGUgdGltZSBwZXJpb2QgPGJyPg0KJmd0OyB3aGVuIHRoZSBkZWxldGUgb3BlcmF0aW9u
cyBhcmUgZXhwZWN0ZWQgdG8gb2NjdXIuIFRoYXQgc2hvdWxkIGdpdmUgeW91IDxicj4NCiZndDsg
YW4gaWRlYSBvZiB3aGF0JiMzOTtzIGhhcHBlbmluZyBhdCB0aGUgZGlzayBsZXZlbCBpbiByZWFs
IHRpbWUgYnV0IG1heSBub3QgPGJyPg0KJmd0OyBvZmZlciBtb3JlIGluZm8gdGhhbiB5b3UmIzM5
O3JlIGFscmVhZHkgc2VlaW5nLjxicj4NCiZndDs8YnI+DQo8YnI+DQpJdCAqc2VlbXMqIHRvIGJl
IGRvaW5nIHNvbWV0aGluZy7CoCBXaGF0IEkgZG9udCB1bmRlcnN0YW5kIGlzIHdoeSBpZiBJIDxi
cj4NCnJ1biBpdCBvbmNlLCBkbyBub3RoaW5nIChubyB3cml0ZXMgLyBzbmFwc2hvdHMgZXRjKSwg
YW5kIHRoZW4gcnVuIHRyaW0gPGJyPg0KYWdhaW4sIGl0IHNlZW1zIHRvIGJlIGRvaW5nIHNvbWV0
aGluZyB3aXRoIGdzdGF0IGV2ZW4gdGhvdWdoIHRoZXJlIDxicj4NCnNob3VsZCBub3QgYmUgYW55
dGhpbmcgdG8gbWFyayBhcyBiZWluZyB0cmltbWVkID88YnI+DQo8YnI+DQpkVDogMS4wMDJzwqAg
dzogMS4wMDBzPGJyPg0KwqDCoEwocSnCoCBvcHMvc8KgwqDCoCByL3PCoMKgIGtCcHPCoMKgIG1z
L3LCoMKgwqAgdy9zwqDCoCBrQnBzwqDCoCBtcy93wqDCoMKgIGQvcyBrQnBzwqDCoCA8YnI+DQpt
cy9kwqDCoCAlYnVzeSBOYW1lPGJyPg0KwqDCoMKgwqAgMMKgwqAgMTI1NMKgwqDCoMKgwqAgMMKg
wqDCoMKgwqAgMMKgwqDCoCAwLjDCoMKgwqAgOTg2wqDCoCA1MjAywqDCoMKgIDIuMMKgwqDCoCAy
NDQgPGJyPg0KODM2MjczM8KgwqDCoCA0LjXCoMKgIDU1LjbCoCBhZGEwPGJyPg0KwqDCoMKgIDEy
wqDCoCAxMjQywqDCoMKgwqDCoCAwwqDCoMKgwqDCoCAwwqDCoMKgIDAuMMKgwqAgMTAxMsKgwqAg
NTIxOMKgwqDCoCAxLjnCoMKgwqAgMjA2IDxicj4NCjQ5NzIwNDHCoMKgwqAgNi4wwqDCoCA2My4z
wqAgYWRhMjxicj4NCsKgwqDCoCAxMsKgwqAgMTI0MsKgwqDCoMKgwqAgMMKgwqDCoMKgwqAgMMKg
wqDCoCAwLjDCoMKgIDEwMTLCoMKgIDUyMTjCoMKgwqAgMS45wqDCoMKgIDIwNiA8YnI+DQo0OTcy
MDQxwqDCoMKgIDYuMMKgwqAgNjMuM8KgIGFkYTJwMTxicj4NCsKgwqDCoMKgIDDCoMKgIDQzMTPC
oMKgwqDCoMKgIDDCoMKgwqDCoMKgIDDCoMKgwqAgMC4wwqDCoCAxMDI0wqDCoCA1MTkwwqDCoMKg
IDAuOMKgwqAgMzI2NiA8YnI+DQo2NDYzODE1wqDCoMKgIDAuNMKgwqAgNjIuOMKgIGFkYTM8YnI+
DQrCoMKgwqDCoCAwwqDCoCAxMjU0wqDCoMKgwqDCoCAwwqDCoMKgwqDCoCAwwqDCoMKgIDAuMMKg
wqDCoCA5ODbCoMKgIDUyMDLCoMKgwqAgMi4wwqDCoMKgIDI0NCA8YnI+DQo4MzYyNzMzwqDCoMKg
IDQuNcKgwqAgNTUuNsKgIGFkYTBwMTxicj4NCsKgwqDCoMKgIDDCoMKgIDQyMzjCoMKgwqDCoMKg
IDDCoMKgwqDCoMKgIDDCoMKgwqAgMC4wwqDCoMKgIDk2MMKgwqAgNDg3NMKgwqDCoCAwLjfCoMKg
IDMyNTQgPGJyPg0KNjI4MDM2MsKgwqDCoCAwLjTCoMKgIDU5LjjCoCBhZGE1PGJyPg0KwqDCoMKg
wqAgMMKgwqAgNDMxM8KgwqDCoMKgwqAgMMKgwqDCoMKgwqAgMMKgwqDCoCAwLjDCoMKgIDEwMjTC
oMKgIDUxOTDCoMKgwqAgMC44wqDCoCAzMjY2IDxicj4NCjY0NjM4MTXCoMKgwqAgMC40wqDCoCA2
Mi44wqAgYWRhM3AxPGJyPg0KwqDCoMKgwqAgMMKgwqAgNDIzOMKgwqDCoMKgwqAgMMKgwqDCoMKg
wqAgMMKgwqDCoCAwLjDCoMKgwqAgOTYwwqDCoCA0ODc0wqDCoMKgIDAuN8KgwqAgMzI1NCA8YnI+
DQo2MjgwMzYywqDCoMKgIDAuNMKgwqAgNTkuOMKgIGFkYTVwMTxicj4NCmRUOiAxLjAwMXPCoCB3
OiAxLjAwMHM8YnI+DQrCoMKgTChxKcKgIG9wcy9zwqDCoMKgIHIvc8KgwqAga0Jwc8KgwqAgbXMv
csKgwqDCoCB3L3PCoMKgIGtCcHPCoMKgIG1zL3fCoMKgwqAgZC9zIGtCcHPCoMKgIDxicj4NCm1z
L2TCoMKgICVidXN5IE5hbWU8YnI+DQrCoMKgwqDCoCAywqDCoCAyMzgxwqDCoMKgwqDCoCAwwqDC
oMKgwqDCoCAwwqDCoMKgIDAuMMKgwqAgMTU4MMKgwqAgOTk0NsKgwqDCoCAwLjnCoMKgwqAgNzY3
IDxicj4NCjU5OTAyODbCoMKgwqAgMS44wqDCoCA3MC4wwqAgYWRhMDxicj4NCsKgwqDCoMKgIDLC
oMKgIDI4MDHCoMKgwqDCoMKgIDDCoMKgwqDCoMKgIDDCoMKgwqAgMC4wwqDCoCAxNTQwwqDCoCA5
NzgywqDCoMKgIDAuOcKgwqAgMTIyNyA8YnI+DQoxMTkzNjUxMMKgwqDCoCAxLjDCoMKgIDY1LjLC
oCBhZGEyPGJyPg0KwqDCoMKgwqAgMsKgwqAgMjgwMcKgwqDCoMKgwqAgMMKgwqDCoMKgwqAgMMKg
wqDCoCAwLjDCoMKgIDE1NDDCoMKgIDk3ODLCoMKgwqAgMC45wqDCoCAxMjI3IDxicj4NCjExOTM2
NTEwwqDCoMKgIDEuMMKgwqAgNjUuMsKgIGFkYTJwMTxicj4NCsKgwqDCoMKgIDDCoMKgIDIwNzLC
oMKgwqDCoMKgIDDCoMKgwqDCoMKgIDDCoMKgwqAgMC4wwqDCoCAxNTI5wqDCoCA5NTY2wqDCoMKg
IDAuOMKgwqDCoCA1MDkgPGJyPg0KMTI1NDk1ODfCoMKgwqAgMi4xwqDCoCA1Ny4wwqAgYWRhMzxi
cj4NCsKgwqDCoMKgIDLCoMKgIDIzODHCoMKgwqDCoMKgIDDCoMKgwqDCoMKgIDDCoMKgwqAgMC4w
wqDCoCAxNTgwwqDCoCA5OTQ2wqDCoMKgIDAuOcKgwqDCoCA3NjcgPGJyPg0KNTk5MDI4NsKgwqDC
oCAxLjjCoMKgIDcwLjDCoCBhZGEwcDE8YnI+DQrCoMKgwqDCoCAwwqDCoCAyMDQywqDCoMKgwqDC
oCAwwqDCoMKgwqDCoCAwwqDCoMKgIDAuMMKgwqAgMTUxN8KgwqAgOTQyN8KgwqDCoCAwLjbCoMKg
wqAgNDkxIDxicj4NCjEyNTQ5NTM1wqDCoMKgIDEuOcKgwqAgNTIuNMKgIGFkYTU8YnI+DQrCoMKg
wqDCoCAwwqDCoCAyMDcywqDCoMKgwqDCoCAwwqDCoMKgwqDCoCAwwqDCoMKgIDAuMMKgwqAgMTUy
OcKgwqAgOTU2NsKgwqDCoCAwLjjCoMKgwqAgNTA5IDxicj4NCjEyNTQ5NTg3wqDCoMKgIDIuMcKg
wqAgNTcuMMKgIGFkYTNwMTxicj4NCsKgwqDCoMKgIDDCoMKgIDIwNDLCoMKgwqDCoMKgIDDCoMKg
wqDCoMKgIDDCoMKgwqAgMC4wwqDCoCAxNTE3wqDCoCA5NDI3wqDCoMKgIDAuNsKgwqDCoCA0OTEg
PGJyPg0KMTI1NDk1MzXCoMKgwqAgMS45wqDCoCA1Mi40wqAgYWRhNXAxPGJyPg0KZFQ6IDEuMDAy
c8KgIHc6IDEuMDAwczxicj4NCsKgwqBMKHEpwqAgb3BzL3PCoMKgwqAgci9zwqDCoCBrQnBzwqDC
oCBtcy9ywqDCoMKgIHcvc8KgwqAga0Jwc8KgwqAgbXMvd8KgwqDCoCBkL3Mga0Jwc8KgwqAgPGJy
Pg0KbXMvZMKgwqAgJWJ1c3kgTmFtZTxicj4NCsKgwqDCoMKgIDLCoMKgIDE5NDnCoMKgwqDCoMKg
IDDCoMKgwqDCoMKgIDDCoMKgwqAgMC4wwqDCoCAxMDk0wqDCoCA1OTI2wqDCoMKgIDEuMsKgwqDC
oCA4MjcgPGJyPg0KMTEyNjcyMDDCoMKgwqAgMS44wqDCoCA3OC44wqAgYWRhMDxicj4NCsKgwqDC
oMKgIDDCoMKgIDIwODPCoMKgwqDCoMKgIDDCoMKgwqDCoMKgIDDCoMKgwqAgMC4wwqDCoCAxMTE1
wqDCoCA2MDM0wqDCoMKgIDAuN8KgwqDCoCA5MzkgPGJyPg0KMTY1Mzc5ODHCoMKgwqAgMS40wqDC
oCA2Ny4ywqAgYWRhMjxicj4NCsKgwqDCoMKgIDDCoMKgIDIwODPCoMKgwqDCoMKgIDDCoMKgwqDC
oMKgIDDCoMKgwqAgMC4wwqDCoCAxMTE1wqDCoCA2MDM0wqDCoMKgIDAuN8KgwqDCoCA5MzkgPGJy
Pg0KMTY1Mzc5ODHCoMKgwqAgMS40wqDCoCA2Ny4ywqAgYWRhMnAxPGJyPg0KwqDCoMKgwqAgMsKg
wqAgMjUyNcKgwqDCoMKgwqAgMMKgwqDCoMKgwqAgMMKgwqDCoCAwLjDCoMKgIDEwOTjCoMKgIDU5
MTTCoMKgwqAgMC44wqDCoCAxMzk5IDxicj4NCjE2MDIxNjE1wqDCoMKgIDEuMcKgwqAgNzkuM8Kg
IGFkYTM8YnI+DQrCoMKgwqDCoCAywqDCoCAxOTQ5wqDCoMKgwqDCoCAwwqDCoMKgwqDCoCAwwqDC
oMKgIDAuMMKgwqAgMTA5NMKgwqAgNTkyNsKgwqDCoCAxLjLCoMKgwqAgODI3IDxicj4NCjExMjY3
MjAwwqDCoMKgIDEuOMKgwqAgNzguOMKgIGFkYTBwMTxicj4NCsKgwqDCoCAxMsKgwqAgMjQ3McKg
wqDCoMKgwqAgMMKgwqDCoMKgwqAgMMKgwqDCoCAwLjDCoMKgIDEwMTjCoMKgIDUzOTnCoMKgwqAg
MS4wwqDCoCAxNDI1IDxicj4NCjE1Mzk1NTY2wqDCoMKgIDEuMcKgwqAgODAuNcKgIGFkYTU8YnI+
DQrCoMKgwqDCoCAywqDCoCAyNTI1wqDCoMKgwqDCoCAwwqDCoMKgwqDCoCAwwqDCoMKgIDAuMMKg
wqAgMTA5OMKgwqAgNTkxNMKgwqDCoCAwLjjCoMKgIDEzOTkgPGJyPg0KMTYwMjE2MTXCoMKgwqAg
MS4xwqDCoCA3OS4zwqAgYWRhM3AxPGJyPg0KwqDCoMKgIDEywqDCoCAyNDcxwqDCoMKgwqDCoCAw
wqDCoMKgwqDCoCAwwqDCoMKgIDAuMMKgwqAgMTAxOMKgwqAgNTM5OcKgwqDCoCAxLjDCoMKgIDE0
MjUgPGJyPg0KMTUzOTU1NjbCoMKgwqAgMS4xwqDCoCA4MC41wqAgYWRhNXAxPGJyPg0KPGJyPg0K
VGhlIHVsdGltYXRlIHByb2JsZW0gaXMgdGhhdCBhZnRlciBhIHdoaWxlIHdpdGggYSBsb3Qgb2Yg
d3JpdGVzLCB0aGUgPGJyPg0KZGlzayBwZXJmb3JtYW5jZSB3aWxsIGJlIHRvYXN0IHVudGlsIEkg
ZG8gYSBtYW51YWwgdHJpbSAtZiBvZiB0aGUgZGlzayA8YnI+DQo6KMKgwqAgdGhpcyBpcyBtb3N0
IG5vdGFibGUgb24gY29uc3VtZXIgV0QgU1NEcy7CoCBJIGhhdmVudCBkb25lIGFueSA8YnI+DQpl
eHRlbnNpdmUgdGVzdHMgd2l0aCBTYW1zdW5nIFNTRHMgdG8gc2VlIGlmIHRoZXJlIGFyZSBwZXJm
b3JtYW5jZSA8YnI+DQpwZW5hbHRpZXMgb3Igbm90LiBJdCBtaWdodCBiZSB0aGF0IHRoZXkgYXJl
IGp1c3QgYmV0dGVyIGF0IG1hc2tpbmcgdGhlIDxicj4NCnByb2JsZW0uwqAgSSBkb250IHNlZSB0
aGUgc2FtZSBpc3N1ZSB3aXRoIFpGUyBvbiBMaW51eCB3aXRoIHRoZSBzYW1lIDxicj4NCmRpc2tz
IC8gaGFyZHdhcmU8YnI+PC9ibG9ja3F1b3RlPjxkaXY+PGJyPjwvZGl2PjxkaXY+V2hlbiB0cmlt
cyBhcmUgZmFzdCwgeW91IHdhbnQgdG8gc2VuZCB0aGVtIHRvIHRoZSBkcml2ZSBhcyBzb29uIGFz
IHlvdTwvZGl2PjxkaXY+a25vdyB0aGUgYmxvY2tzIGFyZSBmcmVlZC4gVUZTIGFsd2F5cyBkb2Vz
IHRoaXMgKGlmIHRyaW0gaXMgZW5hYmxlZCBhdCBhbGwpLjwvZGl2PjxkaXY+WkZTIGhhcyBhIGxv
dCBvZiBrbm9icyB0byBjb250cm9sIHdoZW4gLyBob3cgLyBpZiB0aGlzIGlzIGRvbmUuPC9kaXY+
PGRpdj48YnI+PC9kaXY+PGRpdj48cHJlIGlkPSJnbWFpbC1saW5lMSI+dmZzLnpmcy52ZGV2LnRy
aW1fbWluX2FjdGl2ZTogMTxicj52ZnMuemZzLnZkZXYudHJpbV9tYXhfYWN0aXZlOiAyPGJyPnZm
cy56ZnMudHJpbS5xdWV1ZV9saW1pdDogMTA8YnI+dmZzLnpmcy50cmltLnR4Z19iYXRjaDogMzI8
YnI+dmZzLnpmcy50cmltLm1ldGFzbGFiX3NraXA6IDA8YnI+dmZzLnpmcy50cmltLmV4dGVudF9i
eXRlc19taW46IDMyNzY4PGJyPnZmcy56ZnMudHJpbS5leHRlbnRfYnl0ZXNfbWF4OiAxMzQyMTc3
Mjg8YnI+dmZzLnpmcy5sMmFyYy50cmltX2FoZWFkOiAwPGJyPjxicj48L3ByZT48cHJlIGlkPSJn
bWFpbC1saW5lMSI+SSYjMzk7dmUgbm90IHRyaWVkIHRvIHR1bmUgdGhlc2UgaW4gdGhlIHBhc3Qs
IGJ1dCB5b3UgY2FuIHNlZSBob3cgdGhleSBhZmZlY3QgdGhpbmdzLjxicj48L3ByZT48cHJlIGlk
PSJnbWFpbC1saW5lMSI+PGJyPldhcm5lcjwvcHJlPjwvZGl2PjxkaXY+wqA8L2Rpdj48YmxvY2tx
dW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7
Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+
DQpJIGhhdmUgYW4gb3BlbiBQUiBpbiA8YnI+DQo8YSBocmVmPSJodHRwczovL2J1Z3MuZnJlZWJz
ZC5vcmcvYnVnemlsbGEvc2hvd19idWcuY2dpP2lkPTI3Nzk5MiIgcmVsPSJub3JlZmVycmVyIiB0
YXJnZXQ9Il9ibGFuayI+aHR0cHM6Ly9idWdzLmZyZWVic2Qub3JnL2J1Z3ppbGxhL3Nob3dfYnVn
LmNnaT9pZD0yNzc5OTI8L2E+IHRoYXQgSSB0aGluayA8YnI+DQptaWdodCBhY3R1YWxseSBoYXZl
IDIgc2VwYXJhdGUgcHJvYmxlbXMuPGJyPg0KPGJyPg0KwqDCoMKgwqAgLS0tTWlrZTxicj4NCjxi
cj4NCjxicj4NCiZndDsmZ3Q7IGUuZy4gaGVyZSB3YXMgb25lIGRpc2sgaW4gdGhlIHBvb2wgdGhh
dCB3YXMgdGFraW5nIGEgbG9uZyB0aW1lIGZvciA8YnI+DQomZ3Q7Jmd0OyBlYWNoIHpwb29sIHRy
aW08YnI+DQomZ3Q7Jmd0Ozxicj4NCiZndDsmZ3Q7ICMgdGltZSB0cmltIC1mIC9kZXYvYWRhMTxi
cj4NCiZndDsmZ3Q7IHRyaW0gL2Rldi9hZGExIG9mZnNldCAwIGxlbmd0aCAxMDAwMjA0ODg2MDE2
PGJyPg0KJmd0OyZndDsgMC4wMDB1IDAuMDU3cyAxOjI5LjMzIDAuMCXCoMKgwqDCoMKgIDUrMTg0
ayAwKzBpbyAwcGYrMHc8YnI+DQomZ3Q7Jmd0OyBhbmQgdGhlbiBpZiBJIHJlLXJ1biBpdDxicj4N
CiZndDsmZ3Q7ICPCoCB0aW1lIHRyaW0gLWYgL2Rldi9hZGExPGJyPg0KJmd0OyZndDsgdHJpbSAv
ZGV2L2FkYTEgb2Zmc2V0IDAgbGVuZ3RoIDEwMDAyMDQ4ODYwMTY8YnI+DQomZ3Q7Jmd0OyAwLjAw
MHUgMC4wNTJzIDA6MDQuMTUgMS4yJcKgwqDCoMKgwqAgMSs1MmsgMCswaW8gMHBmKzB3PGJyPg0K
Jmd0OyZndDs8YnI+DQomZ3Q7Jmd0OyA5MCBzZWNvbmRzIGFuZCB0aGVuIDQgc2Vjb25kcyBhZnRl
ciB0aGF0Ljxicj4NCiZndDsmZ3Q7PGJyPg0KJmd0Ozxicj4NCiZndDsgLU1hdHRoZXc8YnI+DQom
Z3Q7PGJyPg0KPGJyPg0KPC9ibG9ja3F1b3RlPjwvZGl2PjwvZGl2Pg0K
--000000000000165b720617793f29--

From nobody Thu May  2 14:19:51 2024
X-Original-To: stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VVbfx1HHjz5JPGS
	for <stable@mlmmj.nyi.freebsd.org>; Thu,  2 May 2024 14:19:53 +0000 (UTC)
	(envelope-from mike@sentex.net)
Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VVbfx0rwPz4JfD
	for <stable@freebsd.org>; Thu,  2 May 2024 14:19:53 +0000 (UTC)
	(envelope-from mike@sentex.net)
Authentication-Results: mx1.freebsd.org;
	none
Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19])
	by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 442EJqoe084754
	(version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL);
	Thu, 2 May 2024 10:19:52 -0400 (EDT)
	(envelope-from mike@sentex.net)
Received: from [IPV6:2607:f3e0:0:4:bd4c:f42f:eabc:4f29] ([IPv6:2607:f3e0:0:4:bd4c:f42f:eabc:4f29])
	by pyroxene2a.sentex.ca (8.18.1/8.15.2) with ESMTPS id 442EJp1N052312
	(version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO);
	Thu, 2 May 2024 10:19:51 -0400 (EDT)
	(envelope-from mike@sentex.net)
Content-Type: multipart/alternative;
 boundary="------------EL2hQbxvGZPICZTUMLYILZPQ"
Message-ID: <a6a53e96-a8ee-48c0-ae76-1e4150679f13@sentex.net>
Date: Thu, 2 May 2024 10:19:51 -0400
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: how to tell if TRIM is working
To: Warner Losh <imp@bsdimp.com>
Cc: Matthew Grooms <mgrooms@shrew.net>, stable@freebsd.org
References: <5e1b5097-c1c0-4740-a491-63c709d01c25@sentex.net>
 <67721332-fa1d-4b3c-aa57-64594ad5d77a@shrew.net>
 <77e203b3-c555-408b-9634-c452cb3a57ac@sentex.net>
 <CANCZdfqx_vhNb2BukbM0bxrf8NH_9sXPKW+Uf=LdoXjw_2w=Dg@mail.gmail.com>
Content-Language: en-US
From: mike tancsa <mike@sentex.net>
Autocrypt: addr=mike@sentex.net; keydata=
 xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP
 xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s
 +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW
 yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH
 VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB
 AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs
 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W
 rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS
 j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ
 cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R
 yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw
 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk
 AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1
 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt
 RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO
 l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7
 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB
 CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8
 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM
 aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X
 rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB
 Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0
 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg==
In-Reply-To: <CANCZdfqx_vhNb2BukbM0bxrf8NH_9sXPKW+Uf=LdoXjw_2w=Dg@mail.gmail.com>
X-Scanned-By: MIMEDefang 2.86
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]
X-Rspamd-Queue-Id: 4VVbfx0rwPz4JfD

This is a multi-part message in MIME format.
--------------EL2hQbxvGZPICZTUMLYILZPQ
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 5/2/2024 10:16 AM, Warner Losh wrote:
>
> When trims are fast, you want to send them to the drive as soon as you
> know the blocks are freed. UFS always does this (if trim is enabled at 
> all).
> ZFS has a lot of knobs to control when / how / if this is done.
>
> vfs.zfs.vdev.trim_min_active: 1
> vfs.zfs.vdev.trim_max_active: 2
> vfs.zfs.trim.queue_limit: 10
> vfs.zfs.trim.txg_batch: 32
> vfs.zfs.trim.metaslab_skip: 0
> vfs.zfs.trim.extent_bytes_min: 32768
> vfs.zfs.trim.extent_bytes_max: 134217728
> vfs.zfs.l2arc.trim_ahead: 0
>
> I've not tried to tune these in the past, but you can see how they affect things.
>
Thanks Warner, I will try and play around with these values to see if 
they impact things.Â  BTW, do you know what / why things would be 
"skipped" during trim events ?

kstat.zfs.zrootoffs.misc.iostats.trim_bytes_failed: 0
kstat.zfs.zrootoffs.misc.iostats.trim_extents_failed: 0
kstat.zfs.zrootoffs.misc.iostats.trim_bytes_skipped: 5968330752
kstat.zfs.zrootoffs.misc.iostats.trim_extents_skipped: 503986
kstat.zfs.zrootoffs.misc.iostats.trim_bytes_written: 181593186304
kstat.zfs.zrootoffs.misc.iostats.trim_extents_written: 303115


--------------EL2hQbxvGZPICZTUMLYILZPQ
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">On 5/2/2024 10:16 AM, Warner Losh
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CANCZdfqx_vhNb2BukbM0bxrf8NH_9sXPKW+Uf=LdoXjw_2w=Dg@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">
        <div class="gmail_quote"><br>
          <div>When trims are fast, you want to send them to the drive
            as soon as you</div>
          <div>know the blocks are freed. UFS always does this (if trim
            is enabled at all).</div>
          <div>ZFS has a lot of knobs to control when / how / if this is
            done.</div>
          <div><br>
          </div>
          <div>
            <pre id="gmail-line1">vfs.zfs.vdev.trim_min_active: 1
vfs.zfs.vdev.trim_max_active: 2
vfs.zfs.trim.queue_limit: 10
vfs.zfs.trim.txg_batch: 32
vfs.zfs.trim.metaslab_skip: 0
vfs.zfs.trim.extent_bytes_min: 32768
vfs.zfs.trim.extent_bytes_max: 134217728
vfs.zfs.l2arc.trim_ahead: 0

</pre>
            <pre id="gmail-line1">I've not tried to tune these in the past, but you can see how they affect things.
</pre>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <p>Thanks Warner, I will try and play around with these values to
      see if they impact things.Â  BTW, do you know what / why things
      would be "skipped" during trim events ?</p>
    <p>kstat.zfs.zrootoffs.misc.iostats.trim_bytes_failed: 0<br>
      kstat.zfs.zrootoffs.misc.iostats.trim_extents_failed: 0<br>
      kstat.zfs.zrootoffs.misc.iostats.trim_bytes_skipped: 5968330752<br>
      kstat.zfs.zrootoffs.misc.iostats.trim_extents_skipped: 503986<br>
      kstat.zfs.zrootoffs.misc.iostats.trim_bytes_written: 181593186304<br>
      kstat.zfs.zrootoffs.misc.iostats.trim_extents_written: 303115<br>
    </p>
    <p><br>
    </p>
  </body>
</html>

--------------EL2hQbxvGZPICZTUMLYILZPQ--

From nobody Thu May  2 14:34:22 2024
X-Original-To: stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VVbzx49vTz5JQKB
	for <stable@mlmmj.nyi.freebsd.org>; Thu,  2 May 2024 14:34:37 +0000 (UTC)
	(envelope-from wlosh@bsdimp.com)
Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VVbzw2q7mz4L75
	for <stable@freebsd.org>; Thu,  2 May 2024 14:34:36 +0000 (UTC)
	(envelope-from wlosh@bsdimp.com)
Authentication-Results: mx1.freebsd.org;
	none
Received: by mail-ej1-x632.google.com with SMTP id a640c23a62f3a-a595c61553cso205928966b.1
        for <stable@freebsd.org>; Thu, 02 May 2024 07:34:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1714660474; x=1715265274; darn=freebsd.org;
        h=cc:to:subject:message-id:date:from:in-reply-to:references
         :mime-version:from:to:cc:subject:date:message-id:reply-to;
        bh=6vum0Fg+BB44P1zj4neZ+AEa+Sfdmg+TqL6+cdoQFrU=;
        b=llTuo2VXDQz+xuJT9mf6SbYdy0CjwIXc1DS+37/6RWXLJ0uydEXW8EZ6FGWRiwktNw
         yCWvmz/+0glaisRfZYPsX1w8RXm7lZ5h7Q0OlHV08TbVBv+Zv/boaCtECcFDggy7kLEu
         +468+N3Du0tw6RrYIjxMn/I6IlpvFLfzlbytESDfrT/SaRlqjbOUop/fMw0TPPLbtk2N
         VQx3y9/nFCgMD7PcPWkhzOElAxgzc0iYbHEZJaQ3IlbL2SsNW30+PIMF8szxNLzS+mJe
         yoNP0Gv59WNTEX2DSuR88PrLGaOBaL9xr1+km//I8+ploIyeB7M/zjnzxqpYh/cQWKQU
         pDeg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1714660474; x=1715265274;
        h=cc:to:subject:message-id:date:from:in-reply-to:references
         :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
         :reply-to;
        bh=6vum0Fg+BB44P1zj4neZ+AEa+Sfdmg+TqL6+cdoQFrU=;
        b=U1QuE4q47sD6GGsaSEieUbOYC9+Zjk/2MhbfQAv7geI+oxJJ3c4RzoVWVmPiHpNdwE
         Ord4jhzhuVAI70evC3a9Y8FVXzurTWluqTlARGxAhOnYtd7RHDIU1SpxbNRiNX+covtq
         3+PMC2Duu676sbbzC8w8K+4wjmzcybYyjND2hdcMCjVqkbM9udyYByLRw3JST5rsK19M
         ezvMUdQfh34FAH7lxWYAoRRlLG7L/ngM0g/4Ss5X9z9nyB6+VJjSYbGhO3te7+u8AW+B
         +lZeAd0Ga5WUnwDPoan0RHnIgmRCiD6jEPISTG58FzZWgYRtXnRNPw7A/Wf7J+ivVOEC
         W0Mw==
X-Forwarded-Encrypted: i=1; AJvYcCVRyT1Iwu0a+rDiSrF7n9mnCGZHwOhRllpnWLo21MdS5+KaPGXqly2LF96KtYHwwgQQzAL/fGU6mw1vSD0poLHBdK4=
X-Gm-Message-State: AOJu0YzF0/JMVjmbXD14PPtCsXJFqUSjCEDD6DqQQi7f+SC4AsdiVMj4
	Vl7XDvXRNSq3HaPjiuJqzRNsXi4SdmNB8XjELEVA00PuRjOPw4byWQ6U9EaHg6G2YOo+Cr+AHlX
	G0k2+u6384PHucTUbkubEXVFn8PbRl2GApm+rKA==
X-Google-Smtp-Source: AGHT+IH+Edc9iXw+w21tXkKYgKc9r5bGogAVxVmF4um3gvk4j2CW+U3blMHFSdicW99WKDFBcBYiPNGzV5TG6rTqgII=
X-Received: by 2002:a17:906:449:b0:a58:994c:8c6a with SMTP id
 e9-20020a170906044900b00a58994c8c6amr3858381eja.26.1714660474011; Thu, 02 May
 2024 07:34:34 -0700 (PDT)
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
References: <5e1b5097-c1c0-4740-a491-63c709d01c25@sentex.net>
 <67721332-fa1d-4b3c-aa57-64594ad5d77a@shrew.net> <77e203b3-c555-408b-9634-c452cb3a57ac@sentex.net>
 <CANCZdfqx_vhNb2BukbM0bxrf8NH_9sXPKW+Uf=LdoXjw_2w=Dg@mail.gmail.com> <a6a53e96-a8ee-48c0-ae76-1e4150679f13@sentex.net>
In-Reply-To: <a6a53e96-a8ee-48c0-ae76-1e4150679f13@sentex.net>
From: Warner Losh <imp@bsdimp.com>
Date: Thu, 2 May 2024 08:34:22 -0600
Message-ID: <CANCZdfqa0=1kJQYpbZQ3z2xdxt8x6L7iYJSjQUc7SGUap8KP5Q@mail.gmail.com>
Subject: Re: how to tell if TRIM is working
To: mike tancsa <mike@sentex.net>
Cc: Matthew Grooms <mgrooms@shrew.net>, stable@freebsd.org
Content-Type: multipart/alternative; boundary="00000000000076dabe06177980f1"
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]
X-Rspamd-Queue-Id: 4VVbzw2q7mz4L75

--00000000000076dabe06177980f1
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Thu, May 2, 2024 at 8:19=E2=80=AFAM mike tancsa <mike@sentex.net> wrote:

> On 5/2/2024 10:16 AM, Warner Losh wrote:
>
>
> When trims are fast, you want to send them to the drive as soon as you
> know the blocks are freed. UFS always does this (if trim is enabled at
> all).
> ZFS has a lot of knobs to control when / how / if this is done.
>
> vfs.zfs.vdev.trim_min_active: 1
> vfs.zfs.vdev.trim_max_active: 2
> vfs.zfs.trim.queue_limit: 10
> vfs.zfs.trim.txg_batch: 32
> vfs.zfs.trim.metaslab_skip: 0
> vfs.zfs.trim.extent_bytes_min: 32768
> vfs.zfs.trim.extent_bytes_max: 134217728
> vfs.zfs.l2arc.trim_ahead: 0
>
>
> I've not tried to tune these in the past, but you can see how they affect=
 things.
>
>
> Thanks Warner, I will try and play around with these values to see if the=
y
> impact things.  BTW, do you know what / why things would be "skipped"
> during trim events ?
>
> kstat.zfs.zrootoffs.misc.iostats.trim_bytes_failed: 0
> kstat.zfs.zrootoffs.misc.iostats.trim_extents_failed: 0
> kstat.zfs.zrootoffs.misc.iostats.trim_bytes_skipped: 5968330752
> kstat.zfs.zrootoffs.misc.iostats.trim_extents_skipped: 503986
> kstat.zfs.zrootoffs.misc.iostats.trim_bytes_written: 181593186304
> kstat.zfs.zrootoffs.misc.iostats.trim_extents_written: 303115
>

A quick look at the code suggests that it is when the extent to be trimmed
is smaller than the extent_bytes_min parameter.

The minimum seems to be a trade off between too many trims to the drive and
making sure that the trims that you do send are maximally effective. By
specifying a smaller size, you'll be freeing up more holes in the
underlying NAND blocks. In some drives, this triggers more data copying
(and more write amp), so you want to set it a bit higher for those. In
other drivers, it improves the efficiency of the GC algorithm, allowing
each underlying block groomed to recover more space for future writes. In
the past, I've found that ZFS' defaults are decent for 2018ish level of
SATA SSDs, but a bit too trim avoidy for newer nvme drives, even the cheap
consumer ones. Though that's just a coarse generalization from my
buildworld workload. Other work loads will have other data patterns, ymmv,
so you need to measure it.

Another way to get statistics, one that I've not been able to measure a
slowdown from, is to enable CAM_IOSCHED_DYNAMIC. Then you get a lot more
statistics about the I/Os in the system, including latency measurements. In
theory, that also allows one to traffic shape the trims to the drive, but
I've had only limited success with that and haven't had the time to make it
a lot better.

Warner

--00000000000076dabe06177980f1
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Thu, May 2, 2024 at 8:19=E2=80=AFA=
M mike tancsa &lt;<a href=3D"mailto:mike@sentex.net">mike@sentex.net</a>&gt=
; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px=
 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>

 =20
   =20
 =20
  <div>
    <div>On 5/2/2024 10:16 AM, Warner Losh
      wrote:<br>
    </div>
    <blockquote type=3D"cite">
     =20
      <div dir=3D"ltr">
        <div class=3D"gmail_quote"><br>
          <div>When trims are fast, you want to send them to the drive
            as soon as you</div>
          <div>know the blocks are freed. UFS always does this (if trim
            is enabled at all).</div>
          <div>ZFS has a lot of knobs to control when / how / if this is
            done.</div>
          <div><br>
          </div>
          <div>
            <pre id=3D"m_8001559206439009662gmail-line1">vfs.zfs.vdev.trim_=
min_active: 1
vfs.zfs.vdev.trim_max_active: 2
vfs.zfs.trim.queue_limit: 10
vfs.zfs.trim.txg_batch: 32
vfs.zfs.trim.metaslab_skip: 0
vfs.zfs.trim.extent_bytes_min: 32768
vfs.zfs.trim.extent_bytes_max: 134217728
vfs.zfs.l2arc.trim_ahead: 0

</pre>
            <pre id=3D"m_8001559206439009662gmail-line1">I&#39;ve not tried=
 to tune these in the past, but you can see how they affect things.
</pre>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <p>Thanks Warner, I will try and play around with these values to
      see if they impact things.=C2=A0 BTW, do you know what / why things
      would be &quot;skipped&quot; during trim events ?</p>
    <p>kstat.zfs.zrootoffs.misc.iostats.trim_bytes_failed: 0<br>
      kstat.zfs.zrootoffs.misc.iostats.trim_extents_failed: 0<br>
      kstat.zfs.zrootoffs.misc.iostats.trim_bytes_skipped: 5968330752<br>
      kstat.zfs.zrootoffs.misc.iostats.trim_extents_skipped: 503986<br>
      kstat.zfs.zrootoffs.misc.iostats.trim_bytes_written: 181593186304<br>
      kstat.zfs.zrootoffs.misc.iostats.trim_extents_written: 303115</p></di=
v></blockquote><div><br></div><div>A quick look at the code suggests that i=
t is when the extent to be trimmed is smaller than the extent_bytes_min par=
ameter.</div><div><br></div><div>The minimum seems to be a trade off betwee=
n too many trims to the drive and making sure that the trims that you do se=
nd are maximally effective. By specifying a smaller size, you&#39;ll be fre=
eing up more holes in the underlying NAND blocks. In some drives, this trig=
gers more data copying (and more write amp), so you want to set it a bit hi=
gher for those. In other drivers, it improves the efficiency of the GC algo=
rithm, allowing each underlying block groomed to recover more space for fut=
ure writes. In the past, I&#39;ve found that ZFS&#39; defaults are decent f=
or 2018ish level of SATA SSDs, but a bit too trim avoidy for newer nvme dri=
ves, even the cheap consumer ones. Though that&#39;s just a coarse generali=
zation from my buildworld workload. Other work loads will have other data p=
atterns, ymmv, so you need to measure it.<br></div><div><br></div><div>Anot=
her way to get statistics, one that I&#39;ve not been able to measure a slo=
wdown from, is to enable CAM_IOSCHED_DYNAMIC. Then you get a lot more stati=
stics about the I/Os in the system, including latency measurements. In theo=
ry, that also allows one to traffic shape the trims to the drive, but I&#39=
;ve had only limited success with that and haven&#39;t had the time to make=
 it a lot better.</div><div><br></div><div>Warner</div><div><br></div></div=
></div>

--00000000000076dabe06177980f1--

From nobody Fri May  3 14:17:51 2024
X-Original-To: stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VWCZL6R70z5K5tK
	for <stable@mlmmj.nyi.freebsd.org>; Fri,  3 May 2024 14:18:02 +0000 (UTC)
	(envelope-from uspensky@x-art.ru)
Received: from x-art.ru (charibdis.x-art.ru [80.70.228.55])
	by mx1.freebsd.org (Postfix) with ESMTP id 4VWCZK3H8vz4rb0
	for <stable@freebsd.org>; Fri,  3 May 2024 14:18:01 +0000 (UTC)
	(envelope-from uspensky@x-art.ru)
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of uspensky@x-art.ru designates 80.70.228.55 as permitted sender) smtp.mailfrom=uspensky@x-art.ru
Received: from gw-old.x-art.ru (gw-old.x-art.ru [192.168.172.252])
	by mta.x-art.ru (Postfix) with ESMTP id 6A9D71BF425;
	Fri,  3 May 2024 17:17:51 +0300 (MSK)
Date: Fri, 3 May 2024 17:17:51 +0300 (MSK)
From: Antony Uspensky <uspensky@x-art.ru>
X-X-Sender: aiu@gw-old.x-art.ru
To: Warner Losh <imp@bsdimp.com>
cc: mike tancsa <mike@sentex.net>, Matthew Grooms <mgrooms@shrew.net>, 
    stable@freebsd.org
Subject: Re: how to tell if TRIM is working
Message-ID: <alpine.BSF.2.00.2405031711340.23742@gw-old.x-art.ru>
User-Agent: Alpine 2.00 (BSF 1167 2008-08-23)
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
X-Spamd-Bar: -
X-Spamd-Result: default: False [-1.15 / 15.00];
	FAKE_REPLY(1.00)[];
	NEURAL_HAM_LONG(-0.97)[-0.968];
	NEURAL_HAM_MEDIUM(-0.92)[-0.921];
	R_SPF_ALLOW(-0.20)[+ip4:80.70.228.55];
	NEURAL_HAM_SHORT(-0.16)[-0.164];
	RCVD_NO_TLS_LAST(0.10)[];
	MIME_GOOD(-0.10)[text/plain];
	ONCE_RECEIVED(0.10)[];
	TO_DN_SOME(0.00)[];
	MIME_TRACE(0.00)[0:+];
	ARC_NA(0.00)[];
	ASN(0.00)[asn:20807, ipnet:80.70.224.0/20, country:RU];
	RCVD_COUNT_ONE(0.00)[1];
	FROM_HAS_DN(0.00)[];
	MLMMJ_DEST(0.00)[stable@freebsd.org];
	FROM_EQ_ENVFROM(0.00)[];
	R_DKIM_NA(0.00)[];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	MID_RHS_MATCH_FROMTLD(0.00)[];
	DMARC_NA(0.00)[x-art.ru];
	RCPT_COUNT_THREE(0.00)[4]
X-Rspamd-Queue-Id: 4VWCZK3H8vz4rb0

On Thu, 2 May 2024, Warner Losh wrote:

> ZFS has a lot of knobs to control when / how / if this is done.

trim(8) works on device level.
If you use ZFS, you rather need zpool-trim(8), which works on pool level.
A.

From nobody Fri May  3 14:33:57 2024
X-Original-To: stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VWCwy6YFzz5K74b
	for <stable@mlmmj.nyi.freebsd.org>; Fri,  3 May 2024 14:34:10 +0000 (UTC)
	(envelope-from wlosh@bsdimp.com)
Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VWCwy4js9z4tsl
	for <stable@freebsd.org>; Fri,  3 May 2024 14:34:10 +0000 (UTC)
	(envelope-from wlosh@bsdimp.com)
Authentication-Results: mx1.freebsd.org;
	none
Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-a5557e3ebcaso526684066b.1
        for <stable@freebsd.org>; Fri, 03 May 2024 07:34:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1714746849; x=1715351649; darn=freebsd.org;
        h=cc:to:subject:message-id:date:from:in-reply-to:references
         :mime-version:from:to:cc:subject:date:message-id:reply-to;
        bh=jxvVqoMtwxGZYEEPj/oMaaDphexeY1fV7xt/Kx/rEhk=;
        b=rOc4V32V9+dR2Clv/27UdluEGjgwFkm3Lj+HIJVshoxbZ/lYGSxnKyfP/NQIRYEKOx
         K9YU9Jo5RKwLIwXpiKMgf4t04KNpJd1A4J7CNX+uQOrra2E4GAwqGHixZLATMMPnHANU
         aO+sJmyp07HuDVoeMjhNb0TZxZ8ZDQmpDXQltkE0Ml4c9FVDAfo7RMSrQPgL8rwLcHN2
         hrysU9xDUz7hOwEsjLYJgLicWp1KtvSWSpX7nL4kIxQmp+/4IzIWup6EsqIIEKfibZ04
         wCtVMEMD+g7N8gsyW60rGSeFqjnWZex2B+ndrO9pOmTH7oQ5oekQfvPcj53kVsllMOeu
         UryQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1714746849; x=1715351649;
        h=cc:to:subject:message-id:date:from:in-reply-to:references
         :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
         :reply-to;
        bh=jxvVqoMtwxGZYEEPj/oMaaDphexeY1fV7xt/Kx/rEhk=;
        b=FxcmMV9Db1J5dF8oXztZ+JHW64+xjqQcW0ZRnz3XpAujLF0b66j3COQx5osxpEkhRP
         bw7JAvknjTJVUFVnmgSTTDQ83sZQDqB2e2E1WkIKUf32GmjrPgVJ90UUt/r7AD5euold
         ZW54CizeXkThxn7HMjlj+0bQ9mKVYYgJe5JeIXWizB3IbVFZDR5cDIkkqGvrSsyh8bxZ
         77O+VM5zTxxtjAbTIycNfGgpvZXfQ/9GxpeqVwsTUiFINt5znUoDopzc6TuDC66Oz3J0
         EGgjRzZApfy7aKundw/O+mnQ966i+84jKPcaSp4RTo75mt84hwmLxHA9Oxml5lt8pi5t
         ToVw==
X-Forwarded-Encrypted: i=1; AJvYcCVjlE0XS67gE59DQqgBnQ9zm8DtySZSAua5MR0mk9k+taSilWhyHlzz4cTpB9NUgMG7uKjcsrwexxUk+65G6O1L9vg=
X-Gm-Message-State: AOJu0YyHGPLiSzxQ4WWJeavX4LMA7JIV1QziLEcgqZMHx3hoDNQXjJrx
	RvCY6k7328fmY0X44v+DzA8GVuGcMDpzG35KMa9epHLnF+Nce3bTZk5XjCNRDfIz+mOGntdw7bR
	V5rFCIEnJYYPBjEf1lhOy8TNQq2NMl9GXLmzF69/rTaumjQV2
X-Google-Smtp-Source: AGHT+IFazotMmAiSA+57DlZEGGthZLFeMAkOMzfpGugJjLm0/ziUnijNSD+BqKNxyG73IZ4ID1SX27JCpOLyluAI1yE=
X-Received: by 2002:a17:906:3b4b:b0:a58:e8cf:664f with SMTP id
 h11-20020a1709063b4b00b00a58e8cf664fmr4686140ejf.23.1714746848985; Fri, 03
 May 2024 07:34:08 -0700 (PDT)
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
References: <alpine.BSF.2.00.2405031711340.23742@gw-old.x-art.ru>
In-Reply-To: <alpine.BSF.2.00.2405031711340.23742@gw-old.x-art.ru>
From: Warner Losh <imp@bsdimp.com>
Date: Fri, 3 May 2024 08:33:57 -0600
Message-ID: <CANCZdfoshnihAxQKW5gFF6b_B_vucZycFAM4NGp2YbYW4r+D5A@mail.gmail.com>
Subject: Re: how to tell if TRIM is working
To: Antony Uspensky <uspensky@x-art.ru>
Cc: mike tancsa <mike@sentex.net>, Matthew Grooms <mgrooms@shrew.net>, stable@freebsd.org
Content-Type: multipart/alternative; boundary="000000000000d05ed606178d9c7e"
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]
X-Rspamd-Queue-Id: 4VWCwy4js9z4tsl

--000000000000d05ed606178d9c7e
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, May 3, 2024 at 8:17=E2=80=AFAM Antony Uspensky <uspensky@x-art.ru> =
wrote:

> On Thu, 2 May 2024, Warner Losh wrote:
>
> > ZFS has a lot of knobs to control when / how / if this is done.
>
> trim(8) works on device level.
> If you use ZFS, you rather need zpool-trim(8), which works on pool level.
>

Ah yes. You need to enable ZFS trim. I'd misread trim as zfs trim. The
other trim destroys all the data
on that partition.

Warner

--000000000000d05ed606178d9c7e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Fri, May 3, 2024 at 8:17=E2=80=AFA=
M Antony Uspensky &lt;<a href=3D"mailto:uspensky@x-art.ru">uspensky@x-art.r=
u</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin=
:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"=
>On Thu, 2 May 2024, Warner Losh wrote:<br>
<br>
&gt; ZFS has a lot of knobs to control when / how / if this is done.<br>
<br>
trim(8) works on device level.<br>
If you use ZFS, you rather need zpool-trim(8), which works on pool level.<b=
r></blockquote><div><br></div><div>Ah yes. You need to enable ZFS trim. I&#=
39;d misread trim as zfs trim. The other trim destroys all the data</div><d=
iv>on that partition.<br></div><div><br></div><div>Warner <br></div></div><=
/div>

--000000000000d05ed606178d9c7e--

From nobody Fri May  3 15:16:58 2024
X-Original-To: stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VWDtN1Fz0z5K9tj
	for <stable@mlmmj.nyi.freebsd.org>; Fri,  3 May 2024 15:17:00 +0000 (UTC)
	(envelope-from uspensky@x-art.ru)
Received: from x-art.ru (charibdis.x-art.ru [80.70.228.55])
	by mx1.freebsd.org (Postfix) with ESMTP id 4VWDtM62k3z52MW
	for <stable@freebsd.org>; Fri,  3 May 2024 15:16:59 +0000 (UTC)
	(envelope-from uspensky@x-art.ru)
Authentication-Results: mx1.freebsd.org;
	none
Received: from gw-old.x-art.ru (gw-old.x-art.ru [192.168.172.252])
	by mta.x-art.ru (Postfix) with ESMTP id E07771BF33C;
	Fri,  3 May 2024 18:16:58 +0300 (MSK)
Date: Fri, 3 May 2024 18:16:58 +0300 (MSK)
From: Antony Uspensky <uspensky@x-art.ru>
X-X-Sender: aiu@gw-old.x-art.ru
To: Warner Losh <imp@bsdimp.com>
cc: mike tancsa <mike@sentex.net>, Matthew Grooms <mgrooms@shrew.net>, 
    stable@freebsd.org
Subject: Re: how to tell if TRIM is working
In-Reply-To: <CANCZdfoshnihAxQKW5gFF6b_B_vucZycFAM4NGp2YbYW4r+D5A@mail.gmail.com>
Message-ID: <alpine.BSF.2.00.2405031810440.23742@gw-old.x-art.ru>
References: <alpine.BSF.2.00.2405031711340.23742@gw-old.x-art.ru> <CANCZdfoshnihAxQKW5gFF6b_B_vucZycFAM4NGp2YbYW4r+D5A@mail.gmail.com>
User-Agent: Alpine 2.00 (BSF 1167 2008-08-23)
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:20807, ipnet:80.70.224.0/20, country:RU]
X-Rspamd-Queue-Id: 4VWDtM62k3z52MW

On Fri, 3 May 2024, Warner Losh wrote:

> Ah yes. You need to enable ZFS trim.

Do you mean autotrim property? No, better to run zpool trim manually.

> The other trim destroys all the data on that partition.

Not always. It sends TRIM subcommand.

A.

From nobody Fri May  3 15:20:26 2024
X-Original-To: stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VWDyd4scCz5KBQV
	for <stable@mlmmj.nyi.freebsd.org>; Fri,  3 May 2024 15:20:41 +0000 (UTC)
	(envelope-from wlosh@bsdimp.com)
Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VWDyc3pk1z535r
	for <stable@freebsd.org>; Fri,  3 May 2024 15:20:40 +0000 (UTC)
	(envelope-from wlosh@bsdimp.com)
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=bsdimp-com.20230601.gappssmtp.com header.s=20230601 header.b=NsbNKh1Z;
	dmarc=none;
	spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2a00:1450:4864:20::631) smtp.mailfrom=wlosh@bsdimp.com
Received: by mail-ej1-x631.google.com with SMTP id a640c23a62f3a-a59a0527241so18834666b.2
        for <stable@freebsd.org>; Fri, 03 May 2024 08:20:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1714749639; x=1715354439; darn=freebsd.org;
        h=cc:to:subject:message-id:date:from:in-reply-to:references
         :mime-version:from:to:cc:subject:date:message-id:reply-to;
        bh=MifgWEMo8ZXZg7mFMxRXyNGrVmSM7R2hfrXaLeO2EsY=;
        b=NsbNKh1Z24fTJWncULoHlAEQvhQn7F32qy6/KbB63fghOCOy3mQAgvEPpjFMzlAYIW
         ACbvsH7/cGc031VY99n5i2VGn1ejvfa0jvcdjXdPbI0ozImV6qcOQQjmS7YAm9ltEuc5
         9BybtFCc6tmaio11WpUDcOl5BIqShMbfH/bJ4qier2dq7fGV55moJtHenjlUZE+5JZRG
         aC8nqeEdDs3AHtDBWJ34Hf1RIxpU2vIveZA9JMO8XCdn3yLW6lE+47p+6BMGd6xHMZig
         3HZ86zFMrXzKT4mKMnQ5OTR4DyapwssNmtc8BGUJBIuupnETri1d7fWjsPVskwGIG76B
         A3TA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1714749639; x=1715354439;
        h=cc:to:subject:message-id:date:from:in-reply-to:references
         :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
         :reply-to;
        bh=MifgWEMo8ZXZg7mFMxRXyNGrVmSM7R2hfrXaLeO2EsY=;
        b=R153irzV7CspE7S1QH6ukSQ6A8UmrHewHO8otgEf/wiN946JVCIkEHwn5zXyU0l/JM
         Zg26fMcxBph97GoGr0MIC+yPli/eNLZPRND0A1KYJ9Uo0+plBq29u+SI407HCsEyT6MQ
         Bei2rTqYpaNCIIZUzSbT4Y+SMPPmvlZEg9kyNhHNrVFVFXGGBw8JZiKublXRiaKPyTFM
         jo92PM3cCWj4G0T22fX/gW5fnTRqSjLbsDUh8HQ08QNENIOUmWISAenyzkLxFsjwWx6y
         awoeAN+xJI9GWE+rI1bsk4RlOwAUfOKY51BsMuUgPY6r9Ops0L8tdZ0Js8HMJoaQe7v3
         DKTg==
X-Gm-Message-State: AOJu0YwmXfiyN5ywgowUnEF9EBD6ihZZ5ezGrrL6k90fplER2U3ZqbFu
	az1HOX9rr5l1ElBNvNTYhWETW8753umoABvN0OHSEcdP9L9Z18O0zTH3ZvXdXURAJNqx1aZ/Ysa
	qXH7Ub5kWFWkMA9rEd2RKPqbgAJ9a2nkiRQmAKOozhnUzieFl
X-Google-Smtp-Source: AGHT+IF/xNWtjobCU3zeBp0tuSjSaZrQtaLBvBup67wnV3teGGcTyA6KGOYq1nf8hMSzAvlv8rszFhqEbjmXyRdpfro=
X-Received: by 2002:a17:907:12c5:b0:a59:9a40:4086 with SMTP id
 vp5-20020a17090712c500b00a599a404086mr982611ejb.19.1714749638570; Fri, 03 May
 2024 08:20:38 -0700 (PDT)
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
References: <5e1b5097-c1c0-4740-a491-63c709d01c25@sentex.net> <67721332-fa1d-4b3c-aa57-64594ad5d77a@shrew.net>
In-Reply-To: <67721332-fa1d-4b3c-aa57-64594ad5d77a@shrew.net>
From: Warner Losh <imp@bsdimp.com>
Date: Fri, 3 May 2024 09:20:26 -0600
Message-ID: <CANCZdfp4vGW=6FrHfLwkUeck5c3TSbVSRwcxS4jqnZmfzNyaUA@mail.gmail.com>
Subject: Re: how to tell if TRIM is working
To: Matthew Grooms <mgrooms@shrew.net>
Cc: stable@freebsd.org
Content-Type: multipart/alternative; boundary="00000000000016188e06178e43f3"
X-Spamd-Bar: --
X-Spamd-Result: default: False [-2.99 / 15.00];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-0.99)[-0.994];
	FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com];
	R_DKIM_ALLOW(-0.20)[bsdimp-com.20230601.gappssmtp.com:s=20230601];
	MIME_GOOD(-0.10)[multipart/alternative,text/plain];
	ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	RCVD_COUNT_ONE(0.00)[1];
	MIME_TRACE(0.00)[0:+,1:+,2:~];
	ARC_NA(0.00)[];
	TO_DN_SOME(0.00)[];
	MISSING_XM_UA(0.00)[];
	R_SPF_NA(0.00)[no SPF record];
	DMARC_NA(0.00)[bsdimp.com];
	RCPT_COUNT_TWO(0.00)[2];
	FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com];
	FROM_HAS_DN(0.00)[];
	RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::631:from];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org];
	RCVD_TLS_LAST(0.00)[];
	MLMMJ_DEST(0.00)[stable@freebsd.org];
	DKIM_TRACE(0.00)[bsdimp-com.20230601.gappssmtp.com:+]
X-Rspamd-Queue-Id: 4VWDyc3pk1z535r

--00000000000016188e06178e43f3
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hey Matthew,

On Wed, May 1, 2024 at 2:25=E2=80=AFPM Matthew Grooms <mgrooms@shrew.net> w=
rote:

> On 5/1/24 14:38, mike tancsa wrote:
> > Kind of struggling to check if TRIM is actually working or not with my
> > SSDs on RELENG_14 in ZFS.
> >
> > On a pool that has almost no files on it (capacity at 0% out of 3TB),
> > should not
> >
> > zpool -w trim <pool> be almost instant after a couple of runs ?
> > Instead it seems to always take about 10min to complete.
> >
> > Looking at the stats,
> >
> > kstat.zfs.tortank1.misc.iostats.trim_bytes_failed: 0
> > kstat.zfs.tortank1.misc.iostats.trim_extents_failed: 0
> > kstat.zfs.tortank1.misc.iostats.trim_bytes_skipped: 2743435264
> > kstat.zfs.tortank1.misc.iostats.trim_extents_skipped: 253898
> > kstat.zfs.tortank1.misc.iostats.trim_bytes_written: 14835526799360
> > kstat.zfs.tortank1.misc.iostats.trim_extents_written: 1169158
> >
> > what and why are bytes being skipped ?
> >
> > One of the drives for example
> >
> >  sysctl -a kern.cam.ada.0
> > kern.cam.ada.0.trim_ticks: 0
> > kern.cam.ada.0.trim_goal: 0
> > kern.cam.ada.0.sort_io_queue: 0
> > kern.cam.ada.0.rotating: 0
> > kern.cam.ada.0.unmapped_io: 1
> > kern.cam.ada.0.flags:
> >
> 0x1be3bde<CAN_48BIT,CAN_FLUSHCACHE,CAN_NCQ,CAN_DMA,WAS_OTAG,CAN_TRIM,OPEN=
,SCTX_INIT,CAN_POWERMGT,CAN_DMA48,CAN_LOG,CAN_WCACHE,CAN_RAHEAD,PROBED,ANNO=
UNCED,DIRTY,PIM_ATA_EXT,UNMAPPEDIO>
> > kern.cam.ada.0.max_seq_zones: 0
> > kern.cam.ada.0.optimal_nonseq_zones: 0
> > kern.cam.ada.0.optimal_seq_zones: 0
> > kern.cam.ada.0.zone_support: None
> > kern.cam.ada.0.zone_mode: Not Zoned
> > kern.cam.ada.0.write_cache: -1
> > kern.cam.ada.0.read_ahead: -1
> > kern.cam.ada.0.trim_lbas: 7771432624
> > kern.cam.ada.0.trim_ranges: 371381
> > kern.cam.ada.0.trim_count: 310842
> > kern.cam.ada.0.delete_method: DSM_TRIM
> >
> > If I take one of the disks out of the pool and replace it with a
> > spare, and do a manual trim it seems to work
> >
> I had a hard time seeing evidence of this at the disk level while
> fiddling with TRIM recently. It appeared that at least some counters are
> driver and operation specific. For example, the da driver appears to
> update counters in some paths but not others. I assume that ada is
> different. There is a bug report for da, but haven't seen any feedback ..=
.
>
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277673
>
> You could try to run gstat with the -d flag during the time period when
> the delete operations are expected to occur. That should give you an
> idea of what's happening at the disk level in real time but may not
> offer more info than you're already seeing.
>

These changes looked good. My apologies for not noticing this sooner
(I think that our CC to the mailing list in bugzilla has stopped working,
so I didn't see the redirect). I've committed the changes, and queued them
to my stable-14 and stable-13 branches.

Thank you so much for your submission.

Warner

--00000000000016188e06178e43f3
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">Hey Matthew,<br></div><br><div class=3D"g=
mail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, May 1, 2024 at 2:=
25=E2=80=AFPM Matthew Grooms &lt;<a href=3D"mailto:mgrooms@shrew.net">mgroo=
ms@shrew.net</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" styl=
e=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddin=
g-left:1ex">On 5/1/24 14:38, mike tancsa wrote:<br>
&gt; Kind of struggling to check if TRIM is actually working or not with my=
 <br>
&gt; SSDs on RELENG_14 in ZFS.<br>
&gt;<br>
&gt; On a pool that has almost no files on it (capacity at 0% out of 3TB), =
<br>
&gt; should not<br>
&gt;<br>
&gt; zpool -w trim &lt;pool&gt; be almost instant after a couple of runs ? =
<br>
&gt; Instead it seems to always take about 10min to complete.<br>
&gt;<br>
&gt; Looking at the stats,<br>
&gt;<br>
&gt; kstat.zfs.tortank1.misc.iostats.trim_bytes_failed: 0<br>
&gt; kstat.zfs.tortank1.misc.iostats.trim_extents_failed: 0<br>
&gt; kstat.zfs.tortank1.misc.iostats.trim_bytes_skipped: 2743435264<br>
&gt; kstat.zfs.tortank1.misc.iostats.trim_extents_skipped: 253898<br>
&gt; kstat.zfs.tortank1.misc.iostats.trim_bytes_written: 14835526799360<br>
&gt; kstat.zfs.tortank1.misc.iostats.trim_extents_written: 1169158<br>
&gt;<br>
&gt; what and why are bytes being skipped ?<br>
&gt;<br>
&gt; One of the drives for example<br>
&gt;<br>
&gt; =C2=A0sysctl -a kern.cam.ada.0<br>
&gt; kern.cam.ada.0.trim_ticks: 0<br>
&gt; kern.cam.ada.0.trim_goal: 0<br>
&gt; kern.cam.ada.0.sort_io_queue: 0<br>
&gt; kern.cam.ada.0.rotating: 0<br>
&gt; kern.cam.ada.0.unmapped_io: 1<br>
&gt; kern.cam.ada.0.flags: <br>
&gt; 0x1be3bde&lt;CAN_48BIT,CAN_FLUSHCACHE,CAN_NCQ,CAN_DMA,WAS_OTAG,CAN_TRI=
M,OPEN,SCTX_INIT,CAN_POWERMGT,CAN_DMA48,CAN_LOG,CAN_WCACHE,CAN_RAHEAD,PROBE=
D,ANNOUNCED,DIRTY,PIM_ATA_EXT,UNMAPPEDIO&gt;<br>
&gt; kern.cam.ada.0.max_seq_zones: 0<br>
&gt; kern.cam.ada.0.optimal_nonseq_zones: 0<br>
&gt; kern.cam.ada.0.optimal_seq_zones: 0<br>
&gt; kern.cam.ada.0.zone_support: None<br>
&gt; kern.cam.ada.0.zone_mode: Not Zoned<br>
&gt; kern.cam.ada.0.write_cache: -1<br>
&gt; kern.cam.ada.0.read_ahead: -1<br>
&gt; kern.cam.ada.0.trim_lbas: 7771432624<br>
&gt; kern.cam.ada.0.trim_ranges: 371381<br>
&gt; kern.cam.ada.0.trim_count: 310842<br>
&gt; kern.cam.ada.0.delete_method: DSM_TRIM<br>
&gt;<br>
&gt; If I take one of the disks out of the pool and replace it with a <br>
&gt; spare, and do a manual trim it seems to work<br>
&gt;<br>
I had a hard time seeing evidence of this at the disk level while <br>
fiddling with TRIM recently. It appeared that at least some counters are <b=
r>
driver and operation specific. For example, the da driver appears to <br>
update counters in some paths but not others. I assume that ada is <br>
different. There is a bug report for da, but haven&#39;t seen any feedback =
...<br>
<br>
<a href=3D"https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277673" rel=
=3D"noreferrer" target=3D"_blank">https://bugs.freebsd.org/bugzilla/show_bu=
g.cgi?id=3D277673</a><br>
<br>
You could try to run gstat with the -d flag during the time period when <br=
>
the delete operations are expected to occur. That should give you an <br>
idea of what&#39;s happening at the disk level in real time but may not <br=
>
offer more info than you&#39;re already seeing.<br></blockquote><div><br></=
div><div>These changes looked good. My apologies for not noticing this soon=
er</div><div>(I think that our CC to the mailing list in bugzilla has stopp=
ed working,</div><div>so I didn&#39;t see the redirect). I&#39;ve committed=
 the changes, and queued them</div><div>to my stable-14 and stable-13 branc=
hes.</div><div><br></div><div>Thank you so much for your submission.</div><=
div><br></div><div>Warner<br></div></div></div>

--00000000000016188e06178e43f3--

From nobody Fri May  3 15:28:04 2024
X-Original-To: stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VWF7T206Sz5KC5Z
	for <stable@mlmmj.nyi.freebsd.org>; Fri,  3 May 2024 15:28:21 +0000 (UTC)
	(envelope-from wlosh@bsdimp.com)
Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VWF7T0Fj8z562M
	for <stable@freebsd.org>; Fri,  3 May 2024 15:28:21 +0000 (UTC)
	(envelope-from wlosh@bsdimp.com)
Authentication-Results: mx1.freebsd.org;
	none
Received: by mail-ej1-x62d.google.com with SMTP id a640c23a62f3a-a5200afe39eso1225932866b.1
        for <stable@freebsd.org>; Fri, 03 May 2024 08:28:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1714750096; x=1715354896; darn=freebsd.org;
        h=cc:to:subject:message-id:date:from:in-reply-to:references
         :mime-version:from:to:cc:subject:date:message-id:reply-to;
        bh=cg/PYfTyAiEE2Dbf98k81owCQcwXUU0iDLEBFAj/yG4=;
        b=bpXOK8HYz2xAeS7i00gui9enGP1xBdGHUqhmLysnZdTKBT3EjS8EL20neA68RGtXSF
         K0xp6c4IFmAX6sZLof4x7jktamKHrau9Q9+8pWpwjWC5tK0OEyUgVzZTvzgtEew/SSKW
         QJSKOYj/cE04+nuaDfc3miLsayiDmAI6sQr6tbiTQVn9C6eZlXOQ9NTg9D8sOPRQv6T8
         963+ULDN/yocOnKUPDQzgkPT/UJzgAtjwm6khTk9J+fNI9fHOG2GPHCES54j5nTMX8e0
         EoAT6NPKObNYyF6uNVIgWwWVQgO7Tbv5VYR6pA14ILMbg1ab2Nk8kAW+18dWqd2kfk7O
         drgQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1714750096; x=1715354896;
        h=cc:to:subject:message-id:date:from:in-reply-to:references
         :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
         :reply-to;
        bh=cg/PYfTyAiEE2Dbf98k81owCQcwXUU0iDLEBFAj/yG4=;
        b=PZgWE72vDVhW2kPfXHe3lavtL7vdOpTjYRYiMyactansPbjDSZg/Vux/LqkLuW4y8d
         4aSCzbYUbbdwb3/pthLAoo9itfRCx81CuVuAY8eccJ7tSv1Llo2J/QkU43xH+VQxvxUn
         IHtbHsZsFrXIyZa0I/7oNjPbvjJH6mnmElNhWq2bwmBR+Goz3DnT4XpzaTRtFHY2LAlF
         7meVhudH8pblkZ/xlr2/xrOA/m2Wcq2keDPdUjceBvwjzbu8Dz0v5LHrJC+8M5sztGni
         zJk9DM+F67iLEppnLT4GSfVpAgalpPkR9+GR0ORu6Sjcgq6W3CnAyDF21PUXK2prNMXD
         Ot2g==
X-Forwarded-Encrypted: i=1; AJvYcCWHbJQbd06IKc+Gx1dePwZxdVRp0UQiuDlOujd2BBXcMjRBVU5w5JN61w5xFa+JcHUAybgB/Uiy+3aHMusah0LWWzg=
X-Gm-Message-State: AOJu0YyieS2LDNmRqQIgYSMpLiGcY0g0eZJxfLVm9LcwTiLnmIAwQazO
	zUDo1bcZPX8YHu1cQfot7CB4bWVsczrgRi+kgiCWhMDBFpcjU7MIQgFqqz0JRhQ08+OSnAo6map
	9aEDvKr4IqBA9vABLflsDj0JNjhEBXxlAbU7XOA==
X-Google-Smtp-Source: AGHT+IHeeOO7efWNhXc62OCMUCYIEwtPl7HzBI/Hzw+BJGYWc+gFJJBGY88f7ft6kh0xRf3K+OfiIHE/AodFW2dxFdQ=
X-Received: by 2002:a17:906:4a8b:b0:a56:ee1:5695 with SMTP id
 x11-20020a1709064a8b00b00a560ee15695mr2155005eju.19.1714750096171; Fri, 03
 May 2024 08:28:16 -0700 (PDT)
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
References: <alpine.BSF.2.00.2405031711340.23742@gw-old.x-art.ru>
 <CANCZdfoshnihAxQKW5gFF6b_B_vucZycFAM4NGp2YbYW4r+D5A@mail.gmail.com> <alpine.BSF.2.00.2405031810440.23742@gw-old.x-art.ru>
In-Reply-To: <alpine.BSF.2.00.2405031810440.23742@gw-old.x-art.ru>
From: Warner Losh <imp@bsdimp.com>
Date: Fri, 3 May 2024 09:28:04 -0600
Message-ID: <CANCZdfrXZBA_scm2T81YQ=k3=4_ri-_by9JRiz1tUAXhQPC0hA@mail.gmail.com>
Subject: Re: how to tell if TRIM is working
To: Antony Uspensky <uspensky@x-art.ru>
Cc: mike tancsa <mike@sentex.net>, Matthew Grooms <mgrooms@shrew.net>, stable@freebsd.org
Content-Type: multipart/alternative; boundary="0000000000005c82cd06178e5efc"
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]
X-Rspamd-Queue-Id: 4VWF7T0Fj8z562M

--0000000000005c82cd06178e5efc
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, May 3, 2024 at 9:16=E2=80=AFAM Antony Uspensky <uspensky@x-art.ru> =
wrote:

> On Fri, 3 May 2024, Warner Losh wrote:
>
> > Ah yes. You need to enable ZFS trim.
>
> Do you mean autotrim property? No, better to run zpool trim manually.
>

It depends on the drives... But with cheap, crappy consumer drives, you're
correct.

I meant that you had to do it via zpool, not any specific method though.


> > The other trim destroys all the data on that partition.
>
> Not always. It sends TRIM subcommand.
>

True. While technically the contents are allowed to be the same, often
times they are not... Hence the warning... and even if they are the same
right now, there's no guarantee they will remain the same in the future...

Warner

--0000000000005c82cd06178e5efc
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Fri, May 3, 2024 at 9:16=E2=80=AFA=
M Antony Uspensky &lt;<a href=3D"mailto:uspensky@x-art.ru">uspensky@x-art.r=
u</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin=
:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"=
>On Fri, 3 May 2024, Warner Losh wrote:<br>
<br>
&gt; Ah yes. You need to enable ZFS trim.<br>
<br>
Do you mean autotrim property? No, better to run zpool trim manually.<br></=
blockquote><div><br></div><div>It depends on the drives... But with cheap, =
crappy consumer drives, you&#39;re correct.</div><div><br></div><div>I mean=
t that you had to do it via zpool, not any specific method though.<br></div=
><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px=
 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
&gt; The other trim destroys all the data on that partition.<br>
<br>
Not always. It sends TRIM subcommand.<br></blockquote><div><br></div><div>T=
rue. While technically the contents are allowed to be the same, often times=
 they are not... Hence the warning... and even if they are the same right n=
ow, there&#39;s no guarantee they will remain the same in the future... <br=
></div><div><br></div><div>Warner<br></div></div></div>

--0000000000005c82cd06178e5efc--

From nobody Fri May  3 15:38:44 2024
X-Original-To: stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VWFMd6CY1z5J08F
	for <stable@mlmmj.nyi.freebsd.org>; Fri,  3 May 2024 15:38:53 +0000 (UTC)
	(envelope-from mike@sentex.net)
Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VWFMd41NFz58Ms
	for <stable@freebsd.org>; Fri,  3 May 2024 15:38:53 +0000 (UTC)
	(envelope-from mike@sentex.net)
Authentication-Results: mx1.freebsd.org;
	none
Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19])
	by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 443Fcjbe039789
	(version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL);
	Fri, 3 May 2024 11:38:46 -0400 (EDT)
	(envelope-from mike@sentex.net)
Received: from [IPV6:2607:f3e0:0:4:f842:7340:92eb:86db] ([IPv6:2607:f3e0:0:4:f842:7340:92eb:86db])
	by pyroxene2a.sentex.ca (8.18.1/8.15.2) with ESMTPS id 443FciER058223
	(version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO);
	Fri, 3 May 2024 11:38:44 -0400 (EDT)
	(envelope-from mike@sentex.net)
Message-ID: <8ac6c44a-7e3b-4c44-a8f8-dc7b4ad8f276@sentex.net>
Date: Fri, 3 May 2024 11:38:44 -0400
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: how to tell if TRIM is working
To: Antony Uspensky <uspensky@x-art.ru>, Warner Losh <imp@bsdimp.com>
Cc: Matthew Grooms <mgrooms@shrew.net>, stable@freebsd.org
References: <alpine.BSF.2.00.2405031711340.23742@gw-old.x-art.ru>
 <CANCZdfoshnihAxQKW5gFF6b_B_vucZycFAM4NGp2YbYW4r+D5A@mail.gmail.com>
 <alpine.BSF.2.00.2405031810440.23742@gw-old.x-art.ru>
Content-Language: en-US
From: mike tancsa <mike@sentex.net>
Autocrypt: addr=mike@sentex.net; keydata=
 xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP
 xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s
 +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW
 yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH
 VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB
 AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs
 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W
 rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS
 j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ
 cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R
 yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw
 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk
 AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1
 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt
 RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO
 l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7
 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB
 CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8
 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM
 aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X
 rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB
 Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0
 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg==
In-Reply-To: <alpine.BSF.2.00.2405031810440.23742@gw-old.x-art.ru>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Scanned-By: MIMEDefang 2.86
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]
X-Rspamd-Queue-Id: 4VWFMd41NFz58Ms

On 5/3/2024 11:16 AM, Antony Uspensky wrote:
> On Fri, 3 May 2024, Warner Losh wrote:
>
>> Ah yes. You need to enable ZFS trim.
>
> Do you mean autotrim property? No, better to run zpool trim manually.
>
>> The other trim destroys all the data on that partition.
>
> Not always. It sends TRIM subcommand.

I notice that with trim -f <device>, it makes a very large difference in 
performance for certain consumer drives (WD Blue vs Samsung for example) 
that had a lot of writes/deletes.Â  On FreeBSD, I am not able to restore 
write performance to reasonable levels unless I do a device level trim 
of the entire disk with trim -f. 
(https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277992).Â  I am still 
trying to figure out why / where this is the case.

 Â Â Â  ---Mike


>
> A.
>

From nobody Fri May  3 15:55:26 2024
X-Original-To: stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VWFl13dyLz5J1LN
	for <stable@mlmmj.nyi.freebsd.org>; Fri,  3 May 2024 15:55:41 +0000 (UTC)
	(envelope-from paul@gromit.dlib.vt.edu)
Received: from gromit.dlib.vt.edu (gromit.dlib.vt.edu [128.173.126.123])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VWFl11Xzbz3xYp
	for <stable@freebsd.org>; Fri,  3 May 2024 15:55:41 +0000 (UTC)
	(envelope-from paul@gromit.dlib.vt.edu)
Authentication-Results: mx1.freebsd.org;
	none
Received: from smtpclient.apple (unknown [IPv6:2601:5cf:407e:55c1::5])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by gromit.dlib.vt.edu (Postfix) with ESMTPSA id 981857EE03;
	Fri, 03 May 2024 11:55:37 -0400 (EDT)
Content-Type: text/plain;
	charset=utf-8
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.500.171.1.1\))
Subject: Re: how to tell if TRIM is working
From: Paul Mather <paul@gromit.dlib.vt.edu>
In-Reply-To: <alpine.BSF.2.00.2405031810440.23742@gw-old.x-art.ru>
Date: Fri, 3 May 2024 11:55:26 -0400
Cc: Warner Losh <imp@bsdimp.com>,
 mike tancsa <mike@sentex.net>,
 Matthew Grooms <mgrooms@shrew.net>,
 Antony Uspensky <uspensky@x-art.ru>
Content-Transfer-Encoding: quoted-printable
Message-Id: <9672AF0D-1E1A-424F-AB7E-1933E9CDCAB8@gromit.dlib.vt.edu>
References: <alpine.BSF.2.00.2405031711340.23742@gw-old.x-art.ru>
 <CANCZdfoshnihAxQKW5gFF6b_B_vucZycFAM4NGp2YbYW4r+D5A@mail.gmail.com>
 <alpine.BSF.2.00.2405031810440.23742@gw-old.x-art.ru>
To: stable@freebsd.org
X-Mailer: Apple Mail (2.3774.500.171.1.1)
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:1312, ipnet:128.173.0.0/16, country:US]
X-Rspamd-Queue-Id: 4VWFl11Xzbz3xYp

On May 3, 2024, at 11:16=E2=80=AFAM, Antony Uspensky <uspensky@x-art.ru> =
wrote:


> On Fri, 3 May 2024, Warner Losh wrote:
>=20
>> Ah yes. You need to enable ZFS trim.
>=20
> Do you mean autotrim property? No, better to run zpool trim manually.


Somewhat recently, 14-STABLE acquired an =
/etc/periodic/daily/801.trim-zfs daily task for running "zpool trim" on =
designated pools.  I assume this will make it into the upcoming =
14.1-RELEASE.

Cheers,

Paul.=

From nobody Fri May  3 21:39:26 2024
X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VWPMs5kP4z5JZlL
	for <freebsd-stable@mlmmj.nyi.freebsd.org>; Fri,  3 May 2024 21:39:37 +0000 (UTC)
	(envelope-from wollman@hergotha.csail.mit.edu)
Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "garrett.wollman.name", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VWPMr5qt8z4X7l
	for <freebsd-stable@freebsd.org>; Fri,  3 May 2024 21:39:36 +0000 (UTC)
	(envelope-from wollman@hergotha.csail.mit.edu)
Authentication-Results: mx1.freebsd.org;
	dkim=none;
	dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bimajority.org (policy=none);
	spf=pass (mx1.freebsd.org: domain of wollman@hergotha.csail.mit.edu designates 2001:470:1f06:ccb::2 as permitted sender) smtp.mailfrom=wollman@hergotha.csail.mit.edu
Received: from hergotha.csail.mit.edu (localhost [127.0.0.1])
	by hergotha.csail.mit.edu (8.17.1/8.17.1) with ESMTPS id 443LdRO4027191
	(version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO)
	for <freebsd-stable@freebsd.org>; Fri, 3 May 2024 17:39:29 -0400 (EDT)
	(envelope-from wollman@hergotha.csail.mit.edu)
Received: (from wollman@localhost)
	by hergotha.csail.mit.edu (8.17.1/8.17.1/Submit) id 443LdRnh027190;
	Fri, 3 May 2024 17:39:27 -0400 (EDT)
	(envelope-from wollman)
Message-ID: <26165.22926.598276.904906@hergotha.csail.mit.edu>
Date: Fri, 3 May 2024 17:39:26 -0400
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
From: Garrett Wollman <wollman@bimajority.org>
To: freebsd-stable@freebsd.org
Subject: ZFS property inheritance broken for `readonly`?
X-Mailer: VM 8.2.0b under 29.2 (amd64-portbld-freebsd13.2)
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (hergotha.csail.mit.edu [0.0.0.0]); Fri, 03 May 2024 17:39:29 -0400 (EDT)
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (hergotha.csail.mit.edu [0.0.0.0]); Fri, 03 May 2024 13:10:08 -0400 (EDT)
X-Spam-Status: No, score=-0.8 required=5.0 tests=ALL_TRUSTED,
	HEADER_FROM_DIFFERENT_DOMAINS,T_SCC_BODY_TEXT_LINE autolearn=disabled
	version=4.0.0
X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on
	hergotha.csail.mit.edu
X-Spamd-Bar: -
X-Spamd-Result: default: False [-1.84 / 15.00];
	SUBJECT_ENDS_QUESTION(1.00)[];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-0.94)[-0.941];
	FORGED_SENDER(0.30)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu];
	R_SPF_ALLOW(-0.20)[+ip6:2001:470:1f06:ccb::2];
	MIME_GOOD(-0.10)[text/plain];
	DMARC_POLICY_SOFTFAIL(0.10)[bimajority.org : SPF not aligned (relaxed), No valid DKIM,none];
	RCPT_COUNT_ONE(0.00)[1];
	TO_DN_NONE(0.00)[];
	FREEFALL_USER(0.00)[wollman];
	ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US];
	MIME_TRACE(0.00)[0:+];
	FROM_HAS_DN(0.00)[];
	R_DKIM_NA(0.00)[];
	MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org];
	TO_MATCH_ENVRCPT_ALL(0.00)[];
	RCVD_COUNT_TWO(0.00)[2];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org];
	FROM_NEQ_ENVFROM(0.00)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu];
	RCVD_TLS_LAST(0.00)[];
	ARC_NA(0.00)[]
X-Rspamd-Queue-Id: 4VWPMr5qt8z4X7l

I've noticed recently (not sure when this started, maybe on the 12->13
transition, which would be the same as the old ZFS to OpenZFS
transition for us) that inheritance for the `readonly` property is
broken.

Here is an example:

[root@nfs-prod-11 /home/wollman]# zfs set readonly=on export/cl
[root@nfs-prod-11 /home/wollman]# zfs get readonly export/cl/u
NAME         PROPERTY  VALUE   SOURCE
export/cl/u  readonly  off     temporary

The child filesystem `export/cl/u` should have inherited the readonly
setting from `export/cl` but instead it (and all other children) have
overridden it, and I have to manually `zfs inherit readonly` on all of
the children to get the proper behavior.  This is quite surprising,
and the first time it happened I was sure that something had gone
wrong.

-GAWollman


From nobody Fri May  3 21:55:35 2024
X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VWPkN2LsHz5JcDH
	for <freebsd-stable@mlmmj.nyi.freebsd.org>; Fri,  3 May 2024 21:55:40 +0000 (UTC)
	(envelope-from SRS0=a8w7=MG=klop.ws=ronald-lists@realworks.nl)
Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [194.109.157.24])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VWPkM4w3cz4YZk
	for <freebsd-stable@freebsd.org>; Fri,  3 May 2024 21:55:39 +0000 (UTC)
	(envelope-from SRS0=a8w7=MG=klop.ws=ronald-lists@realworks.nl)
Authentication-Results: mx1.freebsd.org;
	none
Date: Fri, 3 May 2024 23:55:35 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klop.ws; s=rw2;
	t=1714773336;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 in-reply-to:in-reply-to; bh=QmMne/HEgV5l2e76dpBY8jD6dkWyTm7OLSQKynDb0V8=;
	b=TtPVDgJYhfSTgSNJrVyaQ1Mq25JQ1g8IoWQjJqt3wCaX95TwOMbLeEhlwaWXR7KIPxeuIt
	eXLqTlM4D/maumWL+ap/8tmHDjga99E44Piz616mk4vaWDL71rysj27aHXkxJ/hJEhaZ5B
	lxx7jd3uKuW5R0UKHs+KPxG4FhbFGYz3URd2qla+A3Etu5l4w/2Ab+JQo2RXvtNM3J47yY
	yOGp3d4GROdGBGA6RnM1meC0c0DeYNcuuPsJdF8AP0MLFX/QxkKGhtEwUQ9/7eUpvMh9bU
	Xt0Yg83qXxLx2wbbHBbKQF9hwrh5QhTdtt+dwQXs4sU0XA0WRXsLT19LUljZcQ==
From: Ronald Klop <ronald-lists@klop.ws>
To: Garrett Wollman <wollman@bimajority.org>
Cc: freebsd-stable@freebsd.org
Message-ID: <727612068.8243.1714773335920@localhost>
In-Reply-To: <26165.22926.598276.904906@hergotha.csail.mit.edu>
Subject: Re: ZFS property inheritance broken for `readonly`?
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_8242_1432664771.1714773335864"
X-Mailer: Realworks (701.18)
Importance: Normal
X-Priority: 3 (Normal)
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:3265, ipnet:194.109.0.0/16, country:NL]
X-Rspamd-Queue-Id: 4VWPkM4w3cz4YZk

------=_Part_8242_1432664771.1714773335864
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable

Van: Garrett Wollman <wollman@bimajority.org>
Datum: 3 mei 2024 23:39
Aan: freebsd-stable@freebsd.org
Onderwerp: ZFS property inheritance broken for `readonly`?

>=20
>=20
> I've noticed recently (not sure when this started, maybe on the 12->13
> transition, which would be the same as the old ZFS to OpenZFS
> transition for us) that inheritance for the `readonly` property is
> broken.
>=20
> Here is an example:
>=20
> [root@nfs-prod-11 /home/wollman]# zfs set readonly=3Don export/cl
> [root@nfs-prod-11 /home/wollman]# zfs get readonly export/cl/u
> NAME         PROPERTY  VALUE   SOURCE
> export/cl/u  readonly  off     temporary
>=20
> The child filesystem `export/cl/u` should have inherited the readonly
> setting from `export/cl` but instead it (and all other children) have
> overridden it, and I have to manually `zfs inherit readonly` on all of
> the children to get the proper behavior.  This is quite surprising,
> and the first time it happened I was sure that something had gone
> wrong.
>=20
> -GAWollman
>=20
>=20
>=20
>=20
>=20
>=20


How did you mount the fs?
Could it be that the =E2=80=9Ctemporary=E2=80=9D source of the setting had =
to do with it?
See man zfs.=20
Regards,Ronald

------=_Part_8242_1432664771.1714773335864
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head></head><body><br><p><small><strong>Van:</strong> Garrett Wollma=
n &lt;wollman@bimajority.org&gt;<br><strong>Datum:</strong> 3 mei 2024 23:3=
9<br><strong>Aan:</strong> freebsd-stable@freebsd.org<br><strong>Onderwerp:=
</strong> ZFS property inheritance broken for `readonly`?<br></small></p><b=
lockquote style=3D"margin-left: 5px; border-left: 3px solid #ccc; margin-ri=
ght: 0px; padding-left: 5px;"><div class=3D"MessageRFC822Viewer" id=3D"P"><=
!-- P -->
<!-- processMimeMessage --><div class=3D"TextPlainViewer" id=3D"P.P"><!-- P=
.P -->I've noticed recently (not sure when this started, maybe on the 12-&g=
t;13<br>
transition, which would be the same as the old ZFS to OpenZFS<br>
transition for us) that inheritance for the `readonly` property is<br>
broken.<br>
<br>
Here is an example:<br>
<br>
[root@nfs-prod-11 /home/wollman]# zfs set readonly=3Don export/cl<br>
[root@nfs-prod-11 /home/wollman]# zfs get readonly export/cl/u<br>
NAME &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;PROPERTY &nbsp;VALUE &=
nbsp;&nbsp;SOURCE<br>
export/cl/u &nbsp;readonly &nbsp;off &nbsp;&nbsp;&nbsp;&nbsp;temporary<br>
<br>
The child filesystem `export/cl/u` should have inherited the readonly<br>
setting from `export/cl` but instead it (and all other children) have<br>
overridden it, and I have to manually `zfs inherit readonly` on all of<br>
the children to get the proper behavior. &nbsp;This is quite surprising,<br=
>
and the first time it happened I was sure that something had gone<br>
wrong.<br>
<br>
-GAWollman<br>
<br>
<br>
</div><!-- TextPlainViewer -->
<hr>
</div><!-- MessageRFC822Viewer -->
</blockquote><br><br>How did you mount the fs?<br class=3D"rw_extra"><br>Co=
uld it be that the =E2=80=9Ctemporary=E2=80=9D source of the setting had to=
 do with it?<div>See man zfs.&nbsp;<br class=3D"rw_extra"><br>Regards,<br c=
lass=3D"rw_extra">Ronald<br><br></div></body></html>
------=_Part_8242_1432664771.1714773335864--

From nobody Fri May  3 22:25:03 2024
X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VWQNP68p9z5JfGP
	for <freebsd-stable@mlmmj.nyi.freebsd.org>; Fri,  3 May 2024 22:25:09 +0000 (UTC)
	(envelope-from wollman@hergotha.csail.mit.edu)
Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "garrett.wollman.name", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VWQNP1DYvz4cqH
	for <freebsd-stable@freebsd.org>; Fri,  3 May 2024 22:25:08 +0000 (UTC)
	(envelope-from wollman@hergotha.csail.mit.edu)
Authentication-Results: mx1.freebsd.org;
	none
Received: from hergotha.csail.mit.edu (localhost [127.0.0.1])
	by hergotha.csail.mit.edu (8.17.1/8.17.1) with ESMTPS id 443MP5Lm027577
	(version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO);
	Fri, 3 May 2024 18:25:05 -0400 (EDT)
	(envelope-from wollman@hergotha.csail.mit.edu)
Received: (from wollman@localhost)
	by hergotha.csail.mit.edu (8.17.1/8.17.1/Submit) id 443MP5YB027576;
	Fri, 3 May 2024 18:25:05 -0400 (EDT)
	(envelope-from wollman)
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <26165.25663.720989.835800@hergotha.csail.mit.edu>
Date: Fri, 3 May 2024 18:25:03 -0400
From: Garrett Wollman <wollman@bimajority.org>
To: Ronald Klop <ronald-lists@klop.ws>
Cc: freebsd-stable@freebsd.org
Subject: Re: ZFS property inheritance broken for `readonly`?
In-Reply-To: <727612068.8243.1714773335920@localhost>
References: <26165.22926.598276.904906@hergotha.csail.mit.edu>
	<727612068.8243.1714773335920@localhost>
X-Mailer: VM 8.2.0b under 29.2 (amd64-portbld-freebsd13.2)
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (hergotha.csail.mit.edu [0.0.0.0]); Fri, 03 May 2024 18:25:05 -0400 (EDT)
X-Spam-Status: No, score=-1.8 required=5.0 tests=ALL_TRUSTED,
	HEADER_FROM_DIFFERENT_DOMAINS,NICE_REPLY_A,T_SCC_BODY_TEXT_LINE
	autolearn=disabled version=4.0.0
X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on
	hergotha.csail.mit.edu
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]
X-Rspamd-Queue-Id: 4VWQNP1DYvz4cqH

<<On Fri, 3 May 2024 23:55:35 +0200 (CEST), Ronald Klop <ronald-lists@klop.ws> said:

> How did you mount the fs?

In the normal (completely automatic) way.

-GAWollman


From nobody Sat May  4 00:09:44 2024
X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VWSj53CbFz5JpVX
	for <freebsd-stable@mlmmj.nyi.freebsd.org>; Sat,  4 May 2024 00:09:45 +0000 (UTC)
	(envelope-from brooks@freebsd.org)
Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "smtp.freebsd.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VWSj5129Rz4p3c;
	Sat,  4 May 2024 00:09:45 +0000 (UTC)
	(envelope-from brooks@freebsd.org)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim;
	t=1714781385;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 in-reply-to:in-reply-to:references:references;
	bh=pqdR6zlBCMCvKA8GL0/jrivYkDID74BB3r+MD2sKAAA=;
	b=cWAzH8pcs01xWLXrIUjf51YVAqTJezMv4WnKiWjWA0zcjjzUC/YJbXJrKETyI3D1wscC1A
	XqZLGzDckeElC+r2h5DkYFfH+D3VEranSL2ZJMBcpxUzxbMScvdLMYizn7cXeJ52fJDJiW
	QoBcOz0+hji6Xv4wjmfreuxTrANp4+hys5ksRS3dlJSr9AyfNmw0BCGTA8h0DaJpjLvUPX
	s5/ien8CI+klN1LUoLZ4sdPrYG4mpzygGhFZksOScc73l4c6qGB2UXqGankWQ3jcWmSd+B
	tM5LmvUTUfJgJd8htwMmfnIJkEsOw/Rw/b8GcZWR4tVYOAxH0TpZRo4xAhcHzg==
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1714781385; a=rsa-sha256; cv=none;
	b=WS/zMW5E2bmln9GNPhYFstOqV0TUdM4B7BOcu1brqQN91++oxYFEZNToCTvWOhP3lJCGGW
	wo/JgC/SRb2pYqiEVgf7Vfh4y8IYmFKLQeoGYXTLRmPQTjE7Q44CbjUhMOt6tXP/6iQUD0
	0RBmdypv37Jzkam9pSdfXjCQ92NupilWFNwCOxyLo0MhMCBYWGOnAj8ADsznJpYks+dfE+
	5su5W5CnsgAwZN7rOxj7Rxm7ttq0qstzjG/nlqDctMshqLMNc+S0/I8+ZG3ZcT+E7Y5Ux1
	5+J9CPsZlgjzMXIFomxYMvFYBmZWXIYrwAay0azMRkGzWdISFFncX91kEtaZqA==
ARC-Authentication-Results: i=1;
	mx1.freebsd.org;
	none
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
	s=dkim; t=1714781385;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 in-reply-to:in-reply-to:references:references;
	bh=pqdR6zlBCMCvKA8GL0/jrivYkDID74BB3r+MD2sKAAA=;
	b=HvF9oqRS0HWoijNWtPtB89zZgXrbkVR+40CQq7Vidl3T+7hP/Dn2K97fIBvo2uLhzJF+4v
	xrmR6gpQZuN17u+UqvwYVBsEPJ4zOzL11qWx/hojr+gVte2f9pVHi/ys8TJgE8bPtLhpdl
	o+GXcgfflaiELa7DuY1hJDnHBRj0myu9/XMSn1B+wzQaNSXu0+5hBdolJt/NSBRUCG0AJK
	HrwDGoNyVMrBp0EDAe8WqFE80LDUVjxw4uU7bRsf+6345fLlcXsfAT/9ZodOSRLBgu3fB2
	RGu0wDA5VkVYo9CEsawO9KktE6MdWSdMHz7e75rh0tcEITSPAb6/IpQgybZhBA==
Received: from spindle.one-eyed-alien.net (spindle.one-eyed-alien.net [199.48.129.229])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	(Authenticated sender: brooks/mail)
	by smtp.freebsd.org (Postfix) with ESMTPSA id 4VWSj50PHSzyRG;
	Sat,  4 May 2024 00:09:45 +0000 (UTC)
	(envelope-from brooks@freebsd.org)
Received: by spindle.one-eyed-alien.net (Postfix, from userid 3001)
	id 4925D3C019B; Sat,  4 May 2024 00:09:44 +0000 (UTC)
Date: Sat, 4 May 2024 00:09:44 +0000
From: Brooks Davis <brooks@freebsd.org>
To: Garrett Wollman <wollman@bimajority.org>
Cc: freebsd-stable@freebsd.org
Subject: Re: ZFS property inheritance broken for `readonly`?
Message-ID: <ZjV8yNCWBxVrQENI@spindle.one-eyed-alien.net>
References: <26165.22926.598276.904906@hergotha.csail.mit.edu>
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <26165.22926.598276.904906@hergotha.csail.mit.edu>

On Fri, May 03, 2024 at 05:39:26PM -0400, Garrett Wollman wrote:
> I've noticed recently (not sure when this started, maybe on the 12->13
> transition, which would be the same as the old ZFS to OpenZFS
> transition for us) that inheritance for the `readonly` property is
> broken.
> 
> Here is an example:
> 
> [root@nfs-prod-11 /home/wollman]# zfs set readonly=on export/cl
> [root@nfs-prod-11 /home/wollman]# zfs get readonly export/cl/u
> NAME         PROPERTY  VALUE   SOURCE
> export/cl/u  readonly  off     temporary
> 
> The child filesystem `export/cl/u` should have inherited the readonly
> setting from `export/cl` but instead it (and all other children) have
> overridden it, and I have to manually `zfs inherit readonly` on all of
> the children to get the proper behavior.  This is quite surprising,
> and the first time it happened I was sure that something had gone
> wrong.

I find myself wondering if ZFS was unable to downgrade the mount and
this is a transient state, but the fact that inherit works suggests
otherwise.  I suppose it could be a missing case in the spl layer.

-- Brooks

From nobody Sat May  4 01:31:29 2024
X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VWVWT2Kk6z5JxQG
	for <freebsd-stable@mlmmj.nyi.freebsd.org>; Sat,  4 May 2024 01:31:33 +0000 (UTC)
	(envelope-from wollman@hergotha.csail.mit.edu)
Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "garrett.wollman.name", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VWVWT0b8pz40YJ;
	Sat,  4 May 2024 01:31:33 +0000 (UTC)
	(envelope-from wollman@hergotha.csail.mit.edu)
Authentication-Results: mx1.freebsd.org;
	none
Received: from hergotha.csail.mit.edu (localhost [127.0.0.1])
	by hergotha.csail.mit.edu (8.17.1/8.17.1) with ESMTPS id 4441VUBl028976
	(version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO);
	Fri, 3 May 2024 21:31:31 -0400 (EDT)
	(envelope-from wollman@hergotha.csail.mit.edu)
Received: (from wollman@localhost)
	by hergotha.csail.mit.edu (8.17.1/8.17.1/Submit) id 4441VTeE028974;
	Fri, 3 May 2024 21:31:29 -0400 (EDT)
	(envelope-from wollman)
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <26165.36849.451200.198723@hergotha.csail.mit.edu>
Date: Fri, 3 May 2024 21:31:29 -0400
From: Garrett Wollman <wollman@bimajority.org>
To: Brooks Davis <brooks@freebsd.org>
Cc: freebsd-stable@freebsd.org
Subject: Re: ZFS property inheritance broken for `readonly`?
In-Reply-To: <ZjV8yNCWBxVrQENI@spindle.one-eyed-alien.net>
References: <26165.22926.598276.904906@hergotha.csail.mit.edu>
	<ZjV8yNCWBxVrQENI@spindle.one-eyed-alien.net>
X-Mailer: VM 8.2.0b under 29.2 (amd64-portbld-freebsd13.2)
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (hergotha.csail.mit.edu [0.0.0.0]); Fri, 03 May 2024 21:31:31 -0400 (EDT)
X-Spam-Status: No, score=-1.8 required=5.0 tests=ALL_TRUSTED,
	HEADER_FROM_DIFFERENT_DOMAINS,NICE_REPLY_A,T_SCC_BODY_TEXT_LINE
	autolearn=disabled version=4.0.0
X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on
	hergotha.csail.mit.edu
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]
X-Rspamd-Queue-Id: 4VWVWT0b8pz40YJ

<<On Sat, 4 May 2024 00:09:44 +0000, Brooks Davis <brooks@freebsd.org> said:

>> [I wrote:]
>> The child filesystem `export/cl/u` should have inherited the readonly
>> setting from `export/cl` but instead it (and all other children) have
>> overridden it, and I have to manually `zfs inherit readonly` on all of
>> the children to get the proper behavior.  This is quite surprising,
>> and the first time it happened I was sure that something had gone
>> wrong.

> I find myself wondering if ZFS was unable to downgrade the mount and
> this is a transient state, but the fact that inherit works suggests
> otherwise.  I suppose it could be a missing case in the spl layer.

It definitely *used to* work; this has been part of our standard
migration procedure for a good decade now, and it's only since we
upgraded to 13 that it stopped working.

-GAWollman


From nobody Sat May  4 07:20:48 2024
X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VWfGW2jTYz5K2gK
	for <freebsd-stable@mlmmj.nyi.freebsd.org>; Sat,  4 May 2024 07:20:51 +0000 (UTC)
	(envelope-from SRS0=r5VE=MH=klop.ws=ronald-lists@realworks.nl)
Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [194.109.157.24])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VWfGW0fJ7z4Vds
	for <freebsd-stable@freebsd.org>; Sat,  4 May 2024 07:20:51 +0000 (UTC)
	(envelope-from SRS0=r5VE=MH=klop.ws=ronald-lists@realworks.nl)
Authentication-Results: mx1.freebsd.org;
	none
Date: Sat, 4 May 2024 09:20:48 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klop.ws; s=rw2;
	t=1714807249;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 in-reply-to:in-reply-to:references:references;
	bh=G7GcVsPZeT9zA311aehEszJ4ZOhAO28HOM9MhD/J2dM=;
	b=AzOUkp2RAkVpniAJl52X2VrgOUO6iTXNh3/Xy2Sn34AswBpucM0oKqoLi4Fqxt6b97TDlX
	9PW2qaO0yph9mNaoDkDykKIlNKdQt3HF1vINQB2d1x9H7YX4D5AhJ61yzoxxc0Yz41zIdO
	7epqzzfO2eSY+C1gWXEPJ9oQ49DMUhwc2UtfjK+mAtm64rx23lKifog2d7iTOVK33WFwjF
	YToLrcWzBwMQrOtxAvBq8nSBYn7BV6bhDXu6RLlOzjVQ2rHrjhFecuCg17fno81l0ppi9I
	xd1VhvM6Rh7FAqlDWbqOVO7uEDOsAlYYlgR7I2l8TfwIfJ0tjTMxanZuHzLKUA==
From: Ronald Klop <ronald-lists@klop.ws>
To: Garrett Wollman <wollman@bimajority.org>
Cc: freebsd-stable@freebsd.org
Message-ID: <1810690296.211867.1714807248568@localhost>
In-Reply-To: <26165.25663.720989.835800@hergotha.csail.mit.edu>
References: <26165.22926.598276.904906@hergotha.csail.mit.edu>
	<727612068.8243.1714773335920@localhost>
 <26165.25663.720989.835800@hergotha.csail.mit.edu>
Subject: Re: ZFS property inheritance broken for `readonly`?
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_211866_1262481256.1714807248522"
X-Mailer: Realworks (700.152)
Importance: Normal
X-Priority: 3 (Normal)
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:3265, ipnet:194.109.0.0/16, country:NL]
X-Rspamd-Queue-Id: 4VWfGW0fJ7z4Vds

------=_Part_211866_1262481256.1714807248522
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

Van: Garrett Wollman <wollman@bimajority.org>
Datum: zaterdag, 4 mei 2024 00:25
Aan: Ronald Klop <ronald-lists@klop.ws>
CC: freebsd-stable@freebsd.org
Onderwerp: Re: ZFS property inheritance broken for `readonly`?
> 
> <<On Fri, 3 May 2024 23:55:35 +0200 (CEST), Ronald Klop <ronald-lists@klop.ws> said:
> 
> > How did you mount the fs?
> 
> In the normal (completely automatic) way.
> 
> -GAWollman
>  
> 
> 
> 


Can you elaborate what the "normal way" is for you?
Is bootfs set to a zfs pool? Is /etc/fstab used for mounting ZFS?
Can you provide some copy-paste of your config?

What I found about the topic:

The "temporary" source of a property is documented for Oracle ZFS, but I doubt OpenZFS changed a Tlot in how this works.

https://docs.oracle.com/cd/E36784_01/html/E36835/gamnt.html

For openzfs I found this comment:
https://github.com/openzfs/zfs/issues/985#issuecomment-27997625

My ZFS pools don't have any property with source "temporary". So my suggestion would be to look into why you do have a "temporary" property. It at least indicates why it is not inheriting the property from the parent. But I'm also not sure if it is the key to the solution.

Regards,
Ronald.
 
------=_Part_211866_1262481256.1714807248522
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7bit

<html><head></head><body><br>
<p><strong>Van:</strong> Garrett Wollman &lt;wollman@bimajority.org&gt;<br>
<strong>Datum:</strong> zaterdag, 4 mei 2024 00:25<br>
<strong>Aan:</strong> Ronald Klop &lt;ronald-lists@klop.ws&gt;<br>
<strong>CC:</strong> freebsd-stable@freebsd.org<br>
<strong>Onderwerp:</strong> Re: ZFS property inheritance broken for `readonly`?</p>

<blockquote style="padding-right: 0px; padding-left: 5px; margin-left: 5px; border-left: #000000 2px solid; margin-right: 0px">
<div class="MessageRFC822Viewer" id="P">
<div class="TextPlainViewer" id="P.P">&lt;&lt;On Fri, 3 May 2024 23:55:35 +0200 (CEST), Ronald Klop &lt;ronald-lists@klop.ws&gt; said:<br>
<br>
&gt; How did you mount the fs?<br>
<br>
In the normal (completely automatic) way.<br>
<br>
-GAWollman<br>
&nbsp;</div>

<hr></div>
</blockquote>
<br>
<br>
Can you elaborate what the "normal way" is for you?<br>
Is bootfs set to a zfs pool? Is /etc/fstab used for mounting ZFS?<br>
Can you provide some copy-paste of your config?<br>
<br>
What I found about the topic:<br>
<br>
The "temporary" source of a property is documented for Oracle ZFS, but I doubt OpenZFS changed a Tlot in how this works.<br>
<br>
https://docs.oracle.com/cd/E36784_01/html/E36835/gamnt.html<br>
<br>
For openzfs I found this comment:<br>
https://github.com/openzfs/zfs/issues/985#issuecomment-27997625<br>
<br>
My ZFS pools don't have any property with source "temporary". So my suggestion would be to look into why you do have a "temporary" property. It at least indicates why it is not inheriting the property from the parent. But I'm also not sure if it is the key to the solution.<br>
<br>
Regards,<br>
Ronald.<br>
&nbsp;</body></html>
------=_Part_211866_1262481256.1714807248522--

From nobody Sun May  5 15:41:48 2024
X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VXTL52WCdz5K9M7
	for <freebsd-stable@mlmmj.nyi.freebsd.org>; Sun, 05 May 2024 15:41:49 +0000 (UTC)
	(envelope-from salvadore@freebsd.org)
Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "freefall.freebsd.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VXTL50Qc6z4s88
	for <freebsd-stable@freebsd.org>; Sun,  5 May 2024 15:41:49 +0000 (UTC)
	(envelope-from salvadore@freebsd.org)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim;
	t=1714923709;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding;
	bh=nSdiKnkC3LQt7iFoMEGzkBmhnqhFIumekTHiIgdVbmQ=;
	b=sySmBEmIqe3zVTQw4QqCq28+NoFOxpKVC8DrxTGcAKo9aWp3y1YTvcfy0zWAfoOSZHb8kX
	zDGKe+b496dyzqKo8e1FVCob9vOOTwabYKg0i2dcBKBFyV697QewQEJib4Q5w8E+5Tog34
	LpV+liUr3mjQGIvBRUPqSFsGw/9SSTqZs8CaV5+98tNYBOX8ovfmZ2mZj1RnxdFlUn4iO9
	ehwKRGulOKsxVob4yqnT+UJtl6YVmrbQcbBUEh1tx/uW6I0VQcU89skRLlNGfIEnJEV91J
	UDZ9niIm4M1hA2+OMKyxwo60TY49GVozTtfvLceYXeWHogSrbnGcYLD96EpS1w==
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1714923709; a=rsa-sha256; cv=none;
	b=dmFBqRXmuSBwJukNxqJHL05v00zFGpoGjn6zpkcx+/rb9TzSRAQVb0MTZA2JrcDv+6P1fe
	n4PiUtz1KCPlB+Tme4KAgn+On3GEKlFyTd4APM5Y7OtPhAUHT3sxgUkRl19L9zt+ani6pq
	snrkY8qIDYyKB4gYp2lRlqG++QAlwiR9vmDj9EZmz4iOnJDpYlztssH8UF7GAHh3BHYhHb
	HJ057CayKpz5NcOPqwamOHcn+8mHHZxVNc73382BigstZpgnOmwFmT2qbkB2Sp23fT0Qx+
	NDOPN0XlTAIU++OD/4/BGEPiYmAD+2KbdrMSwyW7x0v8KSHK/u+p68sex+RYYw==
ARC-Authentication-Results: i=1;
	mx1.freebsd.org;
	none
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
	s=dkim; t=1714923709;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding;
	bh=nSdiKnkC3LQt7iFoMEGzkBmhnqhFIumekTHiIgdVbmQ=;
	b=ZDcglLitc8EJllkCuD6Y0S54+rhDKUfHuHehaxKfLNIAQ3fg4+guPzyAycE1ylo8pR/C5i
	3PzSMA+5AQcG3medd/r7x6uBLC9f2o/esazC0gW4K2doWkOUCTnEDdcnRrsKdRN4Ti4hWM
	Nck/O5YNkHgIYwO5cPvfifGWEJo4QpYB1F8+mBZF3lo6A4nZ3XgCw2HX7Q2R4FHX5UzmnB
	h46QYvkgXSFNl/bBi9Lr6Dk+qfJitNcSXGwSQHCu6YFx9UKIx+bq+jMuIOmA3/ojMuOmGW
	7FTGcClgWIzD0rorr3mbFTB2Ott/4iYGfE70u/P9L3PRiEBJmAQmYTeC9pZo4A==
Received: by freefall.freebsd.org (Postfix, from userid 1472)
	id F1606D9A0; Sun, 05 May 2024 15:41:48 +0000 (UTC)
Date: Sun, 5 May 2024 15:41:48 +0000
From: Lorenzo Salvadore <salvadore@freebsd.org>
To: freebsd-stable@freebsd.org
Subject: FreeBSD Status Report - First Quarter 2024
Message-ID: <ZjeovLmDu1Ybvr_5@freefall.freebsd.org>
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit

FreeBSD Status Report First Quarter 2024

Here is the first 2024 status report, with 21 entries.

The New Year brings us many new interesting projects, such as the new libsys
that separates system calls from libc and libpthread or work on a graphical
installer for FreeBSD, which will help making our OS more user-friendly. Of
course, the usual projects keep going on, such as the work on cloud-init,
OpenStack, or the GCC ports. As usual our main teams share their progress with
us.

Have a nice read.

Lorenzo Salvadore, on behalf of the Status Team.

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

A rendered version of this report is available here:
https://www.freebsd.org/status/report-2024-01-2024-03/

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”
Table of Contents

  â€¢ FreeBSD Team Reports
      â–¡ FreeBSD Core Team
      â–¡ FreeBSD Foundation
      â–¡ FreeBSD Release Engineering Team
      â–¡ Cluster Administration Team
      â–¡ Continuous Integration
      â–¡ Ports Collection
  â€¢ Projects
      â–¡ Audio Stack Improvements
      â–¡ Bhyve Improvements
      â–¡ Graphical Installer for FreeBSD
  â€¢ Userland
      â–¡ libsys
      â–¡ PackageKit backend for FreeBSD pkg
  â€¢ Kernel
      â–¡ iwlwifi(4) and wireless for 13.3-RELEASE
  â€¢ Architectures
      â–¡ Ten64, WHLE-LS1, and HoneyComb
  â€¢ Cloud
      â–¡ FreeBSD on Microsoft HyperV and Azure
      â–¡ FreeBSD as a Tier 1 cloud-init Platform
      â–¡ OpenStack on FreeBSD
  â€¢ Documentation
      â–¡ Documentation Engineering Team
  â€¢ Ports
      â–¡ FreshPorts: Notification of new packages
      â–¡ GCC on FreeBSD
      â–¡ Valgrind: port to arm64 on its way
  â€¢ Third Party Projects
      â–¡ Containers and FreeBSD: Pot, Potluck and Potman

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

FreeBSD Team Reports

Entries from the various official and semi-official teams, as found in the
Administration Page.

FreeBSD Core Team

Contact: FreeBSD Core Team <core@FreeBSD.org>

The FreeBSD Core Team is the governing body of FreeBSD.

13.3-RELEASE

FreeBSD 13.3 was released on March 5th, 2024.

The release announcement is at:

https://www.freebsd.org/releases/13.3R/announce/

Along the release engineering team, the project dedicates the 13.3-RELEASE to
Glen Barber, with thanks for his many years of contributions as Release
Engineer.

Future of 32-bit platform support

Core announced Future of 32-bit platform support in FreeBSD for deprecating
32-bit platforms over the next couple of major releases.

Commit bits

  â€¢ Core approved the src commit bit for Bojan NovkoviÄ‡

  â€¢ Core reactivated the src commit bits for Mark Peek, Mark Murray, and
    Lawrence Stewart

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

FreeBSD Foundation

Links:
FreeBSD Foundation URL: https://freebsdfoundation.org/
Technology Roadmap URL: https://freebsdfoundation.org/blog/technology-roadmap/
Donate URL: https://freebsdfoundation.org/donate/
Foundation Partnership Program URL: https://freebsdfoundation.org/our-donors/
freebsd-foundation-partnership-program/
FreeBSD Journal URL: https://freebsdfoundation.org/journal/
Foundation Events URL: https://freebsdfoundation.org/our-work/events/

Contact: Deb Goodkin <deb@FreeBSDFoundation.org>

The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to
supporting and promoting the FreeBSD Project and worldwide community, and
helping to advance the state of FreeBSD. We do this in both technical and
non-technical ways. We are 100% supported by donations from individuals and
corporations and those investments help us fund the:

  â€¢ Software development projects to implement features and functionality in
    FreeBSD

  â€¢ Sponsor and organize conferences and developer summits to provide
    collaborative opportunities and promote FreeBSD

  â€¢ Purchase and support of hardware to improve and maintain FreeBSD
    infrastructure

  â€¢ Resources to improve security, quality assurance, and continuous
    integration efforts

  â€¢ Materials and staff needed to promote, educate, and advocate for FreeBSD

  â€¢ Collaboration between commercial vendors and FreeBSD developers

  â€¢ Representation of the FreeBSD Project in executing contracts, license
    agreements, and other legal arrangements that require a recognized legal
    entity

Operations

We kicked off the new year with ambitious goals to help move the FreeBSD
Project forward by identifying features and functionality to support in the
operating system and increasing our advocacy efforts to increase and expand the
visibility of FreeBSD. Stay tuned for a blog post that will provide more
information on our 2024 goals and plans.

We also published the 2024 Budget. In order to provide greater transparency
about the budgeting process, we wrote a blog post that provides more details on
how funding is allocated, new breakouts of some of the project expense
categories, and more details on where the funding is going.

OS Improvements

During the first quarter of 2024, 180 src, 65 ports, and 18 doc tree commits
identified The FreeBSD Foundation as a sponsor.

Three new projects began this quarter.

  â€¢ Work began to improve FreeBSDâ€™s audio stack and provide audio developers
    with useful tools and frameworks to make sound development on FreeBSD
    easier. Read more in Christos Margiolis Audio Stack Improvements report
    entry.

  â€¢ Olivier Certner began his second contract with the Foundation, and this
    time around, the main goal is to make unionfs stable and useful on FreeBSD.
    Other work may include revamping VFS lookups, improving out-of-memory
    handling, implementing a notification system for en-masse detection of
    filesystem changes such as inotify, and improving console usability.

  â€¢ This quarter, a new project to add hierarchical rate limits to the OpenZFS
    file system began. Pawel Dawidek will add support for limits that will be
    configurable, similar to quotas, but would limit the number of read/write
    operations and read/write bandwidth.

Six projects continued this quarter.

  â€¢ You can read about the continued work to port OpenStack components to
    FreeBSD in Chih-Hsin Changâ€™s OpenStack on FreeBSD report entry.

  â€¢ Work continued to improve cloud-init support for FreeBSD. You can read
    about Mina GaliÄ‡â€™s work in her FreeBSD as a Tier 1 cloud-init Platform
    report entry.

  â€¢ A new joint project began between Advanced Micro Devices (AMD) and The
    FreeBSD Foundation to develop a complete FreeBSD AMD IOMMU driver. This
    work will allow FreeBSD to fully support greater than 256 cores with
    features such as CPU mapping and will also include bhyve integration. For
    those interested in the technical details, follow Konstantin Belousov
    commits tagged with Sponsored by fields for Advanced Micro Devices (AMD)
    and The FreeBSD Foundation.

  â€¢ Refer to Pierre Proncheryâ€™s Graphical Installer for FreeBSD report entry to
    read about the status of FreeBSDâ€™s new graphical installer.

  â€¢ Work continues to port the Vector Packet Processor (VPP) to FreeBSD. VPP is
    an open-source, high-performance user space networking stack that provides
    fast packet processing suitable for software-defined networking and network
    function virtualization applications. Look for a pending article from the
    developer working on the project, Tom Jones, that details the experience of
    porting VPP to FreeBSD.

  â€¢ BjÃ¶rn Zeeb and Cheng Cui continue their wireless work. This quarter was
    mostly focused on bug fixes and stability improvements to LinuxKPI 802.11
    and net80211. Much of this work made it into the 13.3 release.

Here is a sampling of other Foundation-sponsored development completed over the
first quarter of 2024:

  â€¢ FreeBSD was accepted in Google Summer of Code 2024 after receiving 22
    contributor proposals; on May 1, we will learn how many projects we will be
    awarded

  â€¢ OpenSSH: update to 9.6p1 then 9.7p1

  â€¢ Deprecate bsdlabel

  â€¢ Import the kernel parts of bhyve/arm64

  â€¢ Various RISC-V improvements

FreeBSD Infrastructure

A contract was completed to set up a new cluster site at NYI Chicago. You can
read about the details of that project on the Foundationâ€™s blog.

Continuous Integration and Workflow Improvement

As part of our continued support of the FreeBSD Project, the Foundation
supports a full-time staff member dedicated to improving the Projectâ€™s
continuous integration system and the test infrastructure. The full update can
be found within the quarterly status report.

Partnerships and Research

A focus of Partnerships this Quarter has been to educate the industry about the
innovations in the FreeBSD community and the impact that FreeBSD continues to
have as a cornerstone to our digital society. This is an ongoing priority, and
one we invite (encourage) everyone using and working on FreeBSD to join us in.
Greg Wallace, the Foundation Partnerships lead, is grateful for the
opportunities he has had to meet with open source and industry leaders at
Microsoft, Google, AWS, OpenSSF, Alpha-Omega, CISA, Eclipse Foundation, Open
Source Initiative, Apache Software Foundation, Rust Foundation, Red Hat, Linux
Foundation and many others to ensure they have visibility into the key role
FreeBSD plays in the global digital infrastructure. This is a role FreeBSD has
earned through its technical excellence, security by design, high availability,
simplicity of operations, commitment to open source collaboration, and
cohesiveness.

One sees these characteristics of FreeBSD in the important ongoing funded
development work such as porting VPP to FreeBSD, sponsored by RG Nets.

Ensuring industry visibility to the excellence and impact of FreeBSD is vital
to ensuring tier one support for FreeBSD across all key hardware and software
platforms. As a community, every conversation we have with people outside the
BSD communities, and every piece of content we publish, that attest to how
FreeBSD powers our individual and corporate success, brings us one step closer.

To this end, the Foundation is working on a FreeBSD Impact Report that will
aggregate the core and often mission critical role FreeBSD plays in society,
from embedded systems powered by QNX, to payments and check processing, to
digital entertainment, internet and cybersecurity infrastructure.

Our community is stepping up in innumerable ways, including to make sure
FreeBSD supports industry-standard containerized workloadsâ€‰â€”â€‰check out the Open
Container Initiative FreeBSD runtime extension working group.

The recently opened hardware vendor support survey will feed into a hardware
support guide that reflects the collective experience of all respondents that
is intended to help everyone identify hardware vendors that prioritize FreeBSD;
it will also help focus Partnerships' outreach on the priority vendors.

To close, please TELL THE WORLD YOU USE FREEBSD AND WHY. There is no wrong way
to do thisâ€‰â€”â€‰put it on your blog, on your favorite social media channel, list
FreeBSD on your companyâ€™s Open Source page, contact the Foundation about a Case
Study, etc.

Stormshield, a leading cybersecurity company based in Europe, provides a great
example of how vendors that use FreeBSD can do this. The footer of their blogs
says: "A strong supporter of Open Source, Stormshield is an active member (and
sponsor) of the FreeBSD communityâ€¦â€‹Whenever we modify Open Source software,
make patches or add features, we offer them to the community for inclusion."

Advocacy

The first quarter of 2024 marked the beginning of a new era for the Foundation
Advocacy team. We welcomed Kim McMahon in the role of Senior Director of
Advocacy and Community and also brought on two new technical writers to help
increase the frequency and depth of the FreeBSD-related content we produce.
Just some of our expanded Q1 efforts to support FreeBSD are below.

  â€¢ Began work planning the on the May 2024 FreeBSD Developer Summit,
    co-located with BSDCan, taking place May 29-30, 2024 in Ottawa, Canada

  â€¢ Introduced FreeBSD to new and returning folks at State of Open Con 24 in
    London, UK, February 6-7, 2024

  â€¢ Held an Introduction to FreeBSD half-day workshop and staffed a booth at
    SCaLE21x, which took place March 14-17, 2024 in Pasadena, CA. Thanks to
    Gordon Tetlow for his help with the workshop

  â€¢ The Foundation team also worked on a common message on the improvement and
    benefits of FreeBSD to ensure consistency between the FreeBSD Foundation
    and Core Team

  â€¢ Members of the Foundation team served as Administrators for the 2024 Google
    Summer of Code. This year marks the 20th anniversary of Google Summer of
    Code and the 20th year that the FreeBSD Project was accepted as a mentoring
    organization. The Project received 23 applications from prospective interns

  â€¢ Provided an overview of FreeBSD 13.x including the 13.3 release

  â€¢ Worked on the final report of the 2024 FreeBSD Community Survey. Be on the
    lookout for the report at the end of April

  â€¢ In partnership with Innovate UK and Digital Security by Design (DSbD), the
    Foundation held the first annual Digital Security by Design (DSbD)
    Ecosystem Beacon Awards to celebrate innovators working with and enhancing
    CheriBSD

  â€¢ Published numerous blogs including:

      â–¡ What Makes the FreeBSD Governance Model Successful

      â–¡ Guiding the future of FreeBSD releases: Colin Percival, the new Release
        Engineering Team Lead

  â€¢ Authored or participated in a number of Thought Leadership and News
    articles including:

      â–¡ The Cybersecurity Battle Has Come to Hardware

      â–¡ Ampere in the Wild: How FreeBSD Employs Ampere Arm64 Servers in the
        Data Center

      â–¡ ISAs and the Dawning Hardware Security Revolution

      â–¡ Published the March 2024 FreeBSD Update with a new look

      â–¡ Released the November/December 2023 and January/February 2024 issues of
        the FreeBSD Journal now with HTML versions of the articles

Fundraising

Thank you to everyone who gave us a financial contribution last quarter to help
fund our work to support the Project. 2024 started strong with a total of
$250,855 raised this quarter. We are grateful for your investment in FreeBSD!

Please consider supporting our efforts in 2024 by making a donation here:
https://freebsdfoundation.org/donate/.

Or, check out our Partnership opportunities here:
https://freebsdfoundation.org/our-donors/freebsd-foundation-partnership-program/.

Legal/FreeBSD IP

The Foundation owns the FreeBSD trademarks, and it is our responsibility to
protect them. We also provide legal support for the core team to investigate
questions that arise.

Go to https://freebsdfoundation.org to find more about how we support FreeBSD
and how we can help you!

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

FreeBSD Release Engineering Team

Links:
FreeBSD 13.3-RELEASE announcement URL: https://www.freebsd.org/releases/13.3R/announce/
FreeBSD 14.1-RELEASE schedule URL: https://www.freebsd.org/releases/14.1R/schedule/
FreeBSD releases URL: https://download.freebsd.org/releases/ISO-IMAGES/
FreeBSD development snapshots URL: https://download.freebsd.org/snapshots/ISO-IMAGES/

Contact: FreeBSD Release Engineering Team, <re@FreeBSD.org>

The FreeBSD Release Engineering Team is responsible for setting and publishing
release schedules for official project releases of FreeBSD, announcing code
freezes and maintaining the respective branches, among other things.

During the first quarter of the year, the Team managed 13.3-RELEASE, leading to
the final RELEASE build and announcement in March. Planning has started for the
upcoming 14.1-RELEASE cycle.

The Release Engineering Team continued providing weekly development snapshot
builds for the main, stable/14, and stable/13 branches.

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

Cluster Administration Team

Links:
Cluster Administration Team members URL: https://www.freebsd.org/administration/#t-clusteradm

Contact: Cluster Administration Team <clusteradm@FreeBSD.org>

FreeBSD Cluster Administration Team members are responsible for managing the
machines the Project relies on to synchronize its distributed work and
communications.

In this quarter, the team has worked on the following:

  â€¢ Regular support for FreeBSD.org user accounts.

  â€¢ Regular disk and parts support (and replacement) for all physical hosts and
    mirrors.

  â€¢ Set up a new mirror in Chicago.

FreeBSD Official Mirrors Overview

Current locations are Australia, Brazil, Germany, Japan (two full mirror
sites), Malaysia, South Africa, Sweden, Taiwan, United Kingdom (full mirror
site), United States of Americaâ€‰â€”â€‰California, Chicago, New Jersey (primary
site), and Washington.

The hardware and network connection have been generously provided by:

  â€¢ Bytemark Hosting (being decommissioned)

  â€¢ Cloud and SDN Laboratory at BroadBand Tower, Inc

  â€¢ Department of Computer Science, National Yang Ming Chiao Tung University

  â€¢ Equinix

  â€¢ Internet Association of Australia

  â€¢ Internet Systems Consortium

  â€¢ INX-ZA

  â€¢ KDDI Web Communications Inc

  â€¢ Malaysian Research & Education Network

  â€¢ MetaPeer

  â€¢ New York Internet

  â€¢ NIC.br

  â€¢ Teleservice SkÃ¥ne AB (new since 2023Q4)

  â€¢ Your.Org

New official mirrors are always welcome. We have noted the benefits of hosting
single mirrors at Internet Exchange Points globally, as evidenced by our
existing mirrors in Australia, Brazil, and South Africa. If you are affiliated
with or know of any organizations willing to sponsor a single mirror server,
please contact us. We are particularly interested in locations on the United
States West Coast and throughout Europe.

See generic mirrored layout for full mirror site specs and tiny-mirror for a
single mirror site.

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

Continuous Integration

Links:
FreeBSD Jenkins Instance URL: https://ci.FreeBSD.org
FreeBSD CI Tinderbox view URL: https://https://tinderbox.freebsd.org
FreeBSD CI artifact archive URL: https://artifact.ci.FreeBSD.org
Hosted CI wiki URL: https://wiki.FreeBSD.org/HostedCI
3rd Party Software CI URL: https://wiki.FreeBSD.org/3rdPartySoftwareCI
Tickets related to freebsd-testing@ URL:
https://bugs.freebsd.org/bugzilla/buglist.cgi?bug_status=open&email1=testing%40FreeBSD.org&emailassigned_to1=1&emailcc1=1&emailtype1=equals
FreeBSD CI Repository URL: https://github.com/freebsd/freebsd-ci
dev-ci Mailing List URL: https://lists.FreeBSD.org/subscription/dev-ci

Contact: Jenkins Admin <jenkins-admin@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>
Contact: freebsd-testing Mailing List
Contact: IRC #freebsd-ci channel on EFNet

In the first quarter of 2024, we worked with the project contributors and
developers to address their testing requirements. Concurrently, we collaborated
with external projects and companies to enhance their products by testing more
on FreeBSD.

Important completed tasks:

  â€¢ With help from clusteradm, the host running test VMs had disk and memory
    upgraded by reusing the parts of decommissioned machines.

  â€¢ Update the build environment of stable/13 jobs to 13.3-RELEASE.

  â€¢ Turn i386 build on main branch to use cross build on amd64.

Work in progress tasks:

  â€¢ Merging https://reviews.freebsd.org/D43786

  â€¢ Merging https://reviews.freebsd.org/D36257

  â€¢ Adding new hardware purchased by the FreeBSD Foundation to the CI cluster

  â€¢ Designing and implementing pre-commit CI building and testing and pull/
    merge-request based system (to support the workflow working group)

  â€¢ Proof of concept system is in progress.

  â€¢ Designing and implementing use of CI cluster to build release artifacts as
    release engineering does, starting with snapshot builds

  â€¢ Simplifying CI/test environment setting up for contributors and developers

  â€¢ Setting up the CI stage environment and putting the experimental jobs on it

  â€¢ Redesigning the hardware test lab and adding more hardware for testing

Open or queued tasks:

  â€¢ Collecting and sorting CI tasks and ideas

  â€¢ Setting up public network access for the VM guest running tests

  â€¢ Implementing use of bare-metal hardware to run test suites

  â€¢ Adding drm ports building tests against -CURRENT

  â€¢ Planning to run ztest tests

  â€¢ Helping more software get FreeBSD support in its CI pipeline (Wiki pages:
    3rdPartySoftwareCI, HostedCI)

  â€¢ Working with hosted CI providers to have better FreeBSD support

Please see freebsd-testing@ related tickets for more WIP information, and do
not hesitate to join the effort!

Sponsor: The FreeBSD Foundation

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

Ports Collection

Links:
About FreeBSD Ports URL:https://www.FreeBSD.org/ports/
Contributing to Ports URL: https://docs.freebsd.org/en/articles/contributing/#
ports-contributing
Ports Management Team URL: https://www.freebsd.org/portmgr/
Ports Tarball URL: http://ftp.freebsd.org/pub/FreeBSD/ports/ports/

Contact: Tobias C. Berner <portmgr-secretary@FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr@FreeBSD.org>

The Ports Management Team is responsible for overseeing the overall direction
of the Ports Tree, building packages, and personnel matters. Below is what
happened in the last quarter.

According to INDEX, there are currently 32,244 ports in the Ports Collection.
There are currently ~3,300 open ports PRs. The last quarter saw 12,991 commits
by 158 committers on the main branch and 888 commits by 61 committers on the
2024Q1 branch. Compared to last quarter, this means a large increase in the
number of commits on the main branch (up from 9,424) and slightly more
backports to the quarterly branch (up from 781). The number of ports also
increased (up from 31,942).

In Q1 there were around 14,127 commits to main: The most active committers
were:

  â€¢ 2934 sunpoet

  â€¢ 2676 bofh

  â€¢ 1297 yuri

  â€¢ 748 eduardo

  â€¢ 545 jbeich

  â€¢ 347 arrowd

  â€¢ 233 diizzy

  â€¢ 195 yasu

  â€¢ 170 ehaupt

  â€¢ 164 wen

A lot has happened in the ports tree in the last quarter, an excerpt of the
major software upgrades are:

  â€¢ pkg 1.21.0

  â€¢ New USES: ocaml

  â€¢ Default version of gcc switched to 13

  â€¢ Default version of ruby switched to 3.2

  â€¢ Default version of lazarus switched to 3.2.0

  â€¢ Default version of go switched to 1.21

  â€¢ Chromium updated to 123.0.6312.105

  â€¢ Electron-28 updated to 28.2.10

  â€¢ Electron-27 updated to 27.3.9

  â€¢ Firefox updated to 124.0.2

  â€¢ Firefox-esr updated to 115.9.1

  â€¢ KDE updated to Frameworks 5 5.115, Frameworks 6 to 6.0.0 Plasma Desktop 5
    to 5.27.11, Plasma Desktop 6 to 6.0.2

  â€¢ Qt5 updated to 5.15.13

  â€¢ Qt6 updated to 6.6.3

  â€¢ Python updated to 3.11.9, 3.10.14 and 3.8.10

  â€¢ Ruby updated to 3.2.3

  â€¢ Rust updated to 1.77.0

  â€¢ SDL updated to 2.30.2

  â€¢ Sway updated to 1.9

  â€¢ wlroots updated to 1.17.2

  â€¢ Wine updated to 9.0

  â€¢ Xorg server updated to 0.17.2

During the last quarter, FreeBSD Packages Management Team ran 17 exp-runs to
test various ports upgrades, updates to default versions of ports, subpackage
support and base system changes.

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

Projects

Projects that span multiple categories, from the kernel and userspace to the
Ports Collection or external projects.

Audio Stack Improvements

Contact: Christos Margiolis <christos@FreeBSD.org>

The FreeBSD audio stack is one of those fields that does not attract the same
attention and development as others do, since it has been left largely
unmaintained, and, although high in quality, there is still room for
improvementâ€‰â€”â€‰from lack of audio development frameworks, to missing userland
utilities and kernel driver-related bugs. This project is meant to touch on all
those areas, and as such, is more of a general improvement project, than an
implementation of a specific feature.

So far, my focus has been towards the kernel side of the audio stack, with
D43545 being probably the most requested and notable patch. I am also working
on scrapping the rather outdated "snd_clone" audio device cloning framework of
sound(4), and replacing it with DEVFS_CDEVPRIV(9) (D44411).

Some of the future tasks include:

  â€¢ Attempt to find a better (ideally automatic) way to handle snd_hda(4)
    pin-patching.

  â€¢ Implement an oss(3) library and audio(8) utility, in similar fashion to
    mixer(3) and mixer(8).

  â€¢ Write a bluetooth device management utility.

  â€¢ Improve mixer(3) and mixer(8).

  â€¢ Improve documentation and test suite where needed.

A more detailed description can be found here.

You can also follow the development process in freebsd-multimedia@, where I
post regular reports:

  â€¢ Report #1

  â€¢ Report #2

  â€¢ Report #3

  â€¢ Report #4

  â€¢ Report #5

  â€¢ Report #6

  â€¢ Report #7

  â€¢ Report #8

Sponsor: The FreeBSD Foundation

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

Bhyve Improvements

Links:
bhyve production users calls URL: https://callfortesting.org
FreeBSD Wiki - Enterprise Working Group URL: https://wiki.freebsd.org/EnterpriseWorkingGroup
FreeBSD Wiki - EWG - bhyve and jails management tooling URL: https://wiki.freebsd.org/ChrisMoerz/bhyve_management
Jan Bramkampâ€™s work on s6rc URL: http://static.bultmann.eu/s6-talk/
vmstated on GitHub URL: https://github.com/christian-moerz/vmstated
YouTube - vmstated explained URL: https://www.youtube.com/watch?v=f60NCrunXyw

Contact: Chris Moerz <freebsd@ny-central.org>

Bhyve I/O Performance Measurements

Participants of the weekly bhyve production users calls recently discussed
bhyveâ€™s I/O performance. Various ways of measuring and comparing were brought
up, however it was quickly clear that there is currently no formal analysis and
report on this. So, we started this effort in the hopes of better understanding
the various impacts of configuration options for a guest on its I/O
performance. We created a set of shell scripts that harness a FreeBSD guest for
running benchmarks/fio I/O performance measurements under various
configurations. This allows us to compare multiple criteria like bandwidth,
latency, IOPS, and more.

So far, we are testing for

  â€¢ different storage backends (i.e. ahci-hd, nvme, virtio-blk)

  â€¢ different memory settings

  â€¢ different CPU pinning options

  â€¢ different block sizes for the backing storage

  â€¢ different block sizes for accessing virtual disks

We are also pitting results for different CPU manufacturers against each other
and contrasting guest vs host performance to better understand the performance
impact of virtualization.

We plan to continue discussing our results during Michael Dexterâ€™s weekly bhyve
production users call - come join us if you are interested. We also hope to be
able to present the results at EuroBSDCon in Q3.

Bhyve Virtual Machine Tooling

Last year, Greg Wallace at the FreeBSD Foundation founded the Enterprise
Working Group with the specific goal of addressing pain points of Enterprise
users of FreeBSD. One of the work groups that emerged clustered around bhyve
and jails management tooling. After collecting a set of desired features and
functionality, one overarching key point for bhyve emerged: the desire to have
configuration concepts and tooling for bhyve like the ones available for jails.

While other desirable features were identified as well, i.e. TPM software
emulation and snapshot/restore/host-migration, the conceptual tooling question
won over those due to the lower degree of complexity and its clarity on goal
and the path on how to take steps towards it.

Technically, this means working out existing gaps around process supervision
and virtual machine state management. First steps were taken by experimenting
with existing frameworks (i.e. s6rc work by Jan Bramkamp) and
eventuallyâ€‰â€”â€‰through discussions in the weekly bhyve production userâ€™s calls
(organized by Michael Dexter)â€‰â€”â€‰this led to a proof-of-concept implementation
of "vmstated".

Started as an experiment to better understand the problem space of process
supervision and virtual machine state handling, vmstated is constructed of a
daemon and vmstatedctl management utility. It is built with base-only tooling
and libraries and leverages FreeBSD specific constructs like kqueue to minimize
its resource impact.

vmstated is configured via a UCL configuration file (similar to jails.conf)
andâ€‰â€”â€‰in combination with a bhyve_config(5) configuration fileâ€‰â€”â€‰already
provides highest flexibility in configuring virtual machines. vmstatedctl
provides a jail-like command set to start, stop, and retrieve status
information about guests. State transitions can easily be hooked via shell
scripts and allow running additional commands for network or storage set up and
tear down when relevant state changes occur.

An initial release is already in ports as sysutils/vmstated and updates are
pending commit; however, the newest version can be found on GitHub. We are
considering expanding the work; we would also like to invite anyone interested
to join us in this work! Patches, suggestions, feedback, etc. are all very much
welcome!

If you want to know more about our work, come join us at one of Michael
Dexterâ€™s weekly bhyve production users calls or reach me by email.

Documentation

We managed to update a few parts of the Handbook and Porterâ€™s Handbook (thanks
to Ed Maste, Joseph Mingrone, Pau Amma, and Rodney W. Grimes):

  â€¢ several improvements and expansions to the virtualization chapter in the
    FreeBSD Handbook

      â–¡ using a bhyve_config(5) configuration file

      â–¡ jailing bhyve

      â–¡ experimental snapshot and restore feature

      â–¡ setting up a Windows guest

  â€¢ we also have a review (D43940) up for an initial step to improving the
    bhyve man page

      â–¡ this was intentionally started with a structural update first to
        separate the many -s flag options

      â–¡ once this lands, we can move to a more widespread update to the overall
        content

Feedback is obviously very welcomeâ€‰â€”â€‰on the existing content as well as any
additional content we should be looking into!

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

Graphical Installer for FreeBSD

Links:
Slides from AsiaBSDCon 2024 URL: https://people.defora.org/~khorben/FreeBSD/bsdinstall/bsdinstall%20-%20Now%20with%20Graphics!%20-%20AsiaBSDCon%202024%20-%20WIP%20Session.pdf
gbsddialog URL: https://github.com/khorben/gbsddialog
preview video URL: https://youtu.be/jm6byc7N2O4

Contact: Pierre Pronchery <pierre@freebsdfoundation.org>

The first hurdle to overcome when testing a new Operating System is to get it
installed. What is more, the first impression new users gather from an
Operating System is its installation process. The state of the art for
Operating System installers nowadays definitely involves a graphical process.
This is the case for mainstream systems but also for other UNIX systems
comparable to FreeBSD: RedHat Enterprise Linux, Ubuntu, Debian GNU/Linux, or
even Devuan GNU+Linux Regardless of the technical level of the actual user,
this is how the platform will be compared in the public eye.

In practice, FreeBSD has already been derived as a desktop-oriented Operating
System by different projects. Of these, I only found GhostBSD as a maintained
project offering a graphical procedure to install the system. The objective
here was to consider a procedure that FreeBSD could adopt as part of its base
system, in order to ship a graphical installer much like the current installer.
However, GhostBSDâ€™s installer relies on a Gtk+ interface driven with Python,
implying a hefty footprint on the installation media when adopting FreeBSDâ€™s
usual image generation procedure. It would also imply importing and maintaining
new projects into the ports tree.

Instead, with knowledge of the current bsdinstall(8) and bsdconfig(8)
utilities, I envisioned a BSD-licensed replacement for Xdialog(1). Just like
when invoking bsdconfig with the -X switch for graphical mode, it could be
dropped in instead of bsddialog(1) and allow graphical installation - while
sharing the infrastructure of the current installer. To avoid confusion with
the current implementation of Xdialog from the x11/xdialog port, I have named
its replacement gbsddialog(1). It also has to be said that Xdialog is quite
obsolete (latest release in 2006) and this shows visually too.

After creating a Proof-of-Concept prototype past the 14.0 release, I was
provided with a 2-months window by the FreeBSD Foundation, in order to complete
a working implementation. Thanks to a few shortcuts, I was glad to present the
outcome of this effort during the WIP session of AsiaBSDCon 2024, including a
working graphical installer.

Most of the necessary patches are already available for review in FreeBSDâ€™s
Phabricator:

  â€¢ D44279 bsdinstall: implement adduser with bsddialog

  â€¢ D44280 bsdinstall: implement rootpass with bsddialog

  â€¢ D44670 bsdinstall: implement timezone with bsddialog

  â€¢ D44671 bsdinstall: allow forcing a specific partitioning mode

  â€¢ D44672 bsdinstall: obtain the dialog binary from $DIALOG

  â€¢ D44673 bsdinstall: handle command-line options in targets

  â€¢ D44674 bsdinstall: add support for graphical mode

I have tried to keep these patches in growing order of friction expected before
integration.

The most important objective of this project was to improve bsdinstall,
regardless of the success of this integration. From the items above, it should
be noted that D44279, D44280, D44670 are expecting to improve the general look
& feel of the installer, even while in text mode. Similarly, D44671 and D44672
improve the overall versatility of the installer when scripted or customized.
D44673 and D44674 bring it on par with bsdconfig -X, even allowing the
graphical installation of jails.

Some parts are still missing, or made use of shortcuts still unsuitable for
integration:

  â€¢ The "fetchmissingdists" target was avoided by shipping every component on
    the installation media;

  â€¢ The "checksum" and "extract" targets had to be re-implemented with simpler
    code, degrading the user experience also with the regular installer;

  â€¢ Creation of the installation media generates an additional, heavy image
    (almost 8 GB), and is suspected to be hindered by a bug in makefs(8).

The corresponding code can be found in my GitHub fork in the khorben/
bsdinstall-graphical4 branch. As can be guessed from the branch name, depending
on the complexity of rebasing operations, combined with the (hopefully)
progressive integration of the changes proposed, new branches may be added to
keep track of the progress. (In fact a khorben/bsdinstall-graphical5 branch
already exists.)

Still, a lot needs to be done for the installer to reach a new level of
maturity overall. While working on this project, I have received general
complaints on the installer, and calls for a complete rewrite. It is true that
the current code base suffers from a number of issues and limitations. The lack
of a graphical installer is one of many symptoms, which range from the lack of
recovery from errors, of navigability to previous steps, of a general vision of
the installation progress, or of a network-based installer. In the meantime,
this is the installer we have and are familiar with, and I think it can still
be saved and improved.

Special thanks go to Ed Maste and Joe Mingrone for the opportunity, and to
Philippe Audeoud, Tobias C. Berner, Olivier Certner, Jessica Clarke, Olivier
Cochard-LabbÃ©, Baptiste Daroussin, Brad Davis, Michael Dexter, Li-Wen Hsu,
Mateusz Piotrowski, Alfonso Siciliano, Emmanuel Vadot, and Robert Watson for
the feedback, reviews, and encouragements. (If I missed anyone, you know I did
not mean to!)

Sponsor: The FreeBSD Foundation

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

Userland

Changes affecting the base system and programs in it.

libsys

Contact: Brooks Davis <brooks@FreeBSD.org>

The libsys project removes direct system calls from libc.so and libpthread.so
(aka libthr.so) to a separate libsys.so. This will:

  â€¢ Isolate language runtimes from the details of system call implementations.

  â€¢ Better support logging and replay frameworks for systems calls.

  â€¢ Support elimination of the ability to make system calls outside trusted
    code in the runtime linker and libsys.

This work was initially inspired by a compartmentalization prototype in
CheriBSD in 2016. Ali Mashtizadeh and Tal Garfinkel picked that work up and
attempted to upstream it (D14609). Unfortunately we could not figure out how to
review and land the massive reorganization required through a phabricator
review so it languished. Last year the CHERI project once again found a need
for system call separation in a new library-based compartmentalization
framework in CheriBSD so I rebuilt the patch from scratch, committing dozens of
libc cleanups along the way. I landed the first batch of changes on February
5th. Since then I have made a number of refinements to the way we link libsys
as well as which symbols are provided in which library.

Thanks to Konstantin Belousov for many rounds of review and feedback as well as
runtime linker fixes. Thanks to Mark Johnston for runtime linker debugging and
Dimitry Andric for sanitizer fixes. Thanks also to everyone who reported bugs
and helped debug issues.

Known issues (as of the end of the reporting period)

  â€¢ The libsys ABI is not yet considered stable (it is safe to assume __sys_foo
    () will be supported so language runtimes can use it now).

  â€¢ Programs using the address sanitizer must be linked with -lsys (resolved in
    base at publication time).

TODO

  â€¢ Add a libsys.h. (See D44387 and other reviews in the stack.)

  â€¢ Update intro(2) for libsys.

  â€¢ Finalize the ABI. I am likely to reduce the set of _ (underscore) prefixed
    symbols we expose.

  â€¢ MFC the existence of libsys? It is not clear this is practical, but it
    might be possible to MFC something useful for language runtimes.

Help wanted

  â€¢ Port language runtimes that do not use libc to use libsys for system calls
    rather than rolling their own interfaces.

  â€¢ Explore limitations on where system calls can be made similar to OpenBSDâ€™s
    msyscall(2) (now obsolete) and pinsyscalls(2) (not an obvious match to our
    libsys).

Sponsor: AFRL, DARPA

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

PackageKit backend for FreeBSD pkg

Contact: Gleb Popov <arrowd@FreeBSD.org>

PackageKit is a small D-Bus daemon program that serves as a backend for
"application store" type of apps - most notably Plasma Discover and Gnome
Software Center. The latest PackageKit release features a libpkg backend, which
means that you can now use PackageKit-enabled programs on FreeBSD to manage
software. Plasma Discover is already switched to using PackageKit, so you will
get it working out of the box once you update your ports/packages.

If you observe any crashes or bugs in PackageKit please let me know by opening
an issue upstream. If you are interested in contributing, there is a lot of
work to do too!

Sponsor: Serenity Cybersecurity, LLC

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

Kernel

Updates to kernel subsystems/features, driver support, filesystems, and more.

iwlwifi(4) and wireless for 13.3-RELEASE

Links:
Categorised Wireless Problem Reports URL: https://bugs.freebsd.org/bugzilla/showdependencytree.cgi?id=277512&hide_resolved=0

Contact: Bjoern A. Zeeb <bz@FreeBSD.org>
Contact: The FreeBSD wireless mailing list <wireless@FreeBSD.org>

In the first weeks of 2024 focus was on stability for 13.3-RELEASE to finally
make iwlwifi(4) usable. The upcoming 14.1-RELEASE will benefit from this work
too. The response has since generally been positive and iwlwifi(4) supporting
chipsets up to BE200 seems mostly stable, yet still slow.

A lot of testing was provided by the FreeBSD Foundation and by many users.
Massive thanks to everyone who tested, reported back, updated PRs and helped
other users.

I have also slowly started to "categorise" more (old) wireless problem reports
and will try to continue with some spring cleaning throughout the year.

If you have questions or feedback please use the freebsd-wireless mailing list.
That way everyone will see, be able to join in, and the answers will be
publicly archived.

Sponsor: minipci.biz (BE200 hardware)

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

Architectures

Updating platform-specific features and bringing in support for new hardware
platforms.

Ten64, WHLE-LS1, and HoneyComb

Links:
My wiki page with links to some status URL: https://wiki.freebsd.org/BjoernZeeb
/

Contact: Bjoern A. Zeeb <bz@FreeBSD.ORG>

Solid-Runâ€™s HoneyComb, Traverse Technologiesâ€™s Ten64 and some versions of
Conclusive Engineeringâ€™s WHLE-LS1 all are NXP based platforms with the Data
Path Acceleration Architecture Gen2 (DPAA2).

Work has happened to support or improve support for peripherals on these
boards.

For DPAA2 I have local changes which will need review (or further discussion):

  â€¢ Cleanup of memac (MDIO) code reducing bus attachment (ACPI and FDT
    specific) code into more common code.

  â€¢ Cleanup of MC bus attachment code (again ACPI, FDT).

  â€¢ For reasons of mii_fdt.c support on some PHYs on FDT-based platforms
    restructure MAC/MII code and mostly migrate it out of the network interface
    (NI).

  â€¢ Improve Dmitry Salychevâ€™s (dsl) initial SFF/SFP code, prototyping a bus
    similar to MII for SFP with the hope that with more work it can grow into a
    larger, general FreeBSD framework and hooked it up to DPMAC.

  â€¢ With this, minimal support (still fairly hacked up) for "managed" SFP+ mode
    (using the Ten64 terminology) is usable on FDT-based systems using DAC and
    fiber cables.

  â€¢ Add more sysctl statistics to DPMAC and NI.

In short, I mostly cleaned up some of the mess I contributed to during the
initial bring-up.

For the LS1088a based WHLE-LS1 systems changes include:

  â€¢ device-tree file updates.

  â€¢ Added support for the PCA9546 I2C Switch (committed).

  â€¢ Added basic support for the PCAL6524 24-bit Fm+ I2C-bus/SMBus I/O expander.

  â€¢ Added basic support for the PCA9633 4-bit Fm+ I2C-bus LED driver to drive
    the status LEDs.

  â€¢ Added support to program the rgephy(4) LEDs (which needs to be validated).

  â€¢ Started testing the eMMC with MMCCAM and GENERIC but had trouble (needs
    further investigation, seemed fine from firmware for updates).

  â€¢ Tested one of three PCIe slots and USB fine.

For the Ten64:

  â€¢ Most of the basic lifting happened a while ago and it has generally been
    usable.

  â€¢ Detecting the VSC8514 PHY as such went in end of last year.

  â€¢ Used as the default platform to test the DPAA2 changes and SFP/SFP+ code.

In addition Pierre-Luc Drouin has overhauled the Vybrid I2C support now
attaching and working on both FDT- and ACPI-based systems (committed).

Sponsor: Traverse Technologies (Ten64 hardware a while ago, support)

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

Cloud

Updating cloud-specific features and bringing in support for new cloud
platforms.

FreeBSD on Microsoft HyperV and Azure

Links:
Microsoft Azure article on FreeBSD wiki URL: https://wiki.freebsd.org/MicrosoftAzure
Microsoft HyperV article on FreeBSD wiki URL: https://wiki.freebsd.org/HyperV

Contact: Microsoft FreeBSD Integration Services Team <bsdic@microsoft.com>
Contact: freebsd-cloud Mailing List
Contact: The FreeBSD Azure Release Engineering Team <releng-azure@FreeBSD.org>
Contact: Wei Hu <whu@FreeBSD.org>
Contact: Souradeep Chakrabarti <schakrabarti@microsoft.com>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

In this quarter, we have solved all the blocking issues and published the
13.3-RELEASE on Azure Marketplace.

Work in progress tasks:

  â€¢ Automating the image building and publishing process and merging to src/
    release/.

  â€¢ Building and publishing snapshot builds to Azure community gallery.

The above tasks are sponsored by The FreeBSD Foundation, with resources
provided by Microsoft.

Open tasks:

  â€¢ Update FreeBSD-related doc at Microsoft Learn

  â€¢ Support FreeBSD in Azure Pipelines

  â€¢ Update Azure agent port to the latest version

  â€¢ Upstream local modifications of Azure agent

  â€¢ Port Linux Virtual Machine Extensions for Azure

Sponsor: Microsoft for people in Microsoft, and for resources for the rest
Sponsor: The FreeBSD Foundation for everything else

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

FreeBSD as a Tier 1 cloud-init Platform

Links:
cloud-init Website URL: https://cloud-init.io/
cloud-init Documentation URL: https://cloudinit.readthedocs.io/en/latest/

Contact: Mina GaliÄ‡ <freebsd@igalic.co>

cloud-init is the standard way of provisioning servers in the cloud. Over the
past year and a half, thanks to this FreeBSD support has steadily improved.
This year, together with cloud-init developers and the FreeBSD Foundation, we
decided to explicitly focus on making improvements in FreeBSD itself, that will
aid the cloud-init team to test future changes to FreeBSD code-paths
themselves. To achieve this goal, I need to make FreeBSD run in LXD (and
Incus), under the control of lxd-agent (or incus-agent).

Here are some improvements from the recent weeks:

  â€¢ I have written a small testing-framework (in sh, and Iâ€™m slowly porting it
    to OpenTofu/Terraform), which installs the latest version of net/
    cloud-init-devel or net/cloud-init and runs a couple of standard cloud-init
    tests.

  â€¢ To do this, I have created a dedicated public repository which contains the
    latest versions of net/cloud-init-devel and net/cloud-init for FreeBSD 13
    and 14 on amd64 and aarch64.

  â€¢ I have ported Linuxâ€™s vsock testing framework to FreeBSD

  â€¢ I created a driver skeleton for a VirtIO Socket driver, based on the HyperV
    Socket driver.

  â€¢ In doing so, I made numerous improvements to HyperV sockets, some of which
    are accepted, others still need more work.

  â€¢ I have tested and released the latest 24.1 series cloud-init, where the
    cloud-init team and I have finally fixed some longstanding bugs, such as
    moving /run/cloud-init to /var/run/cloud-init on BSD, as well as fixing the
    homedir argument to user_groups to actually do something.

  â€¢ This release also sees numerous fixes to the OpenBSD code-paths from the
    community and not just me.

  â€¢ I have also started an official port for OpenBSD, but that work has stalled
    .

The work to come, in broad strokes:

  â€¢ Finish the FreeBSD VirtIO Socket driver.

  â€¢ Fix Goâ€™s runtime to support VirtIO on FreeBSD.

  â€¢ Port lxd-agentâ€™s dependencies to FreeBSD.

  â€¢ Port lxd-agent to FreeBSD.

That work will be interspersed with more improvements to cloud-init on BSDs,
and more tests on different cloud providers.

Sponsor: The FreeBSD Foundation

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

OpenStack on FreeBSD

Links:
OpenStack URL: https://www.openstack.org/
OpenStack on FreeBSD URL: https://github.com/openstack-on-freebsd

Contact: Chih-Hsin Chang <starbops@hey.com>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

The OpenStack on FreeBSD project aims to seamlessly integrate OpenStack cloud
infrastructure with the FreeBSD operating system. It uses FreeBSDâ€™s unique
features while ensuring compatibility with OpenStack standards.

In the first quarter of 2024, we made significant progress on the OpenStack on
FreeBSD project. This included submitting a proposal for BSDCan 2024 and
attending AsiaBSDCon 2024 to share our porting experiences and gain exposure
for the project. The feedback received at AsiaBSDCon was particularly valuable
and helped in refining the projectâ€™s direction. During this period, we also
reviewed the projectâ€™s phase 1 tasks and made necessary adjustments. We also
planned for phases 2 and 3, aligning them with the projectâ€™s long-term goals.
One technical achievement was verifying the functionality of bhyve serial
console over TCP, an important part of the projectâ€™s infrastructure.
Additionally, we created a demo video showcasing the projectâ€™s progress and
features.

Looking ahead, our focus for the next quarter includes confirming the
feasibility of implementing FreeBSD privilege-management user space tools
leveraging mac(4) and priv(9), simplifying installation steps by transitioning
to FreeBSD ports, and porting OpenStack Ironic to FreeBSD. These tasks will
enhance the projectâ€™s capabilities and compatibility.

Sponsor: The FreeBSD Foundation

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

Documentation

Noteworthy changes in the documentation tree, manual pages, or new external
books/documents.

Documentation Engineering Team

Link: FreeBSD Documentation Project URL: https://www.freebsd.org/docproj/
Link: FreeBSD Documentation Project Primer for New Contributors URL: https://docs.freebsd.org/en/books/fdp-primer/
Link: Documentation Engineering Team URL: https://www.freebsd.org/administration/#t-doceng

Contact: FreeBSD Doceng Team <doceng@FreeBSD.org>

The doceng team is a body to handle some of the meta-project issues associated
with the FreeBSD Documentation Project; for more information, see FreeBSD
Doceng Team Charter.

During the last quarter:

Edward Tomasz NapieraÅ‚a's doc commit bit was taken for safekeeping. Tom Rhodes
's doc commit bit was taken for safekeeping.

FreeBSD Translations on Weblate

Link: Translate FreeBSD on Weblate URL: https://wiki.freebsd.org/Doc/Translation/Weblateurl
Link: FreeBSD Weblate Instance URL: https://translate-dev.freebsd.org/url

Q1 2024 Status

  â€¢ 17 team languages

  â€¢ 189 registered users

Three new translators joined Weblate:

  â€¢ piker3 in Polish team (pl)

  â€¢ chrislongros in Greek team (el)

  â€¢ grip in Italian team (it_IT)

Languages

  â€¢ Chinese (Simplified) (zh-cn) (progress: 7%)

  â€¢ Chinese (Traditional) (zh-tw) (progress: 3%)

  â€¢ Dutch (nl) (progress: 1%)

  â€¢ French (fr) (progress: 1%)

  â€¢ German (de) (progress: 1%)

  â€¢ Greek (el) (progress: 1%)

  â€¢ Indonesian (id) (progress: 1%)

  â€¢ Italian (it) (progress: 5%)

  â€¢ Korean (ko) (progress: 32%)

  â€¢ Norwegian (nb-no) (progress: 1%)

  â€¢ Persian (fa-ir) (progress: 3%)

  â€¢ Polish (progress: 2%)

  â€¢ Portuguese (progress: 0%)

  â€¢ Portuguese (pt-br) (progress: 22%)

  â€¢ Spanish (es) (progress: 36%)

  â€¢ Turkish (tr) (progress: 2%)

We want to thank everyone that contributed, translating or reviewing documents.

And please, help promote this effort on your local user group, we always need
more volunteers.

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

Ports

Changes affecting the Ports Collection, whether sweeping changes that touch
most of the tree, or individual ports themselves.

FreshPorts: Notification of new packages

Links:
FreshPorts URL: https://freshports.org/
FreshPorts blog URL: https://news.freshports.org/

Contact: Dan Langille <dvl@FreeBSD.org>

FreshPorts and FreshSource have reported upon FreeBSD commits for 20 years.
They cover all commits, not just ports.

FreshPorts tracks the commits and extracts data from the port Makefiles to
create a database of information useful to both port maintainers and port
users.

For example, https://www.freshports.org/security/acme.sh/#history shows the
history of the security/acme.sh port, back to its creation in May 2017. Also
available are dependencies, flavors, configuration options, and available
packages. All of this is useful for both users and developers of ports.

Notification: New Package Available

One of the original features of FreshPorts is notification of ports updates.
You can create a list of ports and receive notifications about those ports.
This new feature can also notify when a new package is available for that port.
The use case: a known security vulnerability has been patched. FreshPorts will
tell you the port has been patched, and then you wait for the package. This new
feature will tell you when that package is available.

Details at:

  â€¢ https://github.com/FreshPorts/freshports/issues/542

Help Needed

It has been over 23 years since FreshPorts started. Others must take over
eventually. I have started that process recently. There are several aspects to
FreshPorts:

  â€¢ FreeBSD admin (updating the OS and packages)

  â€¢ front end code (website - mostly PHP)

  â€¢ back end code (commit processing - Perl, Python, shell)

  â€¢ database design (PostgreSQL).

The database does not change very often and requires little maintenance
compared to the applications and OS. The website pretty much runs itself. From
time to time, a change to the FreeBSD ports infrastructure breaks something or
requires a modification, but there is rarely any urgency to fix that. This is
not a huge time commitment. There is a lot of learning. While not a complex
application, FreshPorts is also not trivial.

To contribute, please join the https://lists.freshports.org/mailman/listinfo/freshports-coders
mailing list and let us know what you would like to help
with.

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

GCC on FreeBSD

Links:
GCC Project URL: https://gcc.gnu.org/
GCC 10 release series URL: https://gcc.gnu.org/gcc-10/
GCC 11 release series URL: https://gcc.gnu.org/gcc-11/
GCC 12 release series URL: https://gcc.gnu.org/gcc-12/
GCC 13 release series URL: https://gcc.gnu.org/gcc-13/

Contact: Lorenzo Salvadore <salvadore@FreeBSD.org>

Updating GCC default version to 13 is finally finished. Thanks to Antoine
Brodin who ran the exp-runs and to all other developers and ports maintainers
involved.

As promised in the preceding report, the next goal is to reduce the number of
open bugs for GCC ports. Some work on existing bugs has already started.

In particular, lang/gcc14-devel has long stayed out of date due to some issues
with building the port without any BOOTSTRAP option. Thanks to the help of
other developers and contributors (a special thank to Mark Millard), I noticed
that according to the official documentation building GCC without bootstrap
requires a working GCC binary and thus I switched lang/gcc14-devel to require
that a BOOTSTRAP option is set. However it has later been stated that
bootstrapping GCC using clang and libc++ is officially supported. But it has
also been stated that this is not a high priority.

At the moment lang/gcc14-devel is the only GCC port requiring a BOOTSTRAP
option to be set. The plan is to have all GCC ports for versions greater or
equal than 14 (i.e. future GCC ports) to require such an option: even if
building without bootstrap is more or less officially supported, being low
priority for upstream it increases the burden of maintaining GCC ports for low
results. In case lower versions start to have issues building without
bootstrap, I am going to require bootstrap for those as well.

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

Valgrind: port to arm64 on its way

Links:
Valgrind Home Page URL: https://www.valgrind.org/
Valgrind News URL: https://www.valgrind.org/docs/manual/dist.news.html
arm64 port URL: https://github.com/paulfloyd/freebsdarm64_valgrind

Contact: Paul Floyd <pjfloyd@wanadoo.fr>

The major news, as per the title, is that a port to FreeBSD arm64 (or aarch64)
is now ready. The next steps are to get it reviewed and pushed upstream.

Valgrind 3.23 is due out at the end of April 2024 and devel/valgrind will be
updated shortly after that.

devel/valgrind-devel will get an update as soon as I have pushed the changes
for arm64.

--track-fds=yes now checks for and warns about attempts to close a file
descriptor more than once. Handling of closefrom has been improved to use this
feature.

There are some important fixes for FreeBSD 15, in particular handling the new
libsys.

Here is a list of smaller bugfixes:

  â€¢ Support for FreeBSD 13.3 has been added.

  â€¢ Added a redirect for reallocarray.

  â€¢ Several fixes for aio* functions.

  â€¢ Added a redirect for memccpy.

  â€¢ There is a fix for _umtx_op OP_ROBUST_LISTS.

  â€¢ Added redirects for C23 free_sized and free_aligned_sized.

  â€¢ Correctly propagate the ELF stack protection flags to the guest stack that
    Valgrind synthesizes.

  â€¢ Fixes for --sanity-level-3 and above (only used for Valgrind self-testing
    at runtime).

  â€¢ Several fixes to checking done for semctl.

  â€¢ Fixed argument checking for utrace.

  â€¢ Fixed argument checking for clock_nanosleep.

â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”

Third Party Projects

Many projects build upon FreeBSD or incorporate components of FreeBSD into
their project. As these projects may be of interest to the broader FreeBSD
community, we sometimes include brief updates submitted by these projects in
our quarterly report. The FreeBSD project makes no representation as to the
accuracy or veracity of any claims in these submissions.

Containers and FreeBSD: Pot, Potluck and Potman

Links:
Pot organization on GitHub URL: https://github.com/bsdpot

Contact: Luca Pizzamiglio (Pot) <pizzamig@FreeBSD.org>
Contact: Bretton Vine (Potluck) <bv@honeyguide.eu>
Contact: Michael Gmelin (Potman) <grembo@FreeBSD.org>

Pot is a jail management tool that also supports orchestration through Nomad.
Potluck aims to be to FreeBSD and Pot what Dockerhub is to Linux and Docker: a
repository of Pot flavours and complete container images for usage with Pot and
in many cases Nomad.

During this quarter, there were no new Pot releases.

Potluck saw quite some activity though. Not only have the images been rebuilt
for FreeBSD 14, but also the new Adminer container has been submitted by
first-time contributor Sidicer. Additionally a large number of additional
features, updates and fixes have been committed to containers like
HAProxy-Consul, Grafana, PostgreSQL-Patroni, or Prometheus.

For the Mastodon container, a blog post has been published explaining how to
use it to run your own instance.

As always, feedback and patches are welcome.

Sponsors: Nikulipe UAB, Honeyguide Group

From nobody Sun May  5 17:10:58 2024
X-Original-To: stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VXWK30NGtz5KKdH
	for <stable@mlmmj.nyi.freebsd.org>; Sun, 05 May 2024 17:11:03 +0000 (UTC)
	(envelope-from fm-134872-20240505171059316ffe65fa99388fc8-Lyi5KX@rts-flowmailer.siemens.com)
Received: from mta-65-228.siemens.flowmailer.net (mta-65-228.siemens.flowmailer.net [185.136.65.228])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VXWK20C8Bz47JM
	for <stable@freebsd.org>; Sun,  5 May 2024 17:11:01 +0000 (UTC)
	(envelope-from fm-134872-20240505171059316ffe65fa99388fc8-Lyi5KX@rts-flowmailer.siemens.com)
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=siemens.com header.s=fm1 header.b=WlPXOJZ4;
	dmarc=pass (policy=reject) header.from=siemens.com;
	spf=pass (mx1.freebsd.org: domain of fm-134872-20240505171059316ffe65fa99388fc8-Lyi5KX@rts-flowmailer.siemens.com designates 185.136.65.228 as permitted sender) smtp.mailfrom=fm-134872-20240505171059316ffe65fa99388fc8-Lyi5KX@rts-flowmailer.siemens.com
Received: by mta-65-228.siemens.flowmailer.net with ESMTPSA id 20240505171059316ffe65fa99388fc8
        for <stable@freebsd.org>;
        Sun, 05 May 2024 19:10:59 +0200
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1;
 d=siemens.com; i=Andre.Albsmeier@siemens.com;
 h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Cc;
 bh=iLi/2NB42QfkE2UMrMqP9ps8AwmFXZb0qdmKA/CjT+8=;
 b=WlPXOJZ4wqVlWyScGW6uuxUuU4POgQjaAXchgHW7b5qEaxcLFyLxwJLn4uj3PeBxBQmlXy
 rQKIapli30UEJ/dQ6g7pcJpDx6nWUbCsJmoSGImoOBr8WKsLOCGCJ9L/H0vLuVFi7F0J1R60
 EvsBzb3VnKidiQCtnQPfra7ppcDm0=;
Date: Sun, 5 May 2024 19:10:58 +0200
From: Andre Albsmeier <Andre.Albsmeier@siemens.com>
To: stable@freebsd.org
Cc: Andre.Albsmeier@siemens.com
Subject: 14-STABLE crashes when using geli from automountd executable maps
Message-ID: <Zje9ohAwBBgwR-SE@bali.c4ef04bb578971607fc6a73f3188a722>
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
document_confidentiality: Restricted
X-Flowmailer-Platform: Siemens
Feedback-ID: 519:519-134872:519-21489:flowmailer
X-Spamd-Bar: -----
X-Spamd-Result: default: False [-5.90 / 15.00];
	DWL_DNSWL_MED(-2.00)[siemens.com:dkim];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-1.00)[-0.999];
	DMARC_POLICY_ALLOW(-0.50)[siemens.com,reject];
	FORGED_SENDER(0.30)[Andre.Albsmeier@siemens.com,fm-134872-20240505171059316ffe65fa99388fc8-Lyi5KX@rts-flowmailer.siemens.com];
	R_DKIM_ALLOW(-0.20)[siemens.com:s=fm1];
	R_SPF_ALLOW(-0.20)[+ip4:185.136.65.224/29];
	RWL_MAILSPIKE_VERYGOOD(-0.20)[185.136.65.228:from];
	MIME_GOOD(-0.10)[text/plain];
	RCVD_VIA_SMTP_AUTH(0.00)[];
	ASN(0.00)[asn:50018, ipnet:185.136.65.0/24, country:NL];
	MIME_TRACE(0.00)[0:+];
	RCVD_COUNT_ONE(0.00)[1];
	MISSING_XM_UA(0.00)[];
	RCVD_TLS_ALL(0.00)[];
	RCPT_COUNT_TWO(0.00)[2];
	MLMMJ_DEST(0.00)[stable@freebsd.org];
	FROM_NEQ_ENVFROM(0.00)[Andre.Albsmeier@siemens.com,fm-134872-20240505171059316ffe65fa99388fc8-Lyi5KX@rts-flowmailer.siemens.com];
	FROM_HAS_DN(0.00)[];
	ARC_NA(0.00)[];
	TO_DN_NONE(0.00)[];
	PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	DKIM_TRACE(0.00)[siemens.com:+]
X-Rspamd-Queue-Id: 4VXWK20C8Bz47JM

Before opening a PR, let's see if I'm doing something bad here (shouldn't
be that bad as it worked on FreeBSD-12 :-)).

I can reliably crash 14-STABLE by running "geli attach" from an automountd
executable map if 

a) there is no other geli device in use
b) the mount is done r/w.

To reproduce:

0. Prerequisite:
----------------

We assume there is a working autofs environment and /etc/auto_master is the
master map. The must be no geli device in use.


1. Do this once:
----------------

cat << EOF > /etc/autocrash
#!/bin/sh

case "x\$1" in

  xno_geli)
    exec echo "-fstype=ufs,noatime,async :/dev/md0.eli"
    ;;

  xgeli_ro)
    geli attach -k /bin/ls -p /dev/md0
    exec echo "-fstype=ufs,noatime,async,ro :/dev/md0.eli"
    ;;

  xgeli_rw)
    geli attach -k /bin/ls -p /dev/md0
    exec echo "-fstype=ufs,noatime,async :/dev/md0.eli"
    ;;

esac
EOF

chmod 755 /etc/autocrash

echo "/autocrash autocrash" >> /etc/auto_master

automount


2. Do this each time you want to crash the box:
-----------------------------------------------

kldload geom_eli.ko

dd if=/dev/zero of=/tmp/testcrash bs=64k count=160
mdconfig -a -t vnode -f /tmp/testcrash
geli init -P -K /bin/ls /dev/md0
geli attach -k /bin/ls -p /dev/md0
newfs /dev/md0.eli

# this works w/o crashing (as md0.eli is still attached)
cd /autocrash/no_geli ; ls -la
cd / && umount /autocrash/no_geli
geli detach /dev/md0

# crash it:
cd /autocrash/geli_rw
ls -la

# this also works w/o crashing (as we mount it readonly)
cd /autocrash/geli_ro ; ls -la
cd / && umount /autocrash/geli_ro
geli detach /dev/md0



For some reasons, the box crashes when "geli attach" is executed from within
the /etc/autocrash script. It does not crash when we attach it before and do
only the mount from /etc/autocrash. It also does not crash if there is at
least one more geli device in use.

Debugging the crash gives us this:

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address   = 0x218
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff8071a20a
stack pointer           = 0x28:0xfffffe00743d5da0
frame pointer           = 0x28:0xfffffe00743d5dd0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 1246 (g_eli[1] md0)
rdi: 0000000000000000 rsi: 0000000000000200 rdx: 0000000000000001
rcx: 0000000000000080  r8: 0000000000000001  r9: 0000000000010000
rax: fffff80006da5000 rbx: fffff800063904b0 rbp: fffffe00743d5dd0
r10: 0000000000000001 r11: 0000000000010000 r12: 0000000000000200
r13: 0000000000000008 r14: 0000000000000000 r15: 0000000000000000
trap number             = 12
panic: page fault
cpuid = 1
time = 1714666291
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00743d5b80
vpanic() at vpanic+0xfa/frame 0xfffffe00743d5bb0
panic() at panic+0x43/frame 0xfffffe00743d5c10
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe00743d5c70
trap_pfault() at trap_pfault+0xab/frame 0xfffffe00743d5cd0
calltrap() at calltrap+0x8/frame 0xfffffe00743d5cd0
--- trap 0xc, rip = 0xffffffff8071a20a, rsp = 0xfffffe00743d5da0, rbp = 0xfffffe00743d5dd0 ---
uma_zalloc_arg() at uma_zalloc_arg+0x3a/frame 0xfffffe00743d5dd0
g_eli_alloc_data() at g_eli_alloc_data+0x49/frame 0xfffffe00743d5df0
g_eli_crypto_run() at g_eli_crypto_run+0x97/frame 0xfffffe00743d5e90
g_eli_worker() at g_eli_worker+0x369/frame 0xfffffe00743d5ef0
fork_exit() at fork_exit+0x82/frame 0xfffffe00743d5f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00743d5f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Uptime: 31s
Dumping 212 out of 3438 MB:..8%..16%..23%..31%..46%..53%..61%..76%..83%..91
__curthread () at /src/src-14/sys/amd64/include/pcpu_aux.h:57
57              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) where
#0  __curthread () at /src/src-14/sys/amd64/include/pcpu_aux.h:57
#1  doadump (textdump=textdump@entry=1) at /src/src-14/sys/kern/kern_shutdown.c:405
#2  0xffffffff804f4e40 in kern_reboot (howto=260) at /src/src-14/sys/kern/kern_shutdown.c:523
#3  0xffffffff804f5307 in vpanic (fmt=0xffffffff8081968d "%s", ap=ap@entry=0xfffffe00743d5bf0) at /src/src-14/sys/kern/kern_shutdown.c:967
#4  0xffffffff804f50e3 in panic (fmt=<unavailable>) at /src/src-14/sys/kern/kern_shutdown.c:891
#5  0xffffffff807b0d7c in trap_fatal (frame=0xfffffe00743d5ce0, eva=536) at /src/src-14/sys/amd64/amd64/trap.c:952
#6  0xffffffff807b0e2b in trap_pfault (frame=0xfffffe00743d5ce0, usermode=false, signo=<optimized out>, ucode=0x0)
    at /src/src-14/sys/amd64/amd64/trap.c:760
#7  <signal handler called>
#8  uma_zalloc_arg (zone=0x0, udata=udata@entry=0x0, flags=1) at /src/src-14/sys/vm/uma_core.c:3738
#9  0xffffffff8045a7b9 in uma_zalloc (zone=0x0, zone@entry=0xfffff800063904b0, flags=1) at /src/src-14/sys/vm/uma.h:367
#10 g_eli_alloc_data (bp=bp@entry=0xfffff800063904b0, sz=4096) at /src/src-14/sys/geom/eli/g_eli.c:958
#11 0xffffffff80463c27 in g_eli_crypto_run (wr=wr@entry=0xfffff800068c4880, bp=bp@entry=0xfffff800063904b0)
    at /src/src-14/sys/geom/eli/g_eli_privacy.c:282
#12 0xffffffff8045c6a9 in g_eli_worker (arg=arg@entry=0xfffff800068c4880) at /src/src-14/sys/geom/eli/g_eli.c:752
#13 0xffffffff804b6b02 in fork_exit (callout=0xffffffff8045c340 <g_eli_worker>, arg=0xfffff800068c4880, frame=0xfffffe00743d5f40)
    at /src/src-14/sys/kern/kern_fork.c:1159
#14 <signal handler called>
(kgdb) up 10
#10 g_eli_alloc_data (bp=bp@entry=0xfffff800063904b0, sz=4096) at /src/src-14/sys/geom/eli/g_eli.c:958
958                     bp->bio_driver2 = uma_zalloc(g_eli_uma, M_NOWAIT |
(kgdb) p g_eli_uma
$1 = (uma_zone_t) 0x0
(kgdb)

g_eli_uma being NULL is probably wrong. Probably it got free'ed but shouldn't
or some initialisation is missing.  But why is it NULL when called from
/etc/autocrash and not when run manually or if another geli device already
exists?

From nobody Sun May  5 22:32:40 2024
X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VXfS91t1Wz5Ktd4;
	Sun, 05 May 2024 22:32:41 +0000 (UTC)
	(envelope-from cperciva@FreeBSD.org)
Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "freefall.freebsd.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4VXfS91GJfz4rnV;
	Sun,  5 May 2024 22:32:41 +0000 (UTC)
	(envelope-from cperciva@FreeBSD.org)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim;
	t=1714948361; h=from:from:reply-to:reply-to:subject:subject:date:date:
	 message-id:message-id:to:to:cc:cc;
	bh=t8KFtWkGv3QVC8L2F2oIc5iQdTgoLkVs/Apib/iNR5s=;
	b=qkY/NbCxO8Z8gafTk9bpqxZ9PBsALfEwC17OwE7B43wib4UGaxmNzTL9XkoeoyN1zIbgp3
	V0kiQjR6KWFIQi1V9NZ6Rl+aTXPsX3GszKBd2ecN1GIRTV7YfJ0VO+2tmODm0RkrB4pfIc
	SJzgZSIMJRKVCKgNoeEciy2vX5IEAOAn2cvprDAB1TOKKyX6DDIZbWnaOEJ344qQEVNOeq
	zGszPnzwmTHjK+5wYLQw28t+TD1sEYvl1iIanjCRRr4VqM5ugNdHUXDvsWHblKCzw9l7wH
	QCUrDolbLqDF/Sx+ixOUIYqYf12vPDl5q8TkkKvKptApKI4qI6GkM3sZdq0vtg==
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1714948361; a=rsa-sha256; cv=none;
	b=Rhy7CrjqHPA40nXCd+EY4d1bZi5iEojD5bab2RA09bdzjBQCfjXGyvKnCGIARyZAbxppFj
	KJLKJKCXwKhiFTO55nJF2naMF5DN/On/+naSy1UT1lCq/o2bhShFRxhnF74n3llRo7sj5F
	Kz0NOCF1fpgMeQzBITkDRWsz1kAbumi2aGYRcC7qR+pf1hSm68bhvLJZDiCyRTb0B0rzNY
	0b0y26fSRqW6NcRKCjS5V8abtYymNzZoYlfgbAt8/Ypf3onOovzC25l7+rlnHZsqVj69ez
	zyvkrprcvdkKry2Tn27n2OPngCFJun8wAJlxycBwQUhqhgWt0Poszi/orGLPIQ==
ARC-Authentication-Results: i=1;
	mx1.freebsd.org;
	none
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
	s=dkim; t=1714948361; h=from:from:reply-to:reply-to:subject:subject:date:date:
	 message-id:message-id:to:to:cc:cc;
	bh=t8KFtWkGv3QVC8L2F2oIc5iQdTgoLkVs/Apib/iNR5s=;
	b=bTFDe8InbZbgzvASPmGEmfJ9qe3ZGQKVRVUnNuGSkjF7f+YEMOeDtqeuAIRfTV4ppMSNRo
	RxTlmj3hmKl0sPIsp2xtvA3tsqQ8h0MyVqH1NHIWZ5qWF8gDjGUyIH+HTMM3nx7/3YpcSJ
	zOy0hJagUuMCoohsv3eym+/xfdlsofTLavDcEbQVwryGrXjBTjybtcH7lQAgyipbEsn6Ve
	zIG2ihw5x9a5/VxCnnA2DJOB80SUGW/1w9rzhtXLsIBraWTmzyu4AV2AfPcIngYLfiYpUw
	DP9wcWQaA70LQP7trk7vp3uIWR8q3IwgxdK3zYQ5+cyy0Jm5QJeTrCEUv3YYzw==
Received: by freefall.freebsd.org (Postfix, from userid 1002)
	id 0EF64E626; Sun, 05 May 2024 22:32:41 +0000 (UTC)
To: freebsd-snapshots@FreeBSD.org, freebsd-stable@FreeBSD.org
Cc: FreeBSD Release Engineering Team <re@FreeBSD.org>
Reply-To: FreeBSD Release Engineering Team <re@FreeBSD.org>
Subject: FreeBSD 14.1-BETA1 Now Available
Message-Id: <20240505223241.0EF64E626@freefall.freebsd.org>
Date: Sun, 05 May 2024 22:32:40 +0000 (UTC)
From: Colin Percival <cperciva@FreeBSD.org>
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-stable
List-Help: <mailto:stable+help@freebsd.org>
List-Post: <mailto:stable@freebsd.org>
List-Subscribe: <mailto:stable+subscribe@freebsd.org>
List-Unsubscribe: <mailto:stable+unsubscribe@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
Sender: owner-freebsd-stable@FreeBSD.org

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

The first BETA build of the 14.1-RELEASE release cycle is now available.

Installation images are available for:

o 14.1-BETA1 amd64 GENERIC
o 14.1-BETA1 i386 GENERIC
o 14.1-BETA1 powerpc GENERIC
o 14.1-BETA1 powerpc64 GENERIC64
o 14.1-BETA1 powerpc64le GENERIC64LE
o 14.1-BETA1 powerpcspe MPC85XXSPE
o 14.1-BETA1 armv7 GENERICSD
o 14.1-BETA1 aarch64 GENERIC
o 14.1-BETA1 aarch64 RPI
o 14.1-BETA1 aarch64 PINE64
o 14.1-BETA1 aarch64 PINE64-LTS
o 14.1-BETA1 aarch64 PINEBOOK
o 14.1-BETA1 aarch64 ROCK64
o 14.1-BETA1 aarch64 ROCKPRO64
o 14.1-BETA1 riscv64 GENERICSD

Note regarding arm SD card images: For convenience for those without
console access to the system, a freebsd user with a password of
freebsd is available by default for ssh(1) access.  Additionally,
the root user password is set to root.  It is strongly recommended
to change the password for both users after gaining access to the
system.

Installer images and memory stick images are available here:

    https://download.freebsd.org/releases/ISO-IMAGES/14.1/

The image checksums follow at the end of this e-mail.

If you notice problems you can report them through the Bugzilla PR
system or on the -stable mailing list.

If you would like to use Git to do a source based update of an existing
system, use the "releng/14.1" branch.

A list of changes since 14.0 will be available in the releng/14.1
release notes:

    https://www.freebsd.org/releases/14.1R/relnotes/

Please note, the release notes page is not yet complete, and will be
updated on an ongoing basis as the 14.1-RELEASE cycle progresses.

=== Virtual Machine Disk Images ===

VM disk images are available for the amd64, i386, and aarch64
architectures.  Disk images may be downloaded from the following URL
(or any of the FreeBSD download mirrors):

    https://download.freebsd.org/releases/VM-IMAGES/14.1-BETA1/

BASIC-CI images can be found at:

    https://download.freebsd.org/releases/CI-IMAGES/14.1-BETA1/

The partition layout is:

    ~ 16 kB - freebsd-boot GPT partition type (bootfs GPT label)
    ~ 1 GB  - freebsd-swap GPT partition type (swapfs GPT label)
    ~ 20 GB - freebsd-ufs GPT partition type (rootfs GPT label)

The disk images are available in QCOW2, VHD, VMDK, and raw disk image
formats.  The image download size is approximately 135 MB and 165 MB
respectively (amd64/i386), decompressing to a 21 GB sparse image.

Note regarding arm64/aarch64 virtual machine images: a modified QEMU EFI
loader file is needed for qemu-system-aarch64 to be able to boot the
virtual machine images.  See this page for more information:

    https://wiki.freebsd.org/arm64/QEMU

To boot the VM image, run:

    % qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt  \
	-bios QEMU_EFI.fd -serial telnet::4444,server -nographic \
	-drive if=none,file=VMDISK,id=hd0 \
	-device virtio-blk-device,drive=hd0 \
	-device virtio-net-device,netdev=net0 \
	-netdev user,id=net0

Be sure to replace "VMDISK" with the path to the virtual machine image.

=== Amazon EC2 AMI Images ===

FreeBSD/amd64 EC2 AMI IDs can be retrieved from the Systems Manager
Parameter Store in each region using the keys:

	/aws/service/freebsd/amd64/base/ufs/14.1/BETA1
	/aws/service/freebsd/amd64/base/zfs/14.1/BETA1
	/aws/service/freebsd/amd64/cloud-init/ufs/14.1/BETA1
	/aws/service/freebsd/amd64/cloud-init/zfs/14.1/BETA1

FreeBSD/aarch64 EC2 AMI IDs can be retrieved from the Systems Manager
Parameter Store in each region using the keys:

	/aws/service/freebsd/arm64/base/ufs/14.1/BETA1

ZFS and cloud-init AMIs for aarch64 are not available for 14.1-BETA1 but
should be available starting with 14.1-BETA2.

=== Vagrant Images ===

FreeBSD/amd64 images are available on the Hashicorp Atlas site, and can
be installed by running:

    % vagrant init freebsd/FreeBSD-14.1-BETA1
    % vagrant up

=== Upgrading ===

The freebsd-update(8) utility supports binary upgrades of amd64, i386,
and aarch64 systems running earlier FreeBSD releases.  Systems running
earlier FreeBSD releases can upgrade by first installing any updates for
the currently running release:

	# freebsd-update fetch
	# freebsd-update install

and then downloading the new release:

	# freebsd-update upgrade -r 14.1-BETA1

During this process, freebsd-update(8) may ask the user to help by
merging some configuration files or by confirming that the automatically
performed merging was done correctly.

	# freebsd-update install

The system must be rebooted with the newly installed kernel before
continuing.

	# shutdown -r now

After rebooting, freebsd-update needs to be run again to install the new
userland components:

	# freebsd-update install

It is recommended to rebuild and install all applications if possible,
especially if upgrading from an earlier FreeBSD release, for example,
FreeBSD 12.x.  Alternatively, the user can install misc/compat12x and
other compatibility libraries, afterwards the system must be rebooted
into the new userland:

	# shutdown -r now

Finally, after rebooting, freebsd-update needs to be run again to remove
stale files:

	# freebsd-update install

== ISO CHECKSUMS ==

o 14.1-BETA1 amd64 GENERIC:
  SHA512 (FreeBSD-14.1-BETA1-amd64-bootonly.iso) = 2e7442b96685eb525b9c1b2d5ec795b9124b2b7e9482152b5ad19cce3b31032c47b48fd4bf31e8068d798761d31ca232b8537806f7da206672943969f9ebec2e
  SHA512 (FreeBSD-14.1-BETA1-amd64-bootonly.iso.xz) = 8dcd9b2fd1dcd01adc8b7dfb3715952676020034823ed982250f8aec77f2d5691a1c6f705ebc5bc4a26af57333c5ccd086c9251a7cc2f9ef5678dc3168b1b596
  SHA512 (FreeBSD-14.1-BETA1-amd64-disc1.iso) = df819f4685d13f6cb55a884503b217b5bf26e46051572ced867195b8cd2075389c785263b7eff3a53e361de0e49b12fd715ca8e613430492fab8410ad185cc8f
  SHA512 (FreeBSD-14.1-BETA1-amd64-disc1.iso.xz) = 9aad81b33187f6fd20b2f2e236b1ba03c224852799bd0adfc3f613fffbeb164c5db608ad9796fd1434248e9c69a6307b7f3dca572ad5f72e682adfe5f2307b66
  SHA512 (FreeBSD-14.1-BETA1-amd64-dvd1.iso) = 13bea9e9561b0969355ee042221d982efcd07fce6864b0d934572942b6c675c254003e842a9acf05e829dae6bd49d0b3b888da868cabc379543c3bbfb8955aef
  SHA512 (FreeBSD-14.1-BETA1-amd64-dvd1.iso.xz) = ac5d557eeebde625ca1b7dd888f4bccf090e27457e2aa59458b472a3d0ed8a91f6c1ac6137d7cd8ab9d97e45e0c33ad64b519eba9a52793f708c56d26c2cd727
  SHA512 (FreeBSD-14.1-BETA1-amd64-memstick.img) = c58a95cd335ac38116b6ec0df59287753ab5b8145cadcdb219321a0b50e86d8918776e11c31e3acef7fb8538f084920de4a592ef6b822a5362e27b1be8267400
  SHA512 (FreeBSD-14.1-BETA1-amd64-memstick.img.xz) = 37d64b5a7aa55b77a50bf608590e63e87d547ddc5cc4f4d0cef229e93f3570403fb846e5862bb0d23216f9c65f627f55052441a47703ea74d04ed5670d43e7f8
  SHA512 (FreeBSD-14.1-BETA1-amd64-mini-memstick.img) = 05213b62222bf95eaef0389fe4dd08880905793680b8d8f6e8b66819cc8dd9bf7d14d83f6ee36b1dd372be3c532039f3308e9b480c9ac96c1f1565e3e2d5ef86
  SHA512 (FreeBSD-14.1-BETA1-amd64-mini-memstick.img.xz) = 223b854ee96a86f54d45ee97b4918c7e47c992582ff72d832e96238fa7ed78f6883829fbef17228487f07f73a06bc932fba4186e2e595f132914a5bb616f22d7

  SHA256 (FreeBSD-14.1-BETA1-amd64-bootonly.iso) = 4b0b5bcd6e053b872ea9b23f93bc7c329f28f3104c8b771b7d2739ec3fb7df01
  SHA256 (FreeBSD-14.1-BETA1-amd64-bootonly.iso.xz) = 3ce2cdc362627ffb588e265d22705278ec2dca8aa88bd5967c4194f852c91dbc
  SHA256 (FreeBSD-14.1-BETA1-amd64-disc1.iso) = b3e8b5c31819a66d753f1fcf531e2bcfb552619e3627c1dde519f2e96540edd4
  SHA256 (FreeBSD-14.1-BETA1-amd64-disc1.iso.xz) = 64199e13946d470aee822864af087720e8f978c28bb5b69b6dd78a6a8796eaa6
  SHA256 (FreeBSD-14.1-BETA1-amd64-dvd1.iso) = 11f5b8a1bc88b3fb5a71b9481e349c93227b8478da03fd9f8f0d243f73c68dbd
  SHA256 (FreeBSD-14.1-BETA1-amd64-dvd1.iso.xz) = e4480981624325f6b87cd594be482f8ba239a2acc3b2d849f2a45a72e9a2f02d
  SHA256 (FreeBSD-14.1-BETA1-amd64-memstick.img) = 7df80ee7316a8ad46138c42643cd1e2bd081bb034245c6f2bed8f759864592bf
  SHA256 (FreeBSD-14.1-BETA1-amd64-memstick.img.xz) = f70f54880b393534c2f943dae8515899f4774f2ee3d3d066447d3bae4f5bebf1
  SHA256 (FreeBSD-14.1-BETA1-amd64-mini-memstick.img) = b825070d4e7f73a9a9d5d70c895c3bd8c51614fa7e127b8b97a9dbfb75888542
  SHA256 (FreeBSD-14.1-BETA1-amd64-mini-memstick.img.xz) = fd0aadd3f31658c12647fc9d63219aeeca0c4f466ac91006b5ef6f347d12ad74


o 14.1-BETA1 i386 GENERIC:
  SHA512 (FreeBSD-14.1-BETA1-i386-bootonly.iso) = da142afaf15cd4fb4e1c4e0af70043a351927adf008ce0d3b3e79aac7503b07219ce19147efa973597f4c25bf953ccf44e38b776d6a684849148ea115206c4ee
  SHA512 (FreeBSD-14.1-BETA1-i386-bootonly.iso.xz) = b031773389b031e1bdb280e493c44ed929feb6c845ad2094f412a1a5d808d66970f98d8b176c96a864185706a055fa538a438f157187c8bd70f223c3fa6a317c
  SHA512 (FreeBSD-14.1-BETA1-i386-disc1.iso) = 41000b3db9ee68bb4a43115e3886a1586bc47757d52bcafb3368116695a71a6155636ef5bf7892370ed6329aab23ea96bc5cf6906839f274fd4c81d2c1904d0e
  SHA512 (FreeBSD-14.1-BETA1-i386-disc1.iso.xz) = 36a9eac9c8fb7091b1865546286cf702cdd5c6ec3b96520d560dac2da261ac0492086c8119b09e4579002d9bceac6c4f83f9254e6ae3339f67e43fea8b772bf7
  SHA512 (FreeBSD-14.1-BETA1-i386-dvd1.iso) = 57d446bf0c8784184a65f52ec349bdfa88eee93bf33a1ba6d90a1250bd61c2006666b1895fb7d210f80bdf6713a34c2b4b32e37cee3a8b8d594f4fcfdebe88f2
  SHA512 (FreeBSD-14.1-BETA1-i386-dvd1.iso.xz) = 53e974a2a0d93367e2446a607db8fcd0a1abb9dbdead7aaba0ca233cc196753cdc2d5630856a61a3eba02f67971a26086b4fb075130663d3dea5f1143a39a9b6
  SHA512 (FreeBSD-14.1-BETA1-i386-memstick.img) = fb228d612279728fb813c7edc0c54c48412323ccc857e0a6ab17d537fc6fe8a23d485f499cbac1223c0c055db5c917d1b1b2e7043caeb36f7eae190aea47c76d
  SHA512 (FreeBSD-14.1-BETA1-i386-memstick.img.xz) = 0e7e3f4bb076d8c9f20e89351a2bfbdd0579de1695c35869621c1c48911bf9c6179b3c224e1eae67c6a3d202ad67b7a60f1e3ab618985c0baf55d7c0ef7e09a0
  SHA512 (FreeBSD-14.1-BETA1-i386-mini-memstick.img) = 4723f4c67b23384dd61dbed3cda642739037b180df7b4df91a0d249ba4371c14f79763f12de6450a4c2ebcd451f0cd80cbdfe4010ec9086db4ba78ddbfbc7601
  SHA512 (FreeBSD-14.1-BETA1-i386-mini-memstick.img.xz) = ffdb68edf6f272f17d4ccf7747bb53c944e59d38047dd0bb5b196bef3c82eb5be571b22320a9dceb9c9deffcbeab009e2e99fba03dad35f3ae4ec42ff059455c

  SHA256 (FreeBSD-14.1-BETA1-i386-bootonly.iso) = 77377d1913ec431901a9fc95fb9eb3ff254f988164cffe0bf7e5374351306154
  SHA256 (FreeBSD-14.1-BETA1-i386-bootonly.iso.xz) = 353fcebc0a80773aea48f6cd5f8c30d3e250e7c8d96209e6c6508d5fecb079c6
  SHA256 (FreeBSD-14.1-BETA1-i386-disc1.iso) = fb8e16bcc74f16416a67d5418f9e743cdd74b964fbbfb74e53de5088fddd0dd8
  SHA256 (FreeBSD-14.1-BETA1-i386-disc1.iso.xz) = 03b604bd0317ed0666f2a1e023e78f0953799826b023bad68cdee5d4ef0a6394
  SHA256 (FreeBSD-14.1-BETA1-i386-dvd1.iso) = ab70e78f17eca2c8f1301fc43948c8a3636d71af1a7a7ecf6ed1131f7de8307d
  SHA256 (FreeBSD-14.1-BETA1-i386-dvd1.iso.xz) = e2be55e155718fcbcd8abe8aa6e72719e50b65bbec1b762094e85e378e30b822
  SHA256 (FreeBSD-14.1-BETA1-i386-memstick.img) = 8bd1f7d5a5d135abaeb75305cc16ccdbce5ba7a2165cb07d0cc953b22ff20282
  SHA256 (FreeBSD-14.1-BETA1-i386-memstick.img.xz) = fe050768d60ee95113685f651305622b4497a83c0f237e029d10aded329916be
  SHA256 (FreeBSD-14.1-BETA1-i386-mini-memstick.img) = eb5b6f896429397b3292d7f6eac6a753c1edb55f38f6d1c2bebc081c43a8fc37
  SHA256 (FreeBSD-14.1-BETA1-i386-mini-memstick.img.xz) = 11b01a6f2f1fff3b149d82db446f4499d3fcf8fdbc2a97fc2cc30af5a91a930a


o 14.1-BETA1 powerpc GENERIC:
  SHA512 (FreeBSD-14.1-BETA1-powerpc-bootonly.iso) = f2ef58af0883321a00c156cff5f75892d9cf6c5868b08c60a90181b6cb49d555916042a1f99ccfdff70dab2860355879169e2f4918413660cd83e5a87bf7782f
  SHA512 (FreeBSD-14.1-BETA1-powerpc-bootonly.iso.xz) = f6512f27021545dcf290dc6e85579a0351ac3d33f4689903715412ec400f41cc8cb194a85166f1cfe94d2564a4f60b39eb5e38283dd040d9f98c67d5145ccecf
  SHA512 (FreeBSD-14.1-BETA1-powerpc-disc1.iso) = a1b542783722ca7c6113da50142bec91541bc5ba047fe057b07355583e359434aaff00292c02f099ed037a7189fa97bc82af3b0ef74e0a114637e2b0112b3269
  SHA512 (FreeBSD-14.1-BETA1-powerpc-disc1.iso.xz) = fbc1282c41b0eb825715d36d149f9c38e54a9d72ab88d50999bc050868f3c02abcdf76ce84c6484e216e7f52c28cf58e6af4f44300112757116e58bc846e7420
  SHA512 (FreeBSD-14.1-BETA1-powerpc-dvd1.iso) = 59ec74d36c3dc84af4916a9e2a55a0746c7818c949619423e13d3827d2e2d662ce4c141a799c21abe23434a5f4aeffb40f41e045a2a980d9ddf38c2ef4d8fc92
  SHA512 (FreeBSD-14.1-BETA1-powerpc-dvd1.iso.xz) = 1d27633af67fcfd01547c3271035340c5d4e6750165f9947c61ce002ccdc63c940a24589f4253c819cd00e111747b623c6d7a8de29d6accf7451133dac1c3fca

  SHA256 (FreeBSD-14.1-BETA1-powerpc-bootonly.iso) = f4857305b3e6ce7b207976b220a6fc88eb082eeaaf55ca7d9e38f0e91430f36b
  SHA256 (FreeBSD-14.1-BETA1-powerpc-bootonly.iso.xz) = 5a8816b7338fded6de4daf63dab81e3410172b232a6c84271906393cdd3893b7
  SHA256 (FreeBSD-14.1-BETA1-powerpc-disc1.iso) = f962b1703ea52db572f3b75fe47aa8b671f4337b154af39653e5b1ccb216e7d6
  SHA256 (FreeBSD-14.1-BETA1-powerpc-disc1.iso.xz) = 1c23213b37fb90abe4eed1d98fe730352a1dba8d60921323e18f656c20532198
  SHA256 (FreeBSD-14.1-BETA1-powerpc-dvd1.iso) = a25fcf67cc5b7e2047647a57cfc01cf08f0e4536a33d20db4779b86eef7410c6
  SHA256 (FreeBSD-14.1-BETA1-powerpc-dvd1.iso.xz) = 1fbd409d1ec660ea43d013bb7bdad7b1ee606e36f561fbc4cbcd4f344cf28318


o 14.1-BETA1 powerpc64 GENERIC64:
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpc64-bootonly.iso) = 0ccf8de1fa54bbd254a558d5db064ca1f7844dd9c98c5b5feb29b074520f5ef839f547ae1b082fe2ff6af36a1bf59abdb92e261cbe8f68e8549654a354419d6f
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpc64-bootonly.iso.xz) = ea52bc34a6ec618f3a9d6aadeb1715bf40dcdfc7c95a8fa7e8bb353ccbb7eeda501db86f664b0b5da71c5d72ce921d877ea34ffcf007a55b0ce840ef25d7abed
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpc64-disc1.iso) = 4c9a99a177815a1546a3f7b3f251917a2ab2c8f32aa9a6cb664d00322b37f3b6575409182cbe62a6e80a808fe0567416a0f341a5839a1a2b9b3796e8f27701a9
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpc64-disc1.iso.xz) = 9757f98d1b9780077664fb7538bdc80f52a753668a9f7f8d5c2e2de677f4ae4ed04db9753d356a625d74cc5b823cb1c88bc1f43fe973fdea657cdead9b1baa1d
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpc64-dvd1.iso) = 3faf8a1a0859f2a39479609c37b4e3b9bf28379ab64f4470a0419bd0e34ba7d1233bda5ce8a68a4bb6019a92056c26419c6e45a623b79840c6c2aa541a45fd84
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpc64-dvd1.iso.xz) = f564e5f8db7693af2d62af0c9b4e1868888de124a903277b1ac68a3dfc818cb3b668cb1e725c94aac1899f59094e21e3b333da69ccdff71174ec4f81d7a5ce92

  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpc64-bootonly.iso) = ed8968d7633cee38dd21d5cf05bcebda86ac59b06841e5face6fa43040620767
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpc64-bootonly.iso.xz) = 27e80d7161e89e7b4e180b77fb751063f0d95d2cdc457ce030b093ef3b22d643
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpc64-disc1.iso) = 7920ac5aae9538f14491684970a378f8195ffa83ca8cd5cf2efe422f65ac0f21
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpc64-disc1.iso.xz) = 2402bfc9517e0ff3dd974712b8687f06612a8b9f2858a6401c97637486531116
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpc64-dvd1.iso) = 3d6e74860eec230fb35126cf1e093cc9af21b8cc74aaddb4fa7b6b46cd6c3c2c
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpc64-dvd1.iso.xz) = 97a444ce13720a222fe5f351ad2fe1e0b65873cde3201c02a97d011a32b768a0


o 14.1-BETA1 powerpc64le GENERIC64LE:
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpc64le-bootonly.iso) = bc566f9aae21f182da09b74c3b3fc929a19fbd533195970a4784f955ba039d5562e1941e5b2730353ff45b623f883cf662dbfb3e1a11109e60b035976ecf482f
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpc64le-bootonly.iso.xz) = 97d0c3814b5881175a1b99c77e03cc078a05d686871e11c66becef1814708b698965c8adbaf745052315e783453dd08b64161cd41258bb454abf18ad91fc89d4
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpc64le-disc1.iso) = 5b45151f23ad5d3d2e4a45fca8de063a6923ac3bcf3d119e6775d9119f9a39d48dd5bf300287f8d2f61676f693cfb09c7fabc64035a5d550d619d78b04e195f8
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpc64le-disc1.iso.xz) = dda8cbc04593ad1df0cd1071768adeeb77c0467257bc3047f43eb8cd2e62a75087689f1431c81450b9ea2fab2a80435b2c2a6b71672eb16e21689a0dbf25a3fb
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpc64le-dvd1.iso) = 31bacd89a99491c6b04e15ad85991c046f076ac1b41ebf9b09957f7025c81a841440c38cf6e4f4bee0e8624b41e40a3eb48ec363eac9b2e700121edc4154f6dc
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpc64le-dvd1.iso.xz) = 1e25399d4cecb3b64de21b4ae3b4b375321803e3156bdb626c6d25d9c672cd0042e6a4159a940377815601e0f5f66642e848e65044bccc115b542583a56d1466

  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpc64le-bootonly.iso) = 30dd5b729ef0bfd37826926558da45c9be7d53c35e0d42178a5fb29a7350f267
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpc64le-bootonly.iso.xz) = 0411a4805fe44bdf02d5d3e61ad681be445a3398cca3a03aa491e3241c3bcfd4
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpc64le-disc1.iso) = d3255ddda28f370e067555d875b84c636abb492cb63de80135dd17c213f4f95d
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpc64le-disc1.iso.xz) = e5eacb6c6842f436ca5c3f192c645d8a8c680a7670ad622d6c1e240339dcbf19
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpc64le-dvd1.iso) = f4caa44c0ecc6072e44fe5bff403859afcaa4e965f186dc3137bcdb325ea94d5
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpc64le-dvd1.iso.xz) = 35f30ff90df534449a87f79a8fc7f9cf3025364b6dae1349ce7c5de898d072fd


o 14.1-BETA1 powerpcspe MPC85XXSPE:
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpcspe-bootonly.iso) = bd01918b233fea695610a6853d80eee38b33d35bbfd027dc25935ae3b6758637f164d38627fd59e4c4016947998fe4001de21e5f614335589ea519182f665dc3
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpcspe-bootonly.iso.xz) = 3278cfbebb9ba255a095af644dedf896e96ce6710e105677c114651564f67464363a964bc91d64d7534168f6611fe48aab4ab8f196e6f162acdd8a6db740a514
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpcspe-disc1.iso) = d6b6652d1baaba949652dc0de621fe001891f6fd24e2e84601348656dba29683afbc3927da3a359ad985a3f7ccf1be9974a92e83f6deb9083f23d5fab176f587
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpcspe-disc1.iso.xz) = 0e7638140005bd760112adf8f0db296dfd8908b95f69498fe706c66082e712e5d0004aadb7580cf82dbb8611bb231c4428444aab5ab892d95fb61d6d4001a25c
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpcspe-dvd1.iso) = e1e157ffd803d967da32fd78cb93995b151f9bafb2233488301ec37003018ee2e858e4722d9a3331dfc0825cc33811ec6f1dd6b1c0c75b2e691bbf3f0cbf5aaf
  SHA512 (FreeBSD-14.1-BETA1-powerpc-powerpcspe-dvd1.iso.xz) = a7a6dcffbc8450756b978cc537ec068d58561a09802813e86950e744c65e51c8ea3a286c062802307a028c57ac4de6578764c864d7513a1b00c7ab220eca480c

  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpcspe-bootonly.iso) = a799b972087b64128c658d73ac998f6af7d867fb33caa7555802be4e3afff066
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpcspe-bootonly.iso.xz) = 1deb21530c0da0956073ea64077435aac3eb3a07482c81615fdda1813f133a4b
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpcspe-disc1.iso) = 1e8539245fc64e624a420b2d75d52da9c7554f1a77308057dd0f9f1677ef7aeb
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpcspe-disc1.iso.xz) = 27c4593c891ca92fd342ac6dcd0d9d7a64f6e03f8968d96d910ee37f785ea44f
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpcspe-dvd1.iso) = f1cf226e7c711e01a184924bd17bcba92a5516214d4ddf01ad9288274eeed52f
  SHA256 (FreeBSD-14.1-BETA1-powerpc-powerpcspe-dvd1.iso.xz) = 006912e1a5f762ada0040f0b7fdbe1b4bd8c1ca6d1b980cf30e0d37488499ace


o 14.1-BETA1 armv7 GENERICSD:
  SHA512 (FreeBSD-14.1-BETA1-arm-armv7-GENERICSD.img.xz) = b562c1c3d811f8971cf0bb1f534709fb2323dad43ba3895974389aba1818125f4a53d0ee475087dd215a74be28f0e04157c715aa680e363643c030a0d79b6fad

  SHA256 (FreeBSD-14.1-BETA1-arm-armv7-GENERICSD.img.xz) = 22a06a943230aca939454df6272f1fa24d690bd0231ae288c2174f17e190b6d5


o 14.1-BETA1 aarch64 GENERIC:
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-bootonly.iso) = 7f41997f379b6fba902fa944cec724b89fe32adcb4091f35a91e61dfd3aee2cc23d5ef3292f66c81bbc9b0a5e88756951f80b8b641f7e04fd691b0efa19599e1
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-bootonly.iso.xz) = ed3ed7f1b177b841e7b54550e47313057f35227a94792b17607a6c6cde48af449a108d3b6ac2331251ccdadbf5828fa6f686eb8026865dbadfb37bf57623a1b4
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-disc1.iso) = 2efaf9a4aaffdb6c98ea3a640012091190c9358241db365fa9df751a0a904cfdc813e6567d56d38aed8551661254731baa0396075efc0f61d78052f809472d26
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-disc1.iso.xz) = e26c5b090abe889ed52727cfe7baa3f6f00176029ab4da7829573031c0794069013a50faf2b4ce73741d7566388fbe1e7bfcc23898347fc46b7812725acc87d1
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-dvd1.iso) = 26dbec972151f548a2280e3faf80ca401e92034719dae902d97aed22cc58e8decc8c014283cb656666b747e5e8e1566deb5c686bafb9104d61781b39d9823d93
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-dvd1.iso.xz) = fcba40013daebf1dfa320f95a6959d1bffa11c164ea577e70a727d5e4e79e9ee7a729493d9a35efd824e1710090e9a509deebb23ff1ea3500928238954472fb2
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-memstick.img) = 272b46ccf1670ba475103304cccc3bfe911b80581d6bd6e28936e3dacc82c023867af3fb0dc583d85590405090ce4d1c2d0e590e7ede3f6d51e3744b50b518e0
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-memstick.img.xz) = bbe63f3c2a355fe4b346c78517412e45191550c6335c5cebd9b64f5ddd535ca79f7297764cb1b2335dc243b59ae979b5d1f85a1492a55223f9b433c064009317
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-mini-memstick.img) = aaf308e965811237844900240298e9ded4c409afd72776d61dcd0a717c12e2989e49279ba496fb071f82495e31a65f0e71c5b782779f840726933313cc7c171e
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-mini-memstick.img.xz) = 9bcb48a42943fb1cb0794224e18c3496578a6199dbddf89b65bc8dae9a8e915e9a6a68b1d9dd9c6e06c14fe63c1408fdc5ad5240bf3ca9044a0f5ddff4780a86

  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-bootonly.iso) = a0b28544ec33b3788453a17f2dd73743b6b690284c0b46d5f64fb88ea51c738e
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-bootonly.iso.xz) = ae2bd3480211f9f1ea8b227ad445acbe367f80d007484cc460ab8231b2b16eec
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-disc1.iso) = 6f50b7f03fd1c2d000ffa755911d7c911bb82e54e4e74b528b15ef44405294a1
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-disc1.iso.xz) = 16366d29b5760750a5e4354ae7c139097f190bba7cefa73ae7686265a8b5a770
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-dvd1.iso) = 795255045bd1d9823688b378f27309913b0c5e7a176353c4af40c42332fe35f3
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-dvd1.iso.xz) = b803cd1d0d2290da33a1ff748f01ccc90657daece2ad56af0f6d2d6e1cdef07a
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-memstick.img) = 4aac28561b3173f5a79c8f3ef6dea426f5e70fe78a26035bf9c2904082a38cb4
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-memstick.img.xz) = 935fe2d1a9e94074c4e98d1878dc48e48cce9c5e127f77e8fcae97eccecc726a
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-mini-memstick.img) = f746e2fa5a4a2206e65072048c7ea86a853cba837b395f69e7eec0de22afb4a4
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-mini-memstick.img.xz) = 281a755b59ac955e77654754c3d0082958f0229f029c6fcacf54244e41f2cb59


o 14.1-BETA1 aarch64 RPI:
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-RPI.img.xz) = 83990e13a254f4b4cda0ea23104fdb98b04b6fb966d6cd2ef189a70911f69d5172c3d7fdc0014a56d3b8498791b9b7531d39870fd0f63f4d85ca5b0cb68c6b8a

  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-RPI.img.xz) = 34bcae048666c17b5280d900fd9e9cde148c7a4428215265768fb82fbe18cfb2


o 14.1-BETA1 aarch64 PINE64:
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-PINE64.img.xz) = 9b9212b11d455f1c34609e8d96939a796d0ac6a9cbc61a93cbaf0b40d2e909e2f0f32d1e45204fe9e5f6660e4fe5886dba998fe0cf9ecc3ce2697561ba4820e3

  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-PINE64.img.xz) = 2e810c3aaa3465c729901eba6c0987247b5512d0f3a3d6ecafb457c92403eec4


o 14.1-BETA1 aarch64 PINE64-LTS:
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-PINE64-LTS.img.xz) = 46810334ee776945f533e3e73de344d579ad2899b1e8eed87f8bec96bc19e9f32827ff054e9eca755b6b7756d3c85032217a76115e7c93f60d2e238ca0e960f4

  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-PINE64-LTS.img.xz) = 3cf8066886bca204c7c3b4809e643ede0f33157322c0b0dd216dc83d45927e7f


o 14.1-BETA1 aarch64 PINEBOOK:
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-PINEBOOK.img.xz) = 183da594dc879280b81dda5ba435c3e162da92c36f5d31aa95dcf0317ca6132e12f1a70527ba4d1a1ef1a3075139d51414633ae2f2840e77b19165d9ed960b0b

  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-PINEBOOK.img.xz) = 8b0e0469ac5df4e6d6211ad8ad0e97b71faba59fd8325b7c2539f1d1b9cc5aca


o 14.1-BETA1 aarch64 ROCK64:
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-ROCK64.img.xz) = cf7a9bb3bab8d71bd0447ce8abc88fccc9ececd1b67e881d5801b93b2798f765286011003263683133b7caf4a39cfd936cd6f8d2198e13cd038f818962420bc1

  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-ROCK64.img.xz) = 6f275a8bd51f7b5626fafb13c0154059e70e067499a6226d2ed063e8092ccb06


o 14.1-BETA1 aarch64 ROCKPRO64:
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-ROCKPRO64.img.xz) = cb30d3b2e882acaa537e2caafb4d4487d76357d637ae8ae4235744a3694cc5583f8a50dedd1c056a0f8d88d3ea72707bbecceace6b75e6718cc08942e6d680bf

  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-ROCKPRO64.img.xz) = 38d0e8720e4f8416b473869b8a605d12bb436cb41610238ccb7e953dd17d3e73


o 14.1-BETA1 riscv64 GENERICSD:
  SHA512 (FreeBSD-14.1-BETA1-riscv-riscv64-GENERICSD.img.xz) = bc3fb08e1d4b54bb90f68bef5363ff2c0110f3745741be0701bd3b553a4768096915abe6c144962d7c1bc9a7ab954b6976e8f931b40306fec06a111e362476fd

  SHA256 (FreeBSD-14.1-BETA1-riscv-riscv64-GENERICSD.img.xz) = 15eececd47bdebee4b751735dd2ec24b3d0f19389161ce77f778c70c45e8da9f


== VM IMAGE CHECKSUMS ==

o 14.1-BETA1 amd64:
  SHA512 (FreeBSD-14.1-BETA1-amd64-BASIC-CLOUDINIT-ufs.qcow2.xz) = 58f978e48ea017750c7e16fff8254b6b3ab8f2ef6e1c3c7f74f4573361711cd8b0f273a72ec0181afb2632188f5ed407f9ee37febdd634def25bd35843224913
  SHA512 (FreeBSD-14.1-BETA1-amd64-BASIC-CLOUDINIT-ufs.raw.xz) = 3e5dc1443daf1f0237d6685ff5d96cfd232f488e78e9a4f3b45d5cf3fc9b440ab50f2bfc44b7929db29760fb397b845761e541b2e33cc82334ccaad49bf7e63c
  SHA512 (FreeBSD-14.1-BETA1-amd64-BASIC-CLOUDINIT-zfs.qcow2.xz) = a148807a2fa03a752e16d766e26b560236cbbf1e1983807401fb7ba43b3b7398bfad511276b09dd388f2b27958565c268962cb7f404dc0164e0e331f5e254f05
  SHA512 (FreeBSD-14.1-BETA1-amd64-BASIC-CLOUDINIT-zfs.raw.xz) = 4bae139510737fa913435bae9891b0615c88cf1ebfb23b62e23d6df935d783048413ef298f706216d61df768ba20f76597ac580ce8e6844a7e1c4fc5c630e838
  SHA512 (FreeBSD-14.1-BETA1-amd64.qcow2.xz) = dee3f9befa40583530e7086c9a3630e24837ff3725daeb62500bc4ad532804adef69835635ee771b4adb05d617c23e4d9e00fa372a2733c874d09e63bf60de64
  SHA512 (FreeBSD-14.1-BETA1-amd64.raw.xz) = 927e855e548e17a04b6e99fbb5857e7c495338256cc4729158c8097b7c923030c01414f7f41ff887f5f8d0583d83d776034c484780b6cb78e9935bfe2e3dc5c2
  SHA512 (FreeBSD-14.1-BETA1-amd64.vhd.xz) = bc397c9c121a089be5425f7c05825b7e7d501ce1acc489fbb0fee6cdf5e61944183a7ba806281c865f136d6218ca3459539a0b58a0035c9e652e4e61d397484a
  SHA512 (FreeBSD-14.1-BETA1-amd64.vmdk.xz) = 71b5b3800428f448a77e919bbbf1dfe80e89605bc3a5fe7f73111648f2b2660b27916fd987e33fc2b12a40535403eac5b746f4cd14f80294273498a4924bc2c9

  SHA256 (FreeBSD-14.1-BETA1-amd64-BASIC-CLOUDINIT-ufs.qcow2.xz) = c1a49681030ccd4ffbe90fefd3075bad93656cbbf7ba5ee3d69e43dae7b1afd0
  SHA256 (FreeBSD-14.1-BETA1-amd64-BASIC-CLOUDINIT-ufs.raw.xz) = b522d26653b6c23f7d6a4cd200c3813da5dc597eef1c5a43851cf22dda07a332
  SHA256 (FreeBSD-14.1-BETA1-amd64-BASIC-CLOUDINIT-zfs.qcow2.xz) = 3b9b52a462eda49024845f9b69227b8a13ba0921ce55056aefe0b220ccfef0af
  SHA256 (FreeBSD-14.1-BETA1-amd64-BASIC-CLOUDINIT-zfs.raw.xz) = c1f7f201d9d2075e5e1458a6914678a18e6d55d05ef8d295f1e506a7b50f34b5
  SHA256 (FreeBSD-14.1-BETA1-amd64.qcow2.xz) = d4bad24a77d6a824d8ed84d6a6514ddce2b5b4efede5a625ab98d9801dbd4729
  SHA256 (FreeBSD-14.1-BETA1-amd64.raw.xz) = 361c66a47ddd5de02265b06cc681ad5d1e464979f1cb1c10280ecdaeedd51e10
  SHA256 (FreeBSD-14.1-BETA1-amd64.vhd.xz) = 0b34ae91fc81b7562fb818ad4592d8fa50e7899ce7452a7c80d73542ccc34a03
  SHA256 (FreeBSD-14.1-BETA1-amd64.vmdk.xz) = 583a5b231d5357d677d45641a03671f1fbc385d2420913c9a8b931c129cfb833


o 14.1-BETA1 i386:
  SHA512 (FreeBSD-14.1-BETA1-i386-ufs.qcow2.xz) = 42ef3cf033b08be1aba2385058a8e26a8b430d65760ae92c2ea1b80b64232e2b4bd11ad7a5cf2089d2e9c478f30f9133a0eb9fb233a9dc7f90557d7715129765
  SHA512 (FreeBSD-14.1-BETA1-i386-ufs.raw.xz) = f181dfa354e3f047ab429120215b719cccd7de03bcac259a4ad0a6666865ac82abafa667869ef845b7c259d8a4bec4c0e07b480bb02b89c62dcb976dcfabe206
  SHA512 (FreeBSD-14.1-BETA1-i386-ufs.vhd.xz) = 75b00fe2f929749e3478e5b60aedcbd98a8be7d9652dbd49ba38ba2215ed2b3fb7f6c6c29e0187a18ac6f79a44f78255dedf5e1b6ee8e1a8f03e569eface9276
  SHA512 (FreeBSD-14.1-BETA1-i386-ufs.vmdk.xz) = 054f65afeec133634855f725e0f26e916bb321a7904d19129e37ab62b9cb7f900795755f8a0f1bfa003bf0c9aacbceba52537e9c93a335adb1f06cb6d6129b8b
  SHA512 (FreeBSD-14.1-BETA1-i386-zfs.qcow2.xz) = d149434d2b337f33322dc004caa6f04869bf742dadd2ed2d09cfa59c075863b9fc67ca5a4dc54ea45bd88fcdcf905544128c8a898f7497aab155a2c44a90c25c
  SHA512 (FreeBSD-14.1-BETA1-i386-zfs.raw.xz) = c112fa8f38d830da36f244af8f2d6fff691920f441b696880fe1bd6614abe0e54535a25c9c2657879aa226964d17dd2a384bc2b93395a6f2a763260e94ce7d56
  SHA512 (FreeBSD-14.1-BETA1-i386-zfs.vhd.xz) = 97be62daf1c7bf0e7c17f8663e6ba9e682875c477833185574cc55ecb06ca3152af1a1db85b219f62e5ad5505539f76c6d7aaac65140825869dafb7fdf7848e5
  SHA512 (FreeBSD-14.1-BETA1-i386-zfs.vmdk.xz) = 9f6db0fdaa86cc36c323782e5b17229a9b2a682fc256ff0e07597906c58693571a4f530cb35199e259fb76378a3f05e581233e79cbf7a402fd5e882891541771
  SHA512 (FreeBSD-14.1-BETA1-i386.qcow2.xz) = 42ef3cf033b08be1aba2385058a8e26a8b430d65760ae92c2ea1b80b64232e2b4bd11ad7a5cf2089d2e9c478f30f9133a0eb9fb233a9dc7f90557d7715129765
  SHA512 (FreeBSD-14.1-BETA1-i386.raw.xz) = f181dfa354e3f047ab429120215b719cccd7de03bcac259a4ad0a6666865ac82abafa667869ef845b7c259d8a4bec4c0e07b480bb02b89c62dcb976dcfabe206
  SHA512 (FreeBSD-14.1-BETA1-i386.vhd.xz) = 75b00fe2f929749e3478e5b60aedcbd98a8be7d9652dbd49ba38ba2215ed2b3fb7f6c6c29e0187a18ac6f79a44f78255dedf5e1b6ee8e1a8f03e569eface9276
  SHA512 (FreeBSD-14.1-BETA1-i386.vmdk.xz) = 054f65afeec133634855f725e0f26e916bb321a7904d19129e37ab62b9cb7f900795755f8a0f1bfa003bf0c9aacbceba52537e9c93a335adb1f06cb6d6129b8b

  SHA256 (FreeBSD-14.1-BETA1-i386-ufs.qcow2.xz) = 36df8ffe6535028b2fe597c378665df9ff6accaa737c1bd2374935547f2b9931
  SHA256 (FreeBSD-14.1-BETA1-i386-ufs.raw.xz) = 48956d9d7a16b6c4aa76e2a5a924056a0fc824277a1fce1636a97f21624b21af
  SHA256 (FreeBSD-14.1-BETA1-i386-ufs.vhd.xz) = 5b59abfb05ba5a60be7d8747480b157c32f67bb20b0cb1cc381a698caba9bca0
  SHA256 (FreeBSD-14.1-BETA1-i386-ufs.vmdk.xz) = 49649fc80f911d2b38ae3269e2f5b9ecade47110dd5849c364faf58308a380cd
  SHA256 (FreeBSD-14.1-BETA1-i386-zfs.qcow2.xz) = d94e6b45773f0caadce0177ccc22f163d9be71a91c96ee22edbd43198ac10b58
  SHA256 (FreeBSD-14.1-BETA1-i386-zfs.raw.xz) = 1e6a94163bcd3fa993d2d25156d91f953bfc6f3fcc7b3862c78ca7d80d56b805
  SHA256 (FreeBSD-14.1-BETA1-i386-zfs.vhd.xz) = fe14bac9f42679d1ec653b9cf2918bd3c0c88df7dc356338a5e824f5a791a87f
  SHA256 (FreeBSD-14.1-BETA1-i386-zfs.vmdk.xz) = 7790abd1d988560a4ffdb3f8c9ed1a63dcd77d52f2000e9d840e79eb563303f6
  SHA256 (FreeBSD-14.1-BETA1-i386.qcow2.xz) = 36df8ffe6535028b2fe597c378665df9ff6accaa737c1bd2374935547f2b9931
  SHA256 (FreeBSD-14.1-BETA1-i386.raw.xz) = 48956d9d7a16b6c4aa76e2a5a924056a0fc824277a1fce1636a97f21624b21af
  SHA256 (FreeBSD-14.1-BETA1-i386.vhd.xz) = 5b59abfb05ba5a60be7d8747480b157c32f67bb20b0cb1cc381a698caba9bca0
  SHA256 (FreeBSD-14.1-BETA1-i386.vmdk.xz) = 49649fc80f911d2b38ae3269e2f5b9ecade47110dd5849c364faf58308a380cd


o 14.1-BETA1 aarch64:
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-ufs.qcow2.xz) = ba4cd956188d55329f2052ec881dbc4a973ae70e7719d3bda06e9dfdddb0fc95a9e262e46d181d84d5c7fb11dd5069964cb74879355e0a394dccf703334fe306
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-ufs.raw.xz) = 24452cab06dba25b4a60d3b7738c68dc4aa8369fcf68b0d5a4e3767ba4800c8c9ed284c7a1de142891aa970308d0cb770fea7c6771cb6ea2135e6bf4e3f8a382
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-ufs.vhd.xz) = 3f046118830842c753b8dfd828cf6912c03497c963dd875ac3f15a07230754ab14fbfcb123f180b0b803bf565e9c8b683016c32e1a7d2b06d296b1a3dd1d79a6
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-ufs.vmdk.xz) = 8b4b611419cfab6d7fbe30b13eac18bc6f53f87dc19bdb975ded424c9e48f47d7df7652519784bda6f21f6ea07a92e83d36eb3c50a4d955c7486ec9baa7959b8
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-zfs.qcow2.xz) = 2521063163c70f47e397c6532c8ee3f4636c5d5738c398101b87b56d01dc10388ed6c8b2864951c0b3a556ec337daccf3248cbe3d8c6a3b131c00d0945898037
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-zfs.raw.xz) = f11772ada8d810292f1fdc93b4591dd18cb0f6567f974517e4769d8623ccf201207f04bbd886754cbfc38923165be653bab3af3e867085abb65e0a0f22e8d5b9
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-zfs.vhd.xz) = 1748eb95d6a296dd5926644b6cef9a88eeef7b6463323f6bd61a49b016123ee297a57d4df720edd1a1d38e4e916ee7de45d17398799eea5444dc693275e42361
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64-zfs.vmdk.xz) = fd0d9651f434be7604f596d11aa14eb09c7c30da118a3f0c06639d17cd37a361c9fede1f56643990e7bd54dabfaccad7893eb6aa14664565b4d29530fdd8b444
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64.qcow2.xz) = ba4cd956188d55329f2052ec881dbc4a973ae70e7719d3bda06e9dfdddb0fc95a9e262e46d181d84d5c7fb11dd5069964cb74879355e0a394dccf703334fe306
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64.raw.xz) = 24452cab06dba25b4a60d3b7738c68dc4aa8369fcf68b0d5a4e3767ba4800c8c9ed284c7a1de142891aa970308d0cb770fea7c6771cb6ea2135e6bf4e3f8a382
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64.vhd.xz) = 3f046118830842c753b8dfd828cf6912c03497c963dd875ac3f15a07230754ab14fbfcb123f180b0b803bf565e9c8b683016c32e1a7d2b06d296b1a3dd1d79a6
  SHA512 (FreeBSD-14.1-BETA1-arm64-aarch64.vmdk.xz) = 8b4b611419cfab6d7fbe30b13eac18bc6f53f87dc19bdb975ded424c9e48f47d7df7652519784bda6f21f6ea07a92e83d36eb3c50a4d955c7486ec9baa7959b8

  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-ufs.qcow2.xz) = 02c4bfd75a5b74fa6eb5b47dd9e80592daead8c52048232978a0cbfe8d633ed1
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-ufs.raw.xz) = 65428a24108d1db2293f948beb7aa8fc2fa59fe19e7993cb2ae6d351546f2e80
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-ufs.vhd.xz) = 916887953d21e8a5b3e2bfd80df1ee07fd47957643bfe77f2dfd2d22b5ea64c6
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-ufs.vmdk.xz) = ef5ab42dd438869c150dd9ce325652af2c5788d30f662bfa5e73c7006989a513
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-zfs.qcow2.xz) = a1a020d962beaa3d616d0658a1a9c750d5e14b568bf94f088ee7f8d6c0b225d0
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-zfs.raw.xz) = c7ef3810a7a8d7f2c7704a315355ce9462212f076dc83bc6aaeaa8d15f364be9
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-zfs.vhd.xz) = abb5cc6a4cb1d1e983feae921d3ee24758cdfcc76363b9e9ce545227780d4737
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64-zfs.vmdk.xz) = 44a81e4d8db4451df4a3745fcc6924be725cdfbaf27438d94556d973395c0f29
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64.qcow2.xz) = 02c4bfd75a5b74fa6eb5b47dd9e80592daead8c52048232978a0cbfe8d633ed1
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64.raw.xz) = 65428a24108d1db2293f948beb7aa8fc2fa59fe19e7993cb2ae6d351546f2e80
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64.vhd.xz) = 916887953d21e8a5b3e2bfd80df1ee07fd47957643bfe77f2dfd2d22b5ea64c6
  SHA256 (FreeBSD-14.1-BETA1-arm64-aarch64.vmdk.xz) = ef5ab42dd438869c150dd9ce325652af2c5788d30f662bfa5e73c7006989a513


Love FreeBSD?  Support this and future releases with a donation to
the FreeBSD Foundation!  https://www.freebsdfoundation.org/donate/
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEglY7hNBiDtwN+4ZBOJfy4i5lrT8FAmY4CM4ACgkQOJfy4i5l
rT/qIA//YJpOXK9txGfzr4lxKGPe5iDbNGG8s0DTHx6M/SSFqy61WZp/YrmK/aW+
tL971syz9SqR5JQnOtm4Y3lcfxiPtbpthEXPKiGbF5BLBQYCjUdtSVOq7+MiHzQM
ToEqcuw1OnooYc9XVz2HnlCSPRZeCgU/6wfzWnqeRv5lqwB0xslmXagtKfERfsQm
fKxg8wp0oX8UQKg/+enP/nCk+M7iFNeFjoB+20rDjCJqZt5WeTXk9160v2RG8G5S
TcaVHIfYt6GFvS0f46arQd8d8pTnGJbcHzWI7GBJ5/2vXYmf1ZqqIihsD9SEgvAP
Sz2AsLFr5ztsvosQJHzuNCV4ibM5dZzdDPnFrLWmBtvLEMgqj1h167lJodUXb7qn
8oX4i4eOpoH1DQ5qzCK6ar6NyyaB8eXCycsh9DM0HbwMnIEThI8gKBzXeShq8l6z
KR18fSbYupysSasYv5HiYfM7WlTT2eHJfhrn5ZEsCBrv39VIdjJsj/Uq5VPh/tNf
aIlx7EDItI3BWjs7ZyVC0cakGDC8hcen6UlfE+nTo3TlZho84x3yICqLCj6O2tD5
BYtT3qAZdAyBvUNrkg524D9wPGBRH2kriDmAs0Gd+fJXLWIqe62qi9VY/HPd94Y3
hCYTARAlglPivVN9WNMridctfYVTEQ4EVNwzV1uA1QgZeIGBl+w=
=lGId
-----END PGP SIGNATURE-----