From nobody Mon Nov 4 17:28:42 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xhz6K6nyyz5cLkg for ; Mon, 04 Nov 2024 17:31:37 +0000 (UTC) (envelope-from dch@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xhz6K69Dqz445t for ; Mon, 4 Nov 2024 17:31:37 +0000 (UTC) (envelope-from dch@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730741497; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=I/wz65bCGVbX+oX/tXMj8viiUHJo7T7AooMqXGT6KeE=; b=CDnhZJaDlb7UckjY1vqh9Lq3jpvi5oRgLsl6UK+jK/tlz5aULGN18lgFKO8iFyCKQ+H0Lx b32HmO5KYqCguZDPWyOVbnDUb/IP0/eXdznHewZqZf6TQ7DGvY9PafMnVS+2HrJZFDBf3K TgTwppTsHVyNBs2DgATvvo/MlB2pNjTG4bMe6Vi1YIiZ0rqHOI5KRQTEBN0vGH4Q2gcAk8 gs6BQ0QpE8lCL1Kvw2jSZol9ti0J8jd+kRdcL0gRrolpJJNeAK6KV+wQ33odqBToa2U10G zMxPsB2tyPEs1VVl0VqdTLHV+VgARICUtvawUZ6h1qH+dbvC7iEpBGiGkWlOqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730741497; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=I/wz65bCGVbX+oX/tXMj8viiUHJo7T7AooMqXGT6KeE=; b=XsoIcvserZVlIEHYWU2kurh4Ys8Qjb8xsM7AkOznSTGi5ikfaMfNo5Hip4m0DHRkJnRmr7 PYtMQ1INkOH41Pd/06B3NOwCbBsJd3hVzRE7DHvAP90XLFCCMtYaBTXOHknln4N7Fehtoj /I3DtFUQO4PhxuHzD3RmYx0Z2Q+5l/6nCn3xEx+8qCn7ipsppHEUNKLz2kVxbKl2Oa0ViC avoTf5ieD3g6/DOAE3DBbWWY8/+FtPGjqA7/+krEf1Qtz9Hh5CQqTl4/HZDYJt1ORgwqmP RTjD9utZvnG5hrIYQhfMbZXmbQLJqSdFJpZKuPKX3hxDAbBfOHjz+aqk0uig/w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1730741497; a=rsa-sha256; cv=none; b=lcIweTYkcblrAuad+25Dj9GvJfd1/0anWM6FDS6Fu4EgPnXyKwBOP6Fx7a8IiRt7fJgbrh IB0zL+I3AGnFG46RXtMTMzeAXB6pIUioMv7pYWYf/Ac4JCBY78VBBFReKQ5XZK68BbG7eM PZhirw6IySSguD2E6dZ9YzYTb8kZ6jZaz0vvE8EEKaGFeDdQrnSQczmMeILWK5wyWF/p6G j+5PVQYd8BWJ57OzQmi2lvW6b78VzX/H+izAy+6AUeQ3a84UeWs+2UP1VX4xI2x+k/P7ul IIrXnMWkvWtflXIGMbXOZeHJp9e+ZfJC9SeWcwQzUaM8Pp26JF/FLIlF+/HPJw== Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com [103.168.172.200]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: dch/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Xhz6K5Yvrzh36 for ; Mon, 4 Nov 2024 17:31:37 +0000 (UTC) (envelope-from dch@FreeBSD.org) Received: from phl-compute-02.internal (phl-compute-02.phl.internal [10.202.2.42]) by mailfauth.phl.internal (Postfix) with ESMTP id 62B74120006C for ; Mon, 4 Nov 2024 12:31:37 -0500 (EST) Received: from phl-imap-02 ([10.202.2.81]) by phl-compute-02.internal (MEProxy); Mon, 04 Nov 2024 12:31:37 -0500 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdeliedguddttdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecunecujfgurhepofggff fhvffkufgtgfesthhqredtredtjeenucfhrhhomhepfdffrghvvgcuvehothhtlhgvhhhu sggvrhdfuceouggthheshfhrvggvuefuffdrohhrgheqnecuggftrfgrthhtvghrnhephf evfeekjeekgfegueejgffhgeelvdduveelueehieetleelffelveffffdvveffnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepuggthhdomhgvsh hmthhprghuthhhphgvrhhsohhnrghlihhthidquddvgeeluddtfeeguddquddvudefuddu jeejqdgutghhpeephfhrvggvuefuffdrohhrghesfhgrshhtmhgrihhlrdhfmhdpnhgspg hrtghpthhtohepuddpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepfhhrvggvsghs ugdqfhhssehfrhgvvggsshgurdhorhhg X-ME-Proxy: Feedback-ID: icedc46df:Fastmail Received: by mailuser.phl.internal (Postfix, from userid 501) id 25137B00068; Mon, 4 Nov 2024 12:31:37 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Date: Mon, 04 Nov 2024 17:28:42 +0000 From: "Dave Cottlehuber" To: freebsd-fs Message-Id: <3293802b-3785-4715-8a6b-0802afb6f908@app.fastmail.com> Subject: nvme device errors & zfs Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable What's the best way to see error counters or states on an nvme device? I have a typical mirrored nvme zpool, that reported enough errors in a burst last week, that 1 drive dropped off the bus [1]. After a reboot, it resilvered, I cleared the errors, and it seems fine according to repeated scrubs and a few days of use. I was unable to see any errors from the nvme drive itself, but as its (just) in warranty for 2 more weeks I'd like to know if I should return it. I installed ports `sysutils/nvme-cli` and didn't see anything=20 of note there either: $ doas nvme smart-log /dev/nvme1 0xc0484e41: opc: 0x2 fuse: 0 cid 0 nsid:0xffffffff cmd2: 0 cmd3: 0 : cdw10: 0x7f0002 cdw11: 0 cdw12: 0 cdw13: 0 : cdw14: 0 cdw15: 0 len: 0x200 is_read: 0 <--- 0 cid: 0 status 0 Smart Log for NVME device:nvme1 namespace-id:ffffffff critical_warning : 0 temperature : 39 C available_spare : 100% available_spare_threshold : 10% percentage_used : 3% data_units_read : 121681067 data_units_written : 86619659 host_read_commands : 695211450 host_write_commands : 2187823697 controller_busy_time : 2554 power_cycles : 48 power_on_hours : 6342 unsafe_shutdowns : 38 media_errors : 0 num_err_log_entries : 0 Warning Temperature Time : 0 Critical Composite Temperature Time : 0 Temperature Sensor 1 : 39 C Temperature Sensor 2 : 43 C Thermal Management T1 Trans Count : 0 Thermal Management T2 Trans Count : 0 Thermal Management T1 Total Time : 0 Thermal Management T2 Total Time : 0 =20 [1]: zpool status status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning i= n a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the dev= ice repaired. scan: scrub repaired 0B in 00:17:59 with 0 errors on Thu Oct 31 16:24:= 36 2024 config: NAME STATE READ WRITE CKSUM zroot DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 gpt/zfs0 ONLINE 0 0 0 gpt/zfs1 FAULTED 0 0 0 too many errors A+ Dave =E2=80=94=E2=80=94=E2=80=94 O for a muse of fire, that would ascend the brightest heaven of inventio= n! From nobody Tue Nov 5 01:14:43 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xj9Nv6wnmz5cp31 for ; Tue, 05 Nov 2024 01:14:55 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xj9Nv57Fgz4pZl for ; Tue, 5 Nov 2024 01:14:55 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x102e.google.com with SMTP id 98e67ed59e1d1-2e2bb1efe78so3467089a91.1 for ; Mon, 04 Nov 2024 17:14:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1730769294; x=1731374094; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=8RrC948142/hoeAVeAzOdY9ha8W1/8iwClAgJ4lHIHc=; b=sr31Uxf/0E8rbny/WxhIzpNGcgm8Jfo7wEmXCxaIx8itWNLGXMcy1+yf12kBWahrP4 M4FX61Rlrd/Kopz5LfcGmE5NmXH49ALieqJzmWTHZZtpJRbGMYYxnTvb/dcegxEY3bT+ 7/RqEzHOLtKmGV2XyAVxTDDj4/auIl9GpZa5qV5Goa7T1TNKQdKSBVCjHP+YWUwLmSln kFzZaOrvKiBuoa316xXvM85aZomZ9a1ElDDeSUAdHS/sh3v6siqwj8IMyYfJVIrTPGOy zezMmsSKiZoLZyyxKXiXQWuONjMX5JttXpcHfghjFGWWItY4tUtAJg5wf1YKcgkSZOxk uY6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730769294; x=1731374094; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=8RrC948142/hoeAVeAzOdY9ha8W1/8iwClAgJ4lHIHc=; b=GHsRlmt0KTPjHWdd/Xl7sDgaCNDWmkAJETbaqzuVH+rSwrS545vTtTcHi69umULtKj FoVmcb2LPLwQjh1xceo6no2QglMAVETFsElNQDtD0vuUO7AT69IDVFJJiLdlb2BXGJV9 Ji5FI2MYigkgT9RrdiiZsvfxGMTc/aFKOMsW4X1f7hjyXIHu/J/9m7aLngufgq4G107a Qz5my7mE01Ci8/919NFSB7Ye4jHE0yN43xCbFMwB/T/ee6HB/Sz/M1jwGW2//zSZatVg LFuns3oMsAv+fJcoCIP1djfM/tQoQ9OnRNupOuk6QlpE9SIfELgbBG1lUDt26zY9H0JA uKyA== X-Gm-Message-State: AOJu0YyigoYQXSEHOjJvR5gIU2t7U7XuOOXlHH5vSbBOIDOqqMgV8xRv 2YgzE0+Po9gRrjU7Vi6nC9xT+P5sfP2SsUbPJ0fLlnnIVssuYprAEjBocU9d0K5yXvfYknv3cq7 yV2j8/g+VLs4qHTElgmazKTIzCa55WXq+QhQcPRj9BdRNII6rfzpyAA== X-Google-Smtp-Source: AGHT+IGP6G/rMsDjRLFuekcZkhAIh5eu3gALCckX2U25KX1mYPSxBrwyAIiObTNism65h47j0UBf2md/NOAWAiHfKec= X-Received: by 2002:a17:90b:4d0d:b0:2e9:3056:71dd with SMTP id 98e67ed59e1d1-2e94bce054fmr21611681a91.7.1730769294056; Mon, 04 Nov 2024 17:14:54 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <3293802b-3785-4715-8a6b-0802afb6f908@app.fastmail.com> In-Reply-To: <3293802b-3785-4715-8a6b-0802afb6f908@app.fastmail.com> From: Warner Losh Date: Mon, 4 Nov 2024 18:14:43 -0700 Message-ID: Subject: Re: nvme device errors & zfs To: Dave Cottlehuber Cc: freebsd-fs Content-Type: multipart/alternative; boundary="000000000000f5fce406262020be" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Xj9Nv57Fgz4pZl X-Spamd-Bar: ---- --000000000000f5fce406262020be Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Nov 4, 2024 at 10:31=E2=80=AFAM Dave Cottlehuber = wrote: > What's the best way to see error counters or states on an nvme > device? > Sadly, I think dmesg | grep nvme and/or trolling through /var/log/messages. Nvme drives don't generally keep good counters of errors... > I have a typical mirrored nvme zpool, that reported enough errors > in a burst last week, that 1 drive dropped off the bus [1]. > > After a reboot, it resilvered, I cleared the errors, and it seems > fine according to repeated scrubs and a few days of use. > > I was unable to see any errors from the nvme drive itself, but > as its (just) in warranty for 2 more weeks I'd like to know > if I should return it. > > I installed ports `sysutils/nvme-cli` and didn't see anything > of note there either: > > $ doas nvme smart-log /dev/nvme1 > 0xc0484e41: opc: 0x2 fuse: 0 cid 0 nsid:0xffffffff cmd2: 0 cmd3: 0 > : cdw10: 0x7f0002 cdw11: 0 cdw12: 0 cdw13: 0 > : cdw14: 0 cdw15: 0 len: 0x200 is_read: 0 > <--- 0 cid: 0 status 0 > Smart Log for NVME device:nvme1 namespace-id:ffffffff > critical_warning : 0 > temperature : 39 C > available_spare : 100% > available_spare_threshold : 10% > percentage_used : 3% > data_units_read : 121681067 > data_units_written : 86619659 > host_read_commands : 695211450 > host_write_commands : 2187823697 > controller_busy_time : 2554 > power_cycles : 48 > power_on_hours : 6342 > unsafe_shutdowns : 38 > media_errors : 0 > num_err_log_entries : 0 > Warning Temperature Time : 0 > Critical Composite Temperature Time : 0 > This suggests that the only 'badness' is 38 unsafe shutdowns (likely power failures), since either there were a bunch all at once (maybe when installing) or you've had power off events every week... There's been no reported media errors (or the drive hasn't done a good job of remembering them, though most NVME is better than most for that). > Temperature Sensor 1 : 39 C > Temperature Sensor 2 : 43 C > Thermal Management T1 Trans Count : 0 > Thermal Management T2 Trans Count : 0 > Thermal Management T1 Total Time : 0 > Thermal Management T2 Total Time : 0 > There's been no time where the drive overheated either. That's good. > [1]: zpool status > status: One or more devices are faulted in response to persistent errors. > Sufficient replicas exist for the pool to continue functioning in= a > degraded state. > action: Replace the faulted device, or use 'zpool clear' to mark the devi= ce > repaired. > scan: scrub repaired 0B in 00:17:59 with 0 errors on Thu Oct 31 16:24:3= 6 > 2024 > config: > > NAME STATE READ WRITE CKSUM > zroot DEGRADED 0 0 0 > mirror-0 DEGRADED 0 0 0 > gpt/zfs0 ONLINE 0 0 0 > gpt/zfs1 FAULTED 0 0 0 too many errors > I'm not sure how to reconcile this in the face of the above. I'd have to see the dmesg / messages logs for any non-boot messages for nvme / nda. For bad drives at work, I typically see something like: /var/log/messages.0.bz2:Nov 3 02:48:54 c001 kernel: nvme2: Resetting controller due to a timeout. /var/log/messages.0.bz2:Nov 3 02:48:54 c001 kernel: nvme2: Waiting for reset to complete /var/log/messages.0.bz2:Nov 3 02:49:05 c001 kernel: nvme2: controller ready did not become 0 within 10500 ms for drives that just 'hang' which would cause ZFS to drop them out. I'd see if there's new firmware or return the drive. I also see: nvme8: READ sqid:3 cid:117 nsid:1 lba:1875786352 len:1024 nvme8: nsid:0x1 rsvd2:0 rsvd3:0 mptr:0 prp1:0x40defd000 prp2:0x1395a2400 nvme8: cdw10: 0x6fce3a70 cdw11:0 cdw12:0x3ff cdw13:0 cdw14:0 cdw15:0 nvme8: UNRECOVERED READ ERROR (02/81) crd:0 m:1 dnr:1 p:1 sqid:3 cid:117 cdw0:0 (nda8:nvme8:0:0:1): READ. NCB: opc=3D2 fuse=3D0 nsid=3D1 prp1=3D0 prp2=3D0 cdw=3D6fce3a70 0 3ff 0 0 0 (nda8:nvme8:0:0:1): CAM status: NVME Status Error (nda8:nvme8:0:0:1): Error 5, Retries exhausted g_vfs_done():nda8p8[READ(offset=3D960402063360, length=3D1048576)]error =3D= 5 when there's a media error. But the brand of NVMe drives we by report this as an error: c029.for002.ix# nvmecontrol logpage -p 2 nvme8 SMART/Health Information Log =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D Critical Warning State: 0x04 Available spare: 0 Temperature: 0 Device reliability: 1 Read only: 0 Volatile memory backup: 0 [[... but this says the drive has lost data ]] Power cycles: 106 Power on hours: 30250 Unsafe shutdowns: 19 Media errors: 3 No. error info log entries: 3 Warning Temp Composite Time: 0 Error Temp Composite Time: 0 Temperature 1 Transition Count: 0 Temperature 2 Transition Count: 0 Total Time For Temperature 1: 0 Total Time For Temperature 2: 0 so there's 3 media errors. I can read the log page to find the LBA too (I'm working on enhancing the errors we report for NVMe to include LBA of first error too, but that's not there yet). But since you don't have any media errors, I'd check history to see if the nvme drives are resetting (either successfully or not). But I don't know how to get that data from just the drive logs. Warner --000000000000f5fce406262020be Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Nov 4, 2024 at 10:31=E2=80=AF= AM Dave Cottlehuber <dch@freebsd.org<= /a>> wrote:
W= hat's the best way to see error counters or states on an nvme
device?

I have a typical mirrored nvme zpool, that reported enough errors
in a burst last week, that 1 drive dropped off the bus [1].

After a reboot, it resilvered, I cleared the errors, and it seems
fine according to repeated scrubs and a few days of use.

I was unable to see any errors from the nvme drive itself, but
as its (just) in warranty for 2 more weeks I'd like to know
if I should return it.

I installed ports `sysutils/nvme-cli` and didn't see anything
of note there either:

$ doas nvme smart-log /dev/nvme1
0xc0484e41: opc: 0x2 fuse: 0 cid 0 nsid:0xffffffff cmd2: 0 cmd3: 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : cdw10: 0x7f0002 cdw11: 0 cdw12: 0 cdw1= 3: 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : cdw14: 0 cdw15: 0 len: 0x200 is_read: = 0
<--- 0 cid: 0 status 0
Smart Log for NVME device:nvme1 namespace-id:ffffffff
critical_warning=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 : 0
temperature=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0: 39 C
available_spare=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0: 100%
available_spare_threshold=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 10%
percentage_used=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0: 3%
data_units_read=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0: 121681067
data_units_written=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 : 86619659
host_read_commands=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 : 695211450
host_write_commands=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0: 2187823697
controller_busy_time=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= : 2554
power_cycles=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 : 48
power_on_hours=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 : 6342
unsafe_shutdowns=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 : 38
media_errors=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 : 0
num_err_log_entries=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0: 0
Warning Temperature Time=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0
Critical Composite Temperature Time : 0

This suggests that the only 'badness' is 38 unsafe shutdowns (like= ly power failures), since

Temperature Sensor 1=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= : 39 C
Temperature Sensor 2=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= : 43 C
Thermal Management T1 Trans Count=C2=A0 =C2=A0: 0
Thermal Management T2 Trans Count=C2=A0 =C2=A0: 0
Thermal Management T1 Total Time=C2=A0 =C2=A0 : 0
Thermal Management T2 Total Time=C2=A0 =C2=A0 : 0

=
There's been no time where the drive overheated either. That= 's good.
=C2=A0
[1]: zpool status
status: One or more devices are faulted in response to persistent errors. =C2=A0 =C2=A0 =C2=A0 =C2=A0 Sufficient replicas exist for the pool to conti= nue functioning in a
=C2=A0 =C2=A0 =C2=A0 =C2=A0 degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark th= e device
=C2=A0 =C2=A0 =C2=A0 =C2=A0 repaired.
=C2=A0 scan: scrub repaired 0B in 00:17:59 with 0 errors on Thu Oct 31 16:2= 4:36 2024
config:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 NAME=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 STATE=C2= =A0 =C2=A0 =C2=A0READ WRITE CKSUM
=C2=A0 =C2=A0 =C2=A0 =C2=A0 zroot=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0DEGRADED= =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 mirror-0=C2=A0 =C2=A0 DEGRADED=C2=A0 =C2= =A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 gpt/zfs0=C2=A0 ONLINE=C2=A0 =C2= =A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 gpt/zfs1=C2=A0 FAULTED=C2=A0 =C2= =A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 too many error= s

I'm not sure how to reconcile thi= s in the face of the above. I'd have to see the
dmesg / messa= ges logs for any non-boot messages for nvme / nda.=C2=A0 For bad drives
at work, I typically see something like:

/var/lo= g/messages.0.bz2:Nov =C2=A03 02:48:54 c001 kernel: nvme2: Resetting control= ler due to a timeout.
/var/log/messages.0.bz2:Nov =C2=A03 02:48:54 c001 = kernel: nvme2: Waiting for reset to complete
/var/log/messages.0.bz2:Nov= =C2=A03 02:49:05 c001 kernel: nvme2: controller ready did not become 0 wit= hin 10500 ms

for drives that just 'hang' whi= ch would cause ZFS to drop them out. I'd see if there's new firmwar= e or return
the drive.

I also see:
=
nvme8: READ sqid:3 cid:117 nsid:1 lba:1875786352 len:1024
nvme8: ns= id:0x1 rsvd2:0 rsvd3:0 mptr:0 prp1:0x40defd000 prp2:0x1395a2400
nvme8: c= dw10: 0x6fce3a70 cdw11:0 cdw12:0x3ff cdw13:0 cdw14:0 cdw15:0
nvme8: UNRE= COVERED READ ERROR (02/81) crd:0 m:1 dnr:1 p:1 sqid:3 cid:117 cdw0:0
(nd= a8:nvme8:0:0:1): READ. NCB: opc=3D2 fuse=3D0 nsid=3D1 prp1=3D0 prp2=3D0 cdw= =3D6fce3a70 0 3ff 0 0 0
(nda8:nvme8:0:0:1): CAM status: NVME Status Erro= r
(nda8:nvme8:0:0:1): Error 5, Retries exhausted
g_vfs_done():nda8p8[= READ(offset=3D960402063360, length=3D1048576)]error =3D 5
when there's a media error. But the brand of NVMe drives we= by report this as an error:

c029.for002.ix# nvmec= ontrol logpage -p 2 nvme8
SMART/Health Information Log
=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3DCritical Warning State: =C2=A0 =C2=A0 =C2=A0 =C2=A0 0x04
=C2=A0Availabl= e spare: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0
=C2=A0Temper= ature: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0
= =C2=A0Device reliability: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01
=C2= =A0Read only: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 0
=C2=A0Volatile memory backup: =C2=A0 =C2=A0 =C2=A0 =C2=A00<= /div>
[[... but this says the drive has lost data ]]
Power cy= cles: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 106Power on hours: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 30= 250
Unsafe shutdowns: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1= 9
Media errors: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 3
No. error info log entries: =C2=A0 =C2=A0 3
Warning Temp Com= posite Time: =C2=A0 =C2=A00
Error Temp Composite Time: =C2=A0 =C2=A0 =C2= =A00
Temperature 1 Transition Count: 0
Temperature 2 Transition Count= : 0
Total Time For Temperature 1: =C2=A0 0
Total Time For Temperature= 2: =C2=A0 0

so there's 3 media errors. I = can read the log page to find the LBA too (I'm working on
enh= ancing the errors we report for NVMe to include LBA of first error too, but= that's not
there yet).

But since yo= u don't have any media errors, I'd check history to see if the nvme= drives
are resetting (either successfully or not). But I don'= ;t know how to get that data from just
the drive logs.
=
Warner
--000000000000f5fce406262020be-- From nobody Tue Nov 5 09:14:59 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XjN3D3FXTz5bqqQ for ; Tue, 05 Nov 2024 09:15:20 +0000 (UTC) (envelope-from dch@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XjN3D2j9dz4dMp; Tue, 5 Nov 2024 09:15:20 +0000 (UTC) (envelope-from dch@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730798120; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/YEtnUF4QENFU65afaD/Gwj9NGZ0nE5ZLYwg0txheyk=; b=iWvjwvx4L8BVKDOFmOietjJvkonZQgvYH4YYY2fFWVDfX4KD4l/K+7ytkdXPkA6eKI7LrE O1JwnYbU147wrTS758gEnTVic6mNxE7mCNIXcxoSJlCU2mTARiTr4zrx1v7UWoTBVDnC72 XaRxZGRhM8fW6gzsXW0X3mASa3ORyoUXYxLJEUOsvzZuXc3vbwRooLX0cjsVwn8ZDmCKOE UIxPDiE+ITnQfGhhsALJaaLOEI3kS8Zw59eThPGZh/rILv02GFAN5nHWP9Rh6svB8ilRJY h7PMrxmPS1ANvQobB6k2q9r1o/F0Ka/qXLk9vAelQ1IyCExNdnI66toANW7oAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730798120; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/YEtnUF4QENFU65afaD/Gwj9NGZ0nE5ZLYwg0txheyk=; b=vQUTrlawZvl+NMQuUxAIe3qGEj8228u48ggGW4C9OH5oKtU19TlTIiVOZGUX3Oq4CbLiqF TQSNY0FDDZk6r0791jcRW650wrvk/Waq+IzzOiaEpnh5mf7Liw+h2HOnOpNZrvKXRF88pa L4nI+Dc/oMP5gcGb3q3GMuv8owtFIivpnkKTsj3L8AUp+WaNqPFnZRwvh3UwcuZbUPckcx BREfuI9oo3e+MLuhfCIUDNdwomt/oBc+OPETGuQdJiPRrweYsdQbOlWwPCeSsCazdfP6nm fFMwolNF+A35YsuyY/58+zbaNo+cgoegoibp1OQ4/cdx3ljcefnX45aZvBTFrg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1730798120; a=rsa-sha256; cv=none; b=pug2WthiNInDc9CRe0ggPdLEiFMa/u3ZnvVIf3rx0iK7usGvFNw4/6PhUmCn/IPEYls+8Y FdFAvfY0Pmy1Fs5SLjijqsCCi3LZV6SSTeU552bdO8+/GWBfqWwQkcCL2Ji8yZXog0FGdh Y/QgjVdg/QJl4bMt3K2IIeQUSqoNUq3ptfXxVxujnHWMcFRTPg5VwmeJwNGg9EB1bRN0k2 7jH87/KJfc3jSEPNr18zuK7qpKjpSClyV7Lw2Vm5SshSoPh95Qm9DQGz5RZH2x9081fro+ kXXRZ0llgJnoYXf6494v2V2OiQ21s5tROWOV56IJ8aH2FQkgzXDRCuI4nQhMoA== Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com [103.168.172.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: dch/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4XjN3D1zrGz1G4R; Tue, 5 Nov 2024 09:15:20 +0000 (UTC) (envelope-from dch@freebsd.org) Received: from phl-compute-02.internal (phl-compute-02.phl.internal [10.202.2.42]) by mailfauth.phl.internal (Postfix) with ESMTP id C707F1200043; Tue, 5 Nov 2024 04:15:19 -0500 (EST) Received: from phl-imap-02 ([10.202.2.81]) by phl-compute-02.internal (MEProxy); Tue, 05 Nov 2024 04:15:19 -0500 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdelkedgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpefoggffhf fvvefkjghfufgtgfesthhqredtredtjeenucfhrhhomhepfdffrghvvgcuvehothhtlhgv hhhusggvrhdfuceouggthhesfhhrvggvsghsugdrohhrgheqnecuggftrfgrthhtvghrnh ephfffudefhfdvueefjeehveehudejtddvfeegveetteejteekjeevgeeftdeliedtnecu ffhomhgrihhnpehgihhthhhusgdrtghomhdpshhkuhhnkhifvghrkhhsrdgrthenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegutghhodhmvghs mhhtphgruhhthhhpvghrshhonhgrlhhithihqdduvdegledutdefgeduqdduvddufeduud ejjedquggthheppehfrhgvvggsshgurdhorhhgsehfrghsthhmrghilhdrfhhmpdhnsggp rhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehimhhpsegssh guihhmphdrtghomhdprhgtphhtthhopehfrhgvvggsshguqdhfshesfhhrvggvsghsugdr ohhrgh X-ME-Proxy: Feedback-ID: icedc46df:Fastmail Received: by mailuser.phl.internal (Postfix, from userid 501) id 9A5A9B00068; Tue, 5 Nov 2024 04:15:19 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Date: Tue, 05 Nov 2024 09:14:59 +0000 From: "Dave Cottlehuber" To: "Warner Losh" Cc: freebsd-fs Message-Id: In-Reply-To: References: <3293802b-3785-4715-8a6b-0802afb6f908@app.fastmail.com> Subject: Re: nvme device errors & zfs Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Tue, 5 Nov 2024, at 01:14, Warner Losh wrote: > On Mon, Nov 4, 2024 at 10:31=E2=80=AFAM Dave Cottlehuber wrote: >> What's the best way to see error counters or states on an nvme >> device? > > Sadly, I think dmesg | grep nvme and/or trolling through /var/log/mess= ages. > Nvme drives don't generally keep good counters of errors... Thanks Warner, good to know. brb, writing all my critical data onto stone tablets and vellum... mirrored vdevs from different batches & vendors.. these are samsung 990, mainly chosen for low price at the time:=20 nda0: nda1: https://gist.github.com/dch/6523082557066f7d95af82000be74a8c is more readable version of below. I forgot to mention dmesg prior: Oct 31 16:11:05 wintermute kernel[9406]: nvme1: Resetting controller due= to a timeout. Oct 31 16:11:05 wintermute kernel[9406]: nvme1: event=3D"start" Oct 31 16:11:05 wintermute kernel[9406]: nvme1: Waiting for reset to com= plete Oct 31 16:11:05 wintermute kernel[9406]: nvme1: Waiting for reset to com= plete ... repeated x400 then the vdev is yanked by zfs: Oct 31 16:11:26 wintermute.skunkwerks.at ZFS[30294]: vdev I/O failure, z= pool=3Dzroot path=3D/dev/gpt/zfs1 offset=3D270336 size=3D8192 error=3D6 Oct 31 16:11:26 wintermute.skunkwerks.at ZFS[30305]: vdev state changed,= pool_guid=3D16468628842577781486 vdev_guid=3D1873129772219000696 Oct 31 16:11:26 wintermute.skunkwerks.at ZFS[30309]: vdev is removed, po= ol_guid=3D16468628842577781486 vdev_guid=3D1873129772219000696 Oct 31 16:11:26 wintermute.skunkwerks.at ZFS[30313]: vdev state changed,= pool_guid=3D16468628842577781486 vdev_guid=3D1873129772219000696 Oct 31 16:11:26 wintermute kernel[9426]: nvme1: Waiting for reset to com= plete Oct 31 16:11:26 wintermute kernel[9426]: nvme1: Waiting for reset to com= plete Oct 31 16:11:26 wintermute kernel[9426]: nvme1: Waiting for reset to com= plete Oct 31 16:11:26 wintermute kernel[9426]: nvme1: Waiting for reset to com= plete Oct 31 16:11:26 wintermute kernel[9426]: nvme1: Waiting for reset to com= plete Oct 31 16:11:26 wintermute kernel[9426]: nvme1: controller ready did not= become 0 within 20500 ms Oct 31 16:11:26 wintermute kernel[9426]: nvme1: event=3D"timed_out" Oct 31 16:11:26 wintermute kernel[9426]: nvme1: failing outstanding i/o Oct 31 16:11:26 wintermute kernel[9426]: nvme1: READ sqid:3 cid:126 nsid= :1 lba:667089368 len:2048 Oct 31 16:11:26 wintermute kernel[9426]: nvme1: ABORTED - BY REQUEST (00= /07) crd:0 m:0 dnr:1 p:0 sqid:3 cid:126 cdw0:0 Oct 31 16:11:26 wintermute kernel[9426]: nvme1: WRITE sqid:3 cid:121 nsi= d:1 lba:2533579032 len:256 Oct 31 16:11:26 wintermute kernel[9426]: nvme1: ABORTED - BY REQUEST (00= /07) crd:0 m:0 dnr:1 p:0 sqid:3 cid:121 cdw0:0 Oct 31 16:11:26 wintermute kernel[9426]: nvme1: WRITE sqid:3 cid:127 nsi= d:1 lba:2533579456 len:256 Oct 31 16:11:26 wintermute kernel[9426]: (nda1:nvme1:0:0:1): READ. NCB: = opc=3D2 fuse=3D0 nsid=3D1 prp1=3D0 prp2=3D0 cdw=3D27c2f9d8 0 7ff 0 0 0 Oct 31 16:11:26 wintermute kernel[9426]: nvme1: ABORTED - BY REQUEST (00= /07) crd:0 m:0 dnr:1 p:0 sqid:3 cid:127 cdw0:0 Oct 31 16:11:26 wintermute kernel[9426]: (nda1:nvme1:0:0:1): CAM status:= NVME Status Error Oct 31 16:11:26 wintermute kernel[9426]: (nda1:nvme1:0:0:1): Error 5, Re= tries exhausted Oct 31 16:11:26 wintermute kernel[9426]: nvme1: failing outstanding i/o this also repeated a few times. >> I have a typical mirrored nvme zpool, that reported enough errors >> in a burst last week, that 1 drive dropped off the bus [1]. >>=20 >> After a reboot, it resilvered, I cleared the errors, and it seems >> fine according to repeated scrubs and a few days of use. >>=20 >> I was unable to see any errors from the nvme drive itself, but >> as its (just) in warranty for 2 more weeks I'd like to know >> if I should return it. >>=20 >> I installed ports `sysutils/nvme-cli` and didn't see anything=20 >> of note there either: >>=20 >> $ doas nvme smart-log /dev/nvme1 [snip] `nvmecontrol logpage -p 2 nvme1` returns the same info as the ports nvme-cli tool, but its in base already. I will add this example to the manpage, as I tried logpage 0 & 1 without getting any info back. >> 0xc0484e41: opc: 0x2 fuse: 0 cid 0 nsid:0xffffffff cmd2: 0 cmd3: 0 >> : cdw10: 0x7f0002 cdw11: 0 cdw12: 0 cdw13: 0 >> : cdw14: 0 cdw15: 0 len: 0x200 is_read: 0 >> <--- 0 cid: 0 status 0 >> Smart Log for NVME device:nvme1 namespace-id:ffffffff >> critical_warning : 0 >> temperature : 39 C >> available_spare : 100% >> available_spare_threshold : 10% >> percentage_used : 3% >> data_units_read : 121681067 >> data_units_written : 86619659 >> host_read_commands : 695211450 >> host_write_commands : 2187823697 >> controller_busy_time : 2554 >> power_cycles : 48 >> power_on_hours : 6342 >> unsafe_shutdowns : 38 >> media_errors : 0 >> num_err_log_entries : 0 >> Warning Temperature Time : 0 >> Critical Composite Temperature Time : 0 > > This suggests that the only 'badness' is 38 unsafe shutdowns (likely=20 > power failures), since > either there were a bunch all at once (maybe when installing) or you'v= e=20 > had power off events > every week... Could that be panics? This box typically runs for a fortnight and then gets CURRENT bumped. It also feels like a lot more panics than I would have, I don't know if these counters were zero on purchase tho. > There's been no reported media errors (or the drive hasn't done a good=20 > job of remembering > them, though most NVME is better than most for that). >=20 >> Temperature Sensor 1 : 39 C >> Temperature Sensor 2 : 43 C >> Thermal Management T1 Trans Count : 0 >> Thermal Management T2 Trans Count : 0 >> Thermal Management T1 Total Time : 0 >> Thermal Management T2 Total Time : 0 > > There's been no time where the drive overheated either. That's good. >=20 >> [1]: zpool status >> status: One or more devices are faulted in response to persistent err= ors. >> Sufficient replicas exist for the pool to continue functionin= g in a >> degraded state. >> action: Replace the faulted device, or use 'zpool clear' to mark the = device >> repaired. >> scan: scrub repaired 0B in 00:17:59 with 0 errors on Thu Oct 31 16:= 24:36 2024 >> config: >>=20 >> NAME STATE READ WRITE CKSUM >> zroot DEGRADED 0 0 0 >> mirror-0 DEGRADED 0 0 0 >> gpt/zfs0 ONLINE 0 0 0 >> gpt/zfs1 FAULTED 0 0 0 too many errors > > I'm not sure how to reconcile this in the face of the above. I'd have = to see the > dmesg / messages logs for any non-boot messages for nvme / nda. For b= ad drives > at work, I typically see something like: > > /var/log/messages.0.bz2:Nov 3 02:48:54 c001 kernel: nvme2: Resetting=20 > controller due to a timeout. > /var/log/messages.0.bz2:Nov 3 02:48:54 c001 kernel: nvme2: Waiting fo= r=20 > reset to complete > /var/log/messages.0.bz2:Nov 3 02:49:05 c001 kernel: nvme2: controller=20 > ready did not become 0 within 10500 ms > > for drives that just 'hang' which would cause ZFS to drop them out. I'= d=20 > see if there's new firmware or return > the drive. I will give this a go if this issue returns, its the only one I had in a year so far. > I also see: > nvme8: READ sqid:3 cid:117 nsid:1 lba:1875786352 len:1024 > nvme8: nsid:0x1 rsvd2:0 rsvd3:0 mptr:0 prp1:0x40defd000 prp2:0x1395a24= 00 > nvme8: cdw10: 0x6fce3a70 cdw11:0 cdw12:0x3ff cdw13:0 cdw14:0 cdw15:0 > nvme8: UNRECOVERED READ ERROR (02/81) crd:0 m:1 dnr:1 p:1 sqid:3=20 > cid:117 cdw0:0 > (nda8:nvme8:0:0:1): READ. NCB: opc=3D2 fuse=3D0 nsid=3D1 prp1=3D0 prp2= =3D0=20 > cdw=3D6fce3a70 0 3ff 0 0 0 > (nda8:nvme8:0:0:1): CAM status: NVME Status Error > (nda8:nvme8:0:0:1): Error 5, Retries exhausted > g_vfs_done():nda8p8[READ(offset=3D960402063360, length=3D1048576)]erro= r =3D 5 > > when there's a media error. But the brand of NVMe drives we by report=20 > this as an error: > > c029.for002.ix# nvmecontrol logpage -p 2 nvme8 > SMART/Health Information Log > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > Critical Warning State: 0x04 > Available spare: 0 > Temperature: 0 > Device reliability: 1 > Read only: 0 > Volatile memory backup: 0 > [[... but this says the drive has lost data ]] > Power cycles: 106 > Power on hours: 30250 > Unsafe shutdowns: 19 > Media errors: 3 > No. error info log entries: 3 > Warning Temp Composite Time: 0 > Error Temp Composite Time: 0 > Temperature 1 Transition Count: 0 > Temperature 2 Transition Count: 0 > Total Time For Temperature 1: 0 > Total Time For Temperature 2: 0 > > so there's 3 media errors. I can read the log page to find the LBA too=20 > (I'm working on > enhancing the errors we report for NVMe to include LBA of first error=20 > too, but that's not > there yet). > > But since you don't have any media errors, I'd check history to see if=20 > the nvme drives > are resetting (either successfully or not). But I don't know how to ge= t=20 > that data from just > the drive logs. > > Warner There's nothing at all prior to last week, and I assume the unsafe shutdowns are unrelated panics while running CURRENT for a year. I'll cross my fingers and hope this is a transient bus problem. thanks! Dave =E2=80=94=E2=80=94=E2=80=94 O for a muse of fire, that would ascend the brightest heaven of inventio= n! From nobody Tue Nov 5 11:10:35 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XjQcV1W6Lz5bxxD for ; Tue, 05 Nov 2024 11:10:50 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-yw1-x112e.google.com (mail-yw1-x112e.google.com [IPv6:2607:f8b0:4864:20::112e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XjQcT6zPpz4mMt for ; Tue, 5 Nov 2024 11:10:49 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x112e.google.com with SMTP id 00721157ae682-6e3c3da5bcdso48749737b3.2 for ; Tue, 05 Nov 2024 03:10:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1730805049; x=1731409849; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=416zyT7ZIJNGa3OT2M+Ia9ISpWGAbrbaiUbdWJwb1d0=; b=c+COV0JW6Mshk01bX6rd4pjzZG30wOFcty0c+qcfJDS/SQVl1TzVC8eNkv3GCeGq/X GhqlTbaCSKOGB8o7V0VFn90tEPP8crat009QmR8XiDHYIiQqdI3NNt3CDAYi94dMe0v1 XEMk9b7ABMbMWQb9/EmIJKLr7Lc5peeyB6LFbKpEyb4ktQjbNSq9p4eB0nUCRjpsFlIf tWIq/MhUjwtSM6pqCVgxdLm7TGyTutMTS7qH6Em7DMASk0iFooPl/Q/YMFdf1ETty38a AecUSEcmH5+7HDsWSyWAqg8BKgLXhkZrJlM0Pg9LIMDkbMcnGkZzDAiObDNz2n9c7YcU +Q/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730805049; x=1731409849; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=416zyT7ZIJNGa3OT2M+Ia9ISpWGAbrbaiUbdWJwb1d0=; b=ANndX+HkEVhbGLYnMHQL+i1J7MhhKwKKqShcQ4q0Kr6yuGeoq8d1cgURxDqO2HJmgy hp4dQbOokOav+HeEe6xUGtTVqH92bVS3X2uZHXdzsfa2lX5tv9fS4wF14QU/lz8WcYyY BWUa7W6b9pbjKlNMgfnJ2zliju91Xflp1h9/TOZdfiCGXMywZgIWICh5FgDYv6uIpQ+4 UejlXpgnX+BaAk5fQH6HfMxUHjuVcvC/Wq0S86BzDlim2c4lKJ7NtrNg4eGbB9DH1WDx O4flqTOeGqEHUr8xlCGs0rOZVCupsyMKYmKlgIgJEYY9y4HIqetPBd6/V8tUQSX0G9Ft 2dUw== X-Forwarded-Encrypted: i=1; AJvYcCW0NB4PxEa56Pr3NodWQCIUbAMlG+4MrlmwHKJgSgPeN7rI2j95JTYZZpoEfYB+/QEz+VRMO+DJeUWZ@freebsd.org X-Gm-Message-State: AOJu0YyEXI3rJOmTe5tRL9yC5xyw3qrcgJ9KyM1x+9yEJ8TN1iNX6L/J tg29lHsoofxnjs/KZz+K/TOtzV5u63xpw9lguPuxWFXRhB280qkZCmk/EpzPbCoerAxeehuolaQ = X-Google-Smtp-Source: AGHT+IH/F55duclzy0mdr4VzeriTltUBAjCy2dM0ERqvMt9CW7YxVJ5Lwk+rfOdkE5/xGJSu9aqVpw== X-Received: by 2002:a05:690c:4b8b:b0:6ea:8109:9d16 with SMTP id 00721157ae682-6ea81099e5cmr99034707b3.7.1730805049045; Tue, 05 Nov 2024 03:10:49 -0800 (PST) Received: from mail-yb1-f170.google.com (mail-yb1-f170.google.com. [209.85.219.170]) by smtp.gmail.com with ESMTPSA id 00721157ae682-6ea6f5bd147sm16506867b3.10.2024.11.05.03.10.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 05 Nov 2024 03:10:47 -0800 (PST) Received: by mail-yb1-f170.google.com with SMTP id 3f1490d57ef6-e333af0f528so1777515276.3; Tue, 05 Nov 2024 03:10:46 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCWji0RBxzW2TLn6/KqIAce+zThWGeY6dlkS+i57W7G0iq5tfS1VHow8O7MeNmAW/eUGhVcp0btI1U1p@freebsd.org X-Received: by 2002:a05:690c:9c10:b0:6ea:4e1f:2b40 with SMTP id 00721157ae682-6ea4e1f3d77mr205226207b3.9.1730805046166; Tue, 05 Nov 2024 03:10:46 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <3293802b-3785-4715-8a6b-0802afb6f908@app.fastmail.com> In-Reply-To: From: Tomek CEDRO Date: Tue, 5 Nov 2024 12:10:35 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: nvme device errors & zfs To: Dave Cottlehuber Cc: Warner Losh , freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4XjQcT6zPpz4mMt X-Spamd-Bar: ---- On Tue, Nov 5, 2024 at 10:15=E2=80=AFAM Dave Cottlehuber = wrote: > these are samsung 990, mainly chosen for low price at the time: > nda0: > nda1: These are pretty decent and not really cheap drives! Magician software can upgrade firmware and perform other checks, works on Windoze macOS and Android: https://www.samsung.com/ca/support/model/MZ-V9P2T0B/AM/#downloads > I forgot to mention dmesg prior: > Oct 31 16:11:05 wintermute kernel[9406]: nvme1: Resetting controller due = to a timeout. > Oct 31 16:11:05 wintermute kernel[9406]: nvme1: event=3D"start" > Oct 31 16:11:05 wintermute kernel[9406]: nvme1: Waiting for reset to comp= lete > Oct 31 16:11:05 wintermute kernel[9406]: nvme1: Waiting for reset to comp= lete > ... repeated x400 Another idea is maybe disk overheats and resets itself to cool down? Lots of people in the reviews of various nvme drives asks about temperature and suggests using heatsink ;-) --=20 CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From nobody Tue Nov 5 11:19:46 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XjQqC3n9Bz5byWc for ; Tue, 05 Nov 2024 11:20:07 +0000 (UTC) (envelope-from dch@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XjQqC23wWz4n7N; Tue, 5 Nov 2024 11:20:07 +0000 (UTC) (envelope-from dch@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730805607; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EBVHTHiu2c059VnFeRiTjBgeC8anYdzT+qdT55/L0Ts=; b=CLlAeeenF4HgYZjJlaBfk7FoZOByAsRFKcv/4iOxKoNYMALaWHk2DbmguPyQNPaxCLuR7A j8ec02HBShVHOo74z12jTqYNVkFn7M/fDClbN9tjLLhPLWUXoQZ8JCOq7G7AiwidX1+SY4 7rsr+RGufnDTeVk4W6W8MYmXvcIBbaSUjek8u3D66HSJZF5vVeMyO0sivBQTi8YL5n7tAp uhF5pUfsjjLMqkQWOewIV/De4sb911uxwxh3q/L2HxqxpYMsNTlZeeVPiSQCC1oDYbxSpt Go7XNWqv/qeHMUr6ZtZ4UORW08frqNW+jT6viZX4//YG7YEY5H+YaNhSVeU5GQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730805607; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EBVHTHiu2c059VnFeRiTjBgeC8anYdzT+qdT55/L0Ts=; b=e++tQADb2oMYdEGkaQEyh7YFAebiz1NwDLD1PPNdrAyHrGjKHKwxVx4OQmUV+wIQFLaSkP J0/0nGNRCNceRVNnZbP5GJg+Vvs6kHoJTpv42pQWP8eHSPafJAsNb0nkYncu0g0mlWxJza f/Oz5Ud753wTHgIT8xmdbi2YRXEAArw64/o+UZPS6G/IaQQ81xG2gcV75bTN1DaA43oofQ Zui6jBzmEl9A5ORkEK8hkPQp306XclPxvbPH5xih4DgM+4/9vaA+Zy/sHA7iWSYU4VWzXc M6mVHRxwto2jhhURw7uy1AGckI9UtynSKBv4q+lbT7eT7wPY2L6LRUMPTHgzyg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1730805607; a=rsa-sha256; cv=none; b=p3Vwloo0ftjBU77B8ZNHbNPlrDP0eRSfmUTobI9U6Wetvz7NqwIYvIDFmy+0TCHxccwIDa mfXq8s9eoe77lA2qxQxXO+pYgCPUVG+BlOKgprdeDEDo7fn/pFHAEy/4bceqRkBNOMqS/t pvfzx5zP7eSwRtisB9F2fNucTPK2PolztyurovZrFg39m/ZDm+P+94a6n3dcuKUKAWJHvu sruOOzLVPDqWp53bX8VD+5nMPX5zl5XP5pu89z7uyZr2v/cwnmv3Gn5oOqBqHu+ttGdX4q S2EvxXezT5/lyDtnzNy4DJ5E9FnKUqKx/x+XfAwj9xocIozqNhH68+x/5SHTIg== Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com [103.168.172.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: dch/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4XjQqC1JSyz1Jvf; Tue, 5 Nov 2024 11:20:07 +0000 (UTC) (envelope-from dch@FreeBSD.org) Received: from phl-compute-02.internal (phl-compute-02.phl.internal [10.202.2.42]) by mailfauth.phl.internal (Postfix) with ESMTP id 8799D1200043; Tue, 5 Nov 2024 06:20:06 -0500 (EST) Received: from phl-imap-02 ([10.202.2.81]) by phl-compute-02.internal (MEProxy); Tue, 05 Nov 2024 06:20:06 -0500 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdelkedgvdehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpefoggffhf fvvefkjghfufgtgfesthhqredtredtjeenucfhrhhomhepfdffrghvvgcuvehothhtlhgv hhhusggvrhdfuceouggthheshfhrvggvuefuffdrohhrgheqnecuggftrfgrthhtvghrnh epuefgueekffekvdfhieetgedtvdelgeehiefgjedtleevgefhgfefgfekveekgeehnecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepuggthhdomh gvshhmthhprghuthhhphgvrhhsohhnrghlihhthidquddvgeeluddtfeeguddquddvudef uddujeejqdgutghhpeephfhrvggvuefuffdrohhrghesfhgrshhtmhgrihhlrdhfmhdpnh gspghrtghpthhtohepfedpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepihhmphes sghsughimhhprdgtohhmpdhrtghpthhtohepthhomhgvkhestggvughrohdrihhnfhhopd hrtghpthhtohepfhhrvggvsghsugdqfhhssehfrhgvvggsshgurdhorhhg X-ME-Proxy: Feedback-ID: icedc46df:Fastmail Received: by mailuser.phl.internal (Postfix, from userid 501) id 5A3E3B00068; Tue, 5 Nov 2024 06:20:06 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Date: Tue, 05 Nov 2024 11:19:46 +0000 From: "Dave Cottlehuber" To: "Tomasz CEDRO" Cc: "Warner Losh" , freebsd-fs Message-Id: <620f4e82-aa69-4af0-bd22-b21203ab8745@app.fastmail.com> In-Reply-To: References: <3293802b-3785-4715-8a6b-0802afb6f908@app.fastmail.com> Subject: Re: nvme device errors & zfs Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Tue, 5 Nov 2024, at 11:10, Tomek CEDRO wrote: > Magician software can upgrade firmware and perform other checks, works > on Windoze macOS and Android: that will be difficult, I don't have an nvme capable thing for any of those. > Another idea is maybe disk overheats and resets itself to cool down? that is a great point, the mainboard comes with inbuilt heatsinks, but when I assembled it, the 2nd nvme slot heatsink looked a lot less bulky than the other one, I remember distinctly wondering if it would cope. If it happens again I'll see if I can get a temp measurement at the time of failure. I would hope temperature throttling would not be quite so brutal, to remove itself from the bus entirely, but its a reasonable explanation. A+ Dave =E2=80=94=E2=80=94=E2=80=94 O for a muse of fire, that would ascend the brightest heaven of inventio= n! From nobody Tue Nov 5 12:38:19 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XjSYS1ZFjz5c334 for ; Tue, 05 Nov 2024 12:38:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XjSYR58QDz4t4W for ; Tue, 5 Nov 2024 12:38:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730810299; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=psYHycIbeZ97H0GLiKTKxkz8/B4MxwZ0R2h8gzwo/aA=; b=oXI4c7bRaU+6bB3uWLZd79H5cHgoBOdaERgIbgmIs8IpofO94yV6Vc1y5HjHq8iwiuDfdI NRctCRDFJGrh1BUG82VNZpdFWtV7ZN4HDLNpOqiIml6HG+OHXqGabFhBrefrXPDyGRBqeu hsecLwgf3jGUAD1sS64u0zYfR201Jl8s+nu93G7mf3EAhxt6FVJo8FYqqUVeVQgQeqMnXn TY1Z0tyjcX009U4i8REON99lVUFICcGdf+Mu9Rd42x2CgIy9ZdNqvcOwo7veWW0BPocLAY aNDGblXS53a/xdbK6VZ5KqiPTC0H03RUYk3eGLlpQoDoIPuW97zdZLTafINLbA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1730810299; a=rsa-sha256; cv=none; b=fJ2iEJUwBq9R39bRWppfIJ+9OSbj3qae9kmwAvQI4G8ZTBO38+NKSs/d6b2vHEpglg72OC RZ5Q25A90tXDZR7orF8i1b1FarSsR+mpBH0xkmY+7VNFk6yjx6ZN49m2GfF+pYnh31YYw9 2IVNI/pS6BWa8LC172ObuCIAePFcXbCpsXJTo7xbOLjqpDKCRqlNvdEBNZYPyFMg/7VGej TR0WB+yFVHrnyfLqPByEX6XkKeCuSmNl7L93EzYT+3hjSLWHm+Pi17DWrbnb+giEPsTb60 xLtXgzymZ3FPGy5aBwK3VMSXEuqiGIRvmQqHSL6DeRJkn9JFVdRYouka23RsPA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XjSYR4cZdzHYp for ; Tue, 5 Nov 2024 12:38:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A5CcJJq018535 for ; Tue, 5 Nov 2024 12:38:19 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A5CcJGa018534 for fs@FreeBSD.org; Tue, 5 Nov 2024 12:38:19 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 281734] ls .zfs/snapshot automounts all snapshots Date: Tue, 05 Nov 2024 12:38:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: fneves@registro.br X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D281734 --- Comment #2 from Frederico A C Neves --- Thanks for the clarification and sorry for the misleading subject but does = this behaviour have changed in 14? I don't have any longer access to a 13.x system but I can confirm that on a= ZoL system this behaviour is not shown. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Nov 5 12:47:15 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XjSm44VjGz5c3ty for ; Tue, 05 Nov 2024 12:47:32 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-yb1-xb2e.google.com (mail-yb1-xb2e.google.com [IPv6:2607:f8b0:4864:20::b2e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XjSm40jPYz4tkr for ; Tue, 5 Nov 2024 12:47:31 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; none Received: by mail-yb1-xb2e.google.com with SMTP id 3f1490d57ef6-e290222fdd0so4784639276.2 for ; Tue, 05 Nov 2024 04:47:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1730810851; x=1731415651; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=l3SrmbyUu3a2ZVNrbA8syINGGcYsQhlzh3xKchv9ZO8=; b=dcgjYe23n9IvV29T/AYC4sPnM+Q5t3XbZaE5MksL+if7S5pFfLIQASKg7xHqabhmBE tixkt0+9mVhWmgrvyw5laSc24VZwGcM67WENzski6QWbxIcR+3htT9KEEkLtEpTLgN4n j6xeIcZ2dPJDK268hyGg6DINTxHo7KEIsDQ4ZQql8xPVa3E4HXmJtZ/uGWfQblUFuAx0 +KN2QAavGvKyv7hFYyHRqX2Cpd71fBiR6AI0VW6vf3feQhgwB04JkCaSxY2CHKI2mgPf 5SBHxjfsE+gJZSHeitHTLIKtYWX5HFA+8/RngRJoZSAkBV9E/QnFpsOXvvA+J9wx4onr DPWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730810851; x=1731415651; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l3SrmbyUu3a2ZVNrbA8syINGGcYsQhlzh3xKchv9ZO8=; b=Nyk1JYRG5qYfzPOyi88IMDao1H55/t2POIyfHLi4a5Kj9vmjBUeh5ea1zIcP3lyP90 MlPXmpY6vCKycUPs/r347ci5BVpVCHYzjFcfC3jPEwaU0t7In/X1a+1LLMlS/GYN0j7A N40FXGQvpWnAYabsNuWk3aImzpodj63BQpu6UimMIzhJn6MVtBOgeg1wIZFys+NbykBA XUDquQ6fJ8QX8fDArDsIuQK3vD+FX8WOSuU2AqqpMavmgzK178ql0lse4YWJ4kWhjShw uBYjvEGJqPjUPEf1i5oEIUASU/18ZIgCYCsa+abiDsh2TUwaQActpGrtGNLAupSnje8j Q/Jg== X-Forwarded-Encrypted: i=1; AJvYcCWeekxq01kMT8LOAUNnC3gMppXqG2rLhXschi+KzMY3PjkXPftTjdKUMkRmKbHHwa+U9iGrLLEHF+Wz@freebsd.org X-Gm-Message-State: AOJu0YxfkzOnOb9k86O45Jw9QOcoSk+mvu2oZbsZ1hkwPMiWs5i494Rs D15bwN5eSygZTQptiHFmXGES2n7deROhRgc2ZDo5AgDHo+07MRJ5u/axSKJxRQ== X-Google-Smtp-Source: AGHT+IGxToSObMo+G/nq+QJQDvzgBGhCVP9Dzko9dEEYdtJQJ1wMaf8BuQSzswhLPt+v8TFaPXgd2A== X-Received: by 2002:a05:6902:1246:b0:e33:2270:90a0 with SMTP id 3f1490d57ef6-e332270b00emr7588073276.55.1730810850720; Tue, 05 Nov 2024 04:47:30 -0800 (PST) Received: from mail-yw1-f170.google.com (mail-yw1-f170.google.com. [209.85.128.170]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e30e8acdba1sm2465589276.42.2024.11.05.04.47.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 05 Nov 2024 04:47:29 -0800 (PST) Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-6ea5f68e17aso52556487b3.3; Tue, 05 Nov 2024 04:47:28 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCX1tTDbHBV2b4+xrYJE4i+1Z/MecKI9eQ1qbfnnPKBw04e5xCIOz0VIL1ossQQHpXdP1erILY7D7xHM@freebsd.org X-Received: by 2002:a05:690c:6c08:b0:6e3:fd6:6ccb with SMTP id 00721157ae682-6e9d8932b09mr376373327b3.13.1730810848206; Tue, 05 Nov 2024 04:47:28 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <3293802b-3785-4715-8a6b-0802afb6f908@app.fastmail.com> <620f4e82-aa69-4af0-bd22-b21203ab8745@app.fastmail.com> In-Reply-To: <620f4e82-aa69-4af0-bd22-b21203ab8745@app.fastmail.com> From: Tomek CEDRO Date: Tue, 5 Nov 2024 13:47:15 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: nvme device errors & zfs To: Dave Cottlehuber Cc: Warner Losh , freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4XjSm40jPYz4tkr X-Spamd-Bar: ---- On Tue, Nov 5, 2024 at 12:20=E2=80=AFPM Dave Cottlehuber = wrote: > On Tue, 5 Nov 2024, at 11:10, Tomek CEDRO wrote: > > Magician software can upgrade firmware and perform other checks, works > > on Windoze macOS and Android: > > that will be difficult, I don't have an nvme capable thing for any > of those. Look here :-) https://download.semiconductor.samsung.com/resources/software-resources/Sam= sung_SSD_990_PRO_4B2QJXD7.iso Also seems like nvme-cli can update firmware but there are problems on Samsungs :-) https://github.com/linux-nvme/nvme-cli/issues/2298 > > Another idea is maybe disk overheats and resets itself to cool down? > > that is a great point, the mainboard comes with inbuilt heatsinks, but > when I assembled it, the 2nd nvme slot heatsink looked a lot less > bulky than the other one, I remember distinctly wondering if it would > cope. If it happens again I'll see if I can get a temp measurement > at the time of failure. > > I would hope temperature throttling would not be quite so brutal, to > remove itself from the bus entirely, but its a reasonable explanation. I am also looking at this drive but its a bit too expensive for me right now.. I can see lots of offers with heatsink for that model so this may be the problem.. good to know what to prepare for :-) --=20 CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From nobody Tue Nov 5 13:16:15 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XjTRt5JhWz5c5Wn for ; Tue, 05 Nov 2024 13:18:34 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XjTRs0Wp7z4y4T for ; Tue, 5 Nov 2024 13:18:32 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of li-fbsd@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=li-fbsd@citylink.dinoex.sub.org; dmarc=none; arc=pass ("uucp.dinoex.org:s=M20221114:i=1") Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.18.1/8.18.1) with ESMTPS id 4A5DI7Tv017041 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Tue, 5 Nov 2024 14:18:08 +0100 (CET) (envelope-from li-fbsd@citylink.dinoex.sub.org) ARC-Seal: i=1; a=rsa-sha256; d=uucp.dinoex.org; s=M20221114; t=1730812690; cv=none; b=EhQglI+EuGflFnBl5MePKgoZ04waVz4zcE46MlGdJWBC6UzuYFBwjegnbR8FD7pIP+JBUlh0HtbXdTgR66oSu5opimgNTO6Qs2Gotfmq5y+u6yY3uzp72+hsSoaPKtqu8Uu3a5Tb4IWjb0BL2Rz6tgfVINq1uhFjeB/7ukXTXf8= ARC-Message-Signature: i=1; a=rsa-sha256; d=uucp.dinoex.org; s=M20221114; t=1730812690; c=relaxed/simple; bh=YBEJ1EXpiFIU+kzFyCkTod/XaIifYMqlUi/lI8o2AQE=; h=Received:Received:Received:X-Authentication-Warning:From: X-Newsgroups:Subject:Date:Message-ID:References:Injection-Date: Injection-Info:User-Agent:To:X-Milter:X-Greylist; b=EqWalCHgSQuqcYwycWxvEFtcCry3gZEQs8p5DJoVMCAqRv3+SI0cRjlLdF27FhbijhzTTZau+dzT9QPae4OkaIh9SqA5v1vP0gGTn+QHVF4CRvOsKVLjhRT57pJCrtBiM12B75wJrew985PdUPYvYCqNgP6io50bKr1YD9KzvFE= ARC-Authentication-Results: i=1; uucp.dinoex.org X-MDaemon-Deliver-To: Received: (from uucp@localhost) by uucp.dinoex.org (8.18.1/8.18.1/Submit) with UUCP id 4A5DI7ZT017039 for freebsd-fs@freebsd.org; Tue, 5 Nov 2024 14:18:07 +0100 (CET) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from admn.intra.daemon.contact (localhost [127.0.0.1]) by admn.intra.daemon.contact (8.18.1/8.18.1) with ESMTPS id 4A5DGY52008785 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Tue, 5 Nov 2024 14:16:34 +0100 (CET) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from intra.daemon.contact (news@localhost) by admn.intra.daemon.contact (8.18.1/8.18.1/Submit) with NNTP id 4A5DGFj6008551 for freebsd-fs@freebsd.org; Tue, 5 Nov 2024 14:16:15 +0100 (CET) (envelope-from li-fbsd@citylink.dinoex.sub.org) X-Authentication-Warning: admn.intra.daemon.contact: news set sender to li-fbsd@citylink.dinoex.sub.org using -f From: "Peter 'PMc' Much" X-Newsgroups: m2n.fbsd.fs Subject: Re: nvme device errors & zfs Date: Tue, 5 Nov 2024 13:16:15 -0000 (UTC) Message-ID: References: <3293802b-3785-4715-8a6b-0802afb6f908@app.fastmail.com> <620f4e82-aa69-4af0-bd22-b21203ab8745@app.fastmail.com> Injection-Date: Tue, 5 Nov 2024 13:16:15 -0000 (UTC) Injection-Info: admn.intra.daemon.contact; logging-data="87927"; mail-complaints-to="usenet@citylink.dinoex.sub.org" User-Agent: slrn/1.0.3 (FreeBSD) To: freebsd-fs@freebsd.org X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Tue, 05 Nov 2024 14:18:10 +0100 (CET) X-Spamd-Result: default: False [-3.92 / 15.00]; ARC_ALLOW(-1.00)[uucp.dinoex.org:s=M20221114:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; NEURAL_HAM_LONG(-0.99)[-0.992]; NEURAL_HAM_SHORT(-0.93)[-0.927]; FORGED_SENDER(0.30)[pmc@citylink.dinoex.sub.org,li-fbsd@citylink.dinoex.sub.org]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; R_DKIM_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; HAS_XAW(0.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; FROM_NEQ_ENVFROM(0.00)[pmc@citylink.dinoex.sub.org,li-fbsd@citylink.dinoex.sub.org]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[sub.org]; RCVD_TLS_LAST(0.00)[]; TO_DN_NONE(0.00)[] X-Rspamd-Queue-Id: 4XjTRs0Wp7z4y4T X-Spamd-Bar: --- List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org On 2024-11-05, Dave Cottlehuber wrote: > I would hope temperature throttling would not be quite so brutal, to > remove itself from the bus entirely, but its a reasonable explanation. It might be a reasonable choice to protect the data first. Also people will then notice that there is a problem and not complain about bad performance. If a more elegant reaction is desired, that might be implemented by obtaining the current temperature and dynamically issuing some "nvmecontrol power -p x -w y ..." as appropriate. (From what I hear, these options behave rather device specific, so some testing may be required) https://gitr.daemon.contact/tools/tree/heatctl.rb#n218 I'm not yet doing temperature-driven nvme performance steering, but practically everything else: fan engage, scrub pausing, cpu consumtion (via rctl) etc. cheerio, PMc From nobody Tue Nov 5 14:46:35 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XjWPR6zdjz5cBlh for ; Tue, 05 Nov 2024 14:46:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XjWPR5wK1z56kK for ; Tue, 5 Nov 2024 14:46:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730817995; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AmD9VLPlB5GgSaizHOo5sfV4Y6pVZaSNHjzeEaPuZGo=; b=BKGGxtRKw8hkL4LYEpZIZMbMc1fWloBV/9lujRpD07p9/iTIPrddlp5LtQg6nTGNB1hszo mXWp4uMO37vKgcN8gxMDl95uKEVuhGZJGt84PAl72pOIr9h+A4zLPQgYRc7TA7PRdmI7Oc xoZ2gYQtdMRZcdQGnj+e60q3leZ3EemBWr3Y+2eydxkKLhpaAE6Z9ck7+uN7/7C/y8i08j JRgUek/I3iSs3K3gaROIAUsVLM6a8+dXs+3QckZ3R/sbzgwwghhL1wjW2gx1bUJie8UbzB qUzyb6W/nDMJBLcAmxQfVIiaf0cV3EeuIq6z822jNM6CkTe0jd0VAc9hOZqhdw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1730817995; a=rsa-sha256; cv=none; b=qW5SfCk+8H1qKmDt2AKeVI+odOKSTXU4PXzpYixYy0UXrmch3puPKFWT+6obcbG3+kQZMQ 2pq2ylk0S8g6dnRwaMmrhaYQAnEZu9GpnhHSOlkZvys+MG465/JAR9iRoGUYwwkaBl4RYe LqVNs3Gs9fdnfxKFlXjKzIr3Jybxjhh63CdWcXkIS471/P+Lh0Kn/nNs1FY9Hsz0Uurxrd o4QHbm8izLtDwSbRh/K3bgfndd5MeP2yfCtzl9qsj81xX6+UFKxM7FPNVajI7uLUqnyk6X 7qwCddySLXOFhz0IzOBmyY3vwa1tOQOvLofvVQWFTuCZlsKjDIm+MdVZmfLnag== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XjWPR5HqCzMC4 for ; Tue, 5 Nov 2024 14:46:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A5EkZvT028787 for ; Tue, 5 Nov 2024 14:46:35 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A5EkZJF028785 for fs@FreeBSD.org; Tue, 5 Nov 2024 14:46:35 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 281734] ls .zfs/snapshot automounts all snapshots Date: Tue, 05 Nov 2024 14:46:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D281734 --- Comment #3 from Andriy Gapon --- (In reply to Frederico A C Neves from comment #2) On FreeBSD it's been like this as long as I can recall. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Nov 5 17:21:25 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XjZrL08CYz5cLyY for ; Tue, 05 Nov 2024 17:21:38 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XjZrK4CZJz49Sj for ; Tue, 5 Nov 2024 17:21:37 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-7ed9f1bcb6bso4012255a12.1 for ; Tue, 05 Nov 2024 09:21:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1730827296; x=1731432096; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=8wUoOFcKh/mU98sh41JycnhYHa0nEHJVb6XCDd0qtTE=; b=bTyTpI4E+BgE3yagZI8OSSGTP4+n7nF9wFTWDtATHJ+X8u3gJ1q9LTBFCh9f0bjYxu /C1OBZAMmWLcnZ3ImaVhNr6efMzVhx2KBtltQIlmvAIBlp7n0Fn07k4+UBW9KVWzCL6t tJg5/lcElp+lNOfMELibGbgCZHrf85ABGbO3sAiB8w9T6rdvG+y22CF9HDtoluDGEySa DvW+ecTv42uVEqN0cafkjegwmHV0gKVwzBCEe+ceDMavJnOOJGlHcAwkmFPulnzgi/bU pZ5oKRXQkr3tN3fL5UQHpcAKtANc0e1RJiYEFmhXOGXn6psm0EGXSq3Z/l7QNsWqFWE6 xr6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730827296; x=1731432096; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=8wUoOFcKh/mU98sh41JycnhYHa0nEHJVb6XCDd0qtTE=; b=QN4/ep5vG5ASGsahAogoGCjZt396tP/HgQdDeiJuQ6wxYh+SzJtrGj+fdoLHop8vgf Ym49RQhHMNpjPNF+irDdAS1mCFbM7BWrHC9Ia+p+VQ7kbj8xbmjsJkiACeO3VArUZTn2 83LKnd6bMwtFY7QAYRnNzQs9znw0dZXgrSqaBI+t+tWuqgIRGgNJ15Ze8EojU64G+eWg MU94JAhi4Y2vPi7YfLcXYu5PAbWSr/tNLOnUqW4bjJiwy6+dPXzVc0sypEcDDWBAoONV 3a5L2GTtb+QwYw69Ph4bzNr8ULtgYYLdYAXuY9f/roztvgHfmENabmjgFGfUUKvBX9FF bRQQ== X-Forwarded-Encrypted: i=1; AJvYcCVbDqGjg/zqn5XbpOmYPROGDnMZLG1AgaqwxmYCQBjNjCqg22O4v9Pr9Wuwx1S3e+2H0xI8MFW0RRtU@freebsd.org X-Gm-Message-State: AOJu0Yw9WdXkwusOHn3QKvCXvItLSbTGGg1pC3l9WAuu5ZM1HttYHxJT GVZMVFDW2jTVhRxt6DvGTXY4ZH0HlK97KTSVgbELHcG4ZbX1BUaaStoSBOjvxHrz35wVcFigl/H INUOxmZ7PPYF7mCuW6OB1c8cUO9RdvF0Ku3gU3A== X-Google-Smtp-Source: AGHT+IGFcP9BoHU587mIMse8+wsVJceEpTAdUbzHvSu+GxwsEMdsNo2o0LchBWHkazPY++qyAAWqgNym/vH4odFHL8Q= X-Received: by 2002:a17:90a:9a8d:b0:2c9:a3ca:cc98 with SMTP id 98e67ed59e1d1-2e8f1057e99mr40553660a91.7.1730827296258; Tue, 05 Nov 2024 09:21:36 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <3293802b-3785-4715-8a6b-0802afb6f908@app.fastmail.com> <620f4e82-aa69-4af0-bd22-b21203ab8745@app.fastmail.com> In-Reply-To: <620f4e82-aa69-4af0-bd22-b21203ab8745@app.fastmail.com> From: Warner Losh Date: Tue, 5 Nov 2024 10:21:25 -0700 Message-ID: Subject: Re: nvme device errors & zfs To: Dave Cottlehuber Cc: Tomasz CEDRO , freebsd-fs Content-Type: multipart/alternative; boundary="00000000000029567e06262da24c" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4XjZrK4CZJz49Sj X-Spamd-Bar: ---- --00000000000029567e06262da24c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Nov 5, 2024 at 4:20=E2=80=AFAM Dave Cottlehuber w= rote: > On Tue, 5 Nov 2024, at 11:10, Tomek CEDRO wrote: > > Magician software can upgrade firmware and perform other checks, works > > on Windoze macOS and Android: > > that will be difficult, I don't have an nvme capable thing for any > of those. > nvmecontrol updates firmware just fine, though. > > Another idea is maybe disk overheats and resets itself to cool down? > > that is a great point, the mainboard comes with inbuilt heatsinks, but > when I assembled it, the 2nd nvme slot heatsink looked a lot less > bulky than the other one, I remember distinctly wondering if it would > cope. If it happens again I'll see if I can get a temp measurement > at the time of failure. > > I would hope temperature throttling would not be quite so brutal, to > remove itself from the bus entirely, but its a reasonable explanation. > What's supposed to happen is that the temperature climbs slowly enough that there's a chance for it to kick in. It might be thermals, but I'd expect at least some indications that it's thermal. Heat sinks are cheap enough, if it's really thermal. log page 2 has the temperature. How often does the reset happen? A firmware upgrade might solve that problem if they are older. There might just be a bug that causes the firmware to 'trap' and it takes several seconds for the SoC / controller to reboot. Warner > A+ > Dave > =E2=80=94=E2=80=94=E2=80=94 > O for a muse of fire, that would ascend the brightest heaven of invention= ! > --00000000000029567e06262da24c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Tue, Nov 5, 2024 at 4:20=E2=80=AFA= M Dave Cottlehuber <dch@freebsd.org> wrote:
On= Tue, 5 Nov 2024, at 11:10, Tomek CEDRO wrote:
> Magician software can upgrade firmware and perform other checks, works=
> on Windoze macOS and Android:

that will be difficult, I don't have an nvme capable thing for any
of those.

> Another idea is maybe disk overheats and resets itself to cool down?
that is a great point, the mainboard comes with inbuilt heatsinks, but
when I assembled it, the 2nd nvme slot heatsink looked a lot less
bulky than the other one, I remember distinctly wondering if it would
cope. If it happens again I'll see if I can get a temp measurement
at the time of failure.

I would hope temperature throttling would not be quite so brutal, to
remove itself from the bus entirely, but its a reasonable explanation.
<= /blockquote>

What's supposed to happen is that the t= emperature climbs slowly enough
that there's a chance for it = to kick in. It might be thermals, but I'd expect
at least som= e indications that it's thermal. Heat sinks are cheap enough,
if it's really thermal. log page 2 has the temperature.

=C2=A0
A+
Dave
=E2=80=94=E2=80=94=E2=80=94
O for a muse of fire, that would ascend the brightest heaven of invention!<= br>
--00000000000029567e06262da24c-- From nobody Tue Nov 5 19:09:00 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XjdDJ71Fxz5cSnD for ; Tue, 05 Nov 2024 19:09:04 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fhigh-a8-smtp.messagingengine.com (fhigh-a8-smtp.messagingengine.com [103.168.172.159]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XjdDH6Xy7z4MVT for ; Tue, 5 Nov 2024 19:09:03 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=cwL18W1M; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="j ZPgOI2"; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 103.168.172.159 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from phl-compute-09.internal (phl-compute-09.phl.internal [10.202.2.49]) by mailfhigh.phl.internal (Postfix) with ESMTP id 08C8D1140237 for ; Tue, 5 Nov 2024 14:09:03 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-09.internal (MEProxy); Tue, 05 Nov 2024 14:09:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :message-id:mime-version:reply-to:subject:subject:to:to; s=fm3; t=1730833743; x=1730920143; bh=xbANKzbjXoa2PnFr1a+2AXRcHwvZWwAW VRTiUPvIg1o=; b=cwL18W1MeApDg1F+G9n57fzBd6BzcN+E5aMHVQOHAtHP1BGs jGbksSElIzFztn8YkTN7tgEbdy+WSnXQjHt7T1KmKYEdh7Jfn4GEpVBXNQlVU8of CwTQx3SGIIpQM8h+nrrCEoG9lCVgYHUBaRbwyYP6dgjH1XusuJU4Cc5Ojj+oNFxS WcSPqlFdXdhUkGSY7gtJ9zkZUczXkr+sL6x2sGtW7V8rqyBJ1bAgPIWvX2ZB4Us0 nNOnFXoMoPEtYbglnOu9RoOHL0o1deD1dJGEkA2k6qL0HaWmhm/deomjbgolzM4G BMaV2oi6j+VaOGJmtG95lnRok1ZIFdEaRvD0Pg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1730833743; x= 1730920143; bh=xbANKzbjXoa2PnFr1a+2AXRcHwvZWwAWVRTiUPvIg1o=; b=j ZPgOI2r8ekV7JW1pnuaf48h4jO6t4OuzDmOF2jccQBTUaywN9pLo6S/BjGjfXEI4 b6hdcNO22Pnx4zO55qQRof/LWmkWQIIuYka4lpyUhARTq9h32kEOagdFSqRVAOGR xiFWdY7gzNsHl8+LdfuSPohI6vwVaBbfFeYqHSVzQJ1xVumpbclH3EKhrs6gAnHD qTpa5oeuQCTj+qjqjLE2gK/esQSr/T7Xe0pWGRZIW/wYI0p92JKFVKL1T5CjRe4A ZEv3AEGnwWcHtWizQ1AclSa3d6qXPOqgtYlKhKhq7g3wY2Ua5mw3ZUZyGRabh3Fz +yCl1yebTL1VHYFt5MPrQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrtddtgdelvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukf ggtggusehttdertddttddvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhm qeenucggtffrrghtthgvrhhnpeevudffiedvffffgffhgeefjeefffdtieetheetkeefhf dvfefgtedtueehgeffueenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgr ihhlfhhrohhmpehvohhiugesfhdqmhdrfhhmpdhnsggprhgtphhtthhopedupdhmohguvg epshhmthhpohhuthdprhgtphhtthhopehfrhgvvggsshguqdhfshesfhhrvggvsghsugdr ohhrgh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Tue, 5 Nov 2024 14:09:02 -0500 (EST) Date: Tue, 5 Nov 2024 19:09:00 +0000 From: void To: freebsd-fs@freebsd.org Subject: problem with zfs raidz3 install on an 8-disk system, can't find loader Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline X-Spamd-Result: default: False [-3.68 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.88)[-0.884]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; MID_RHS_NOT_FQDN(0.50)[]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; RWL_MAILSPIKE_VERYGOOD(-0.20)[103.168.172.159:from]; R_SPF_ALLOW(-0.20)[+ip4:103.168.172.128/27]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[103.168.172.159:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FREEMAIL_FROM(0.00)[f-m.fm]; RCVD_COUNT_THREE(0.00)[3]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4XjdDH6Xy7z4MVT X-Spamd-Bar: --- Hi, the context here is * installing from usb2 stick * tried with FreeBSD-14.2-PRERELEASE-amd64-20241024-5ae76ff5138e-269296-memstick.img * also FreeBSD-15.0-CURRENT-amd64-20241031-5212b9500116-273335-memstick.img hardware is ProLiant DL380e Gen8 with 8* 4Tb seagate constellation attached to Smart Array P440 Controller in HBA (JBOD) mode. upon reboot: ### Attempting Boot From Hard Drive (C:) BIOS drive C: is disk0 zio_read error: 5 zio_read error: 5 zio_read error: 5 ZFS: i/o error - all block copies unavailable ZFS: failed to read pool zroot directory object Can't find /boot/zfsloader Can't find /boot/loader Can't find /boot/kernel/kernel FreeBSD/x86 boot Default: /boot/kernel/kernel boot: Can't find /boot/kernel/kernel FreeBSD/x86 boot Default: /boot/kernel/kernel boot: ### This is after the zfs install process runs without error. To be safe, I made sure that before running the install, i went into the shell prompt of the installer and ran gpart destroy -F da0 (repeated for da1-7) to make sure they were just raw disks presented. The installer when started saw all the disks and allows the creation of raidz3 with all of them. but upon removing the usb key and cold-reset, the loader can't be found. If it's booted to the (usb stick) installer again, then going into shell prompt, then running 'zpool import zroot' then 'zfs mount -a', the newly-created filesystem is seen. But it won't boot without the usb key. In an atempt to get round the problem, after zfs load, I made with zfs create: zfs create zroot/boot and copied all the stuff in /boot on the installer into zroot/boot (mounted at that time as /zroot/boot) and tried to boot that to no avail. Is there a way round this? -- From nobody Tue Nov 5 21:37:25 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XjhWd653vz5bdLr; Tue, 05 Nov 2024 21:37:33 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fhigh-a2-smtp.messagingengine.com (fhigh-a2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XjhWc5lHNz4cyW; Tue, 5 Nov 2024 21:37:32 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b="O oRvxbk"; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=HdNRkmkY; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 103.168.172.153 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from phl-compute-11.internal (phl-compute-11.phl.internal [10.202.2.51]) by mailfhigh.phl.internal (Postfix) with ESMTP id 291F91140280; Tue, 5 Nov 2024 16:37:28 -0500 (EST) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-11.internal (MEProxy); Tue, 05 Nov 2024 16:37:28 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to :reply-to:subject:subject:to:to; s=fm3; t=1730842648; x= 1730929048; bh=mHLiTTDzf8oiwPK20RBu+K5Qe3jcDkwxNEe12PD7JrM=; b=O oRvxbkLJSTyVIegUKUE7KCHWXjd3pSL9hf0ifAmqBR1hZsbRWhjH9Z74wbm0EcRl fk4vF6Y6oGIvm/xtLLxiGfxLPyFkme3wdZc2B85Abw+hB+JzkRDPNRZLqxlDqYDA KuGPnsXde9pwIXVQOX5a2jHqrK/h+shHUVqwqPg1Kt5g5J7e5DhmieCKaTzuLw2q RCB6+Y/a09F+Zl3SNU/Z1ySgdw4j/AyY0YbhKizxKr6/eON4BQQaPEmL3Ida+uIg PcRBhhORrkm1vLmBODK4F4gMzaFc9Fu35l/ByD3thv3nEZXFy+dFX8VDw3boey0t TF8VEWkP3j72NxxQjO8NQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:reply-to:subject :subject:to:to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1730842648; x=1730929048; bh=mHLiTTDzf8oiwPK20RBu+K5Qe3jc DkwxNEe12PD7JrM=; b=HdNRkmkYmXeuTLHdsiNRFwbsBHzLWYpdamM9qqLFSXKl KHPDOk+Wsy94tLfZCPYZHrDpsCo92FtGbm/W/ddWVG8J6/supEVZFo4+YNyiUqCV kxrrw5E0cEKJFhDtkL9e2gelVQoCfQQaI16eOoAOsBPanaNoAyBprRV/pHAcjJYa GiePtvz9TD16BMz4bvUV45Elasoc4lmvDzkOksObz0z9hUKxwYAzWyOO0Sf7gfvM ouT2dWuUiKr4GDhdi3sxt+Uauk+EFBuPgUAWSKZxjfTPUb6I7TYYId/E0unLGyzm JNiMVEk2qGd77UhNqXKSp8ZLYRCL9o9QqzOz6aNZDw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrtddtgdduvddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhrfhggtggujgesthdtredttddtvdenucfhrhhomhepvhhoihguuceovhhoihgusehf qdhmrdhfmheqnecuggftrfgrthhtvghrnhepledtiedugfeihfekhfffhfetueehfefhle evhfevvefftddvhfekveeuvdefledunecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomhepvhhoihgusehfqdhmrdhfmhdpnhgspghrtghpthhtohepvd dpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepfhhrvggvsghsugdqfhhssehfrhgv vggsshgurdhorhhgpdhrtghpthhtohepshihshhinhhsthgrlhhlsehfrhgvvggsshgurd horhhg X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 5 Nov 2024 16:37:27 -0500 (EST) Date: Tue, 5 Nov 2024 21:37:25 +0000 From: void To: freebsd-fs@freebsd.org Cc: sysinstall@freebsd.org Subject: Re: problem with zfs raidz3 install on an 8-disk system, can't find loader Message-ID: Reply-To: sysinstall@freebsd.org Mail-Followup-To: freebsd-fs@freebsd.org, sysinstall@freebsd.org References: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-3.60 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_SPF_ALLOW(-0.20)[+ip4:103.168.172.128/27]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[103.168.172.153:from]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; FREEMAIL_FROM(0.00)[f-m.fm]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org,sysinstall@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim]; HAS_REPLYTO(0.00)[sysinstall@freebsd.org] X-Rspamd-Queue-Id: 4XjhWc5lHNz4cyW X-Spamd-Bar: --- On Tue, Nov 05, 2024 at 07:09:00PM +0000, void wrote: >Hi, > >the context here is * installing from usb2 stick >* tried with FreeBSD-14.2-PRERELEASE-amd64-20241024-5ae76ff5138e-269296-memstick.img >* also FreeBSD-15.0-CURRENT-amd64-20241031-5212b9500116-273335-memstick.img > >hardware is ProLiant DL380e Gen8 with 8* 4Tb seagate constellation >attached to Smart Array P440 Controller in HBA (JBOD) mode. > >upon reboot: >### >Attempting Boot From Hard Drive (C:) BIOS drive C: is disk0 >zio_read error: 5 zio_read error: 5 >zio_read error: 5 ZFS: i/o error - all block >copies unavailable >ZFS: failed to read pool zroot directory object >Can't find /boot/zfsloader >Can't find /boot/loader >Can't find /boot/kernel/kernel >FreeBSD/x86 boot >Default: /boot/kernel/kernel >boot: >Can't find /boot/kernel/kernel >FreeBSD/x86 boot >Default: /boot/kernel/kernel >boot: > >### > >This is after the zfs install process runs without error. >To be safe, I made sure that before running the install, i went into >the shell prompt of the installer and ran gpart destroy -F da0 (repeated for da1-7) >to make sure they were just raw disks presented. The installer when >started saw all the disks and allows the creation of raidz3 with all >of them. but upon removing the usb key and cold-reset, the loader >can't be found. > >If it's booted to the (usb stick) installer again, then going into shell prompt, >then running 'zpool import zroot' then 'zfs mount -a', the newly-created >filesystem is seen. > >But it won't boot without the usb key. > >In an atempt to get round the problem, after zfs load, I made with zfs create: > >zfs create zroot/boot > >and copied all the stuff in /boot on the installer into zroot/boot >(mounted at that time as /zroot/boot) and tried to boot that to no >avail. > >Is there a way round this? >-- > Seems the way round this is to firstly install using ufs rather than zfs, which results in a bootable system. Planning to make the other disks zfs. So, no root-on-zfs at this time. -- From nobody Thu Nov 7 12:58:32 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XkhwK4hMDz5bqnw; Thu, 07 Nov 2024 12:58:57 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from fhigh-b6-smtp.messagingengine.com (fhigh-b6-smtp.messagingengine.com [202.12.124.157]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XkhwH6wXrz4j3q; Thu, 7 Nov 2024 12:58:55 +0000 (UTC) (envelope-from dch@skunkwerks.at) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=skunkwerks.at header.s=fm1 header.b=n7tRC7Ff; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="b AqgjT4"; spf=pass (mx1.freebsd.org: domain of dch@skunkwerks.at designates 202.12.124.157 as permitted sender) smtp.mailfrom=dch@skunkwerks.at; dmarc=pass (policy=none) header.from=skunkwerks.at Received: from phl-compute-02.internal (phl-compute-02.phl.internal [10.202.2.42]) by mailfhigh.stl.internal (Postfix) with ESMTP id BFFC325400E8; Thu, 7 Nov 2024 07:58:53 -0500 (EST) Received: from phl-imap-02 ([10.202.2.81]) by phl-compute-02.internal (MEProxy); Thu, 07 Nov 2024 07:58:53 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skunkwerks.at; h=cc:cc:content-transfer-encoding:content-type:content-type :date:date:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:subject:subject:to:to; s=fm1; t=1730984333; x=1731070733; bh=JxBUqnp78XaIEtQ9WLxACYXRHBY+LHkP v8wtel//nvY=; b=n7tRC7FfHbcy0NLlgrMocZ9lOsmOCfpWImh/Z0YIK8MgRO2T Xv3TxHm77vWkWSYKKHqxyA8eIB/MhST4PGXhF7nA+wSyX/qfjtQTXEIrESW4dxD+ FMu158RNTTvs2Xj2TYSrr5PEFHK6I+xlLKuo3JKVLuffYMe+uAhrOMUs5cI7BVTg nHJHd95cLN2gPBs71bKWKUhuj5dYw8nOzQ/2775iJBb7qF0/+6dlbQabeKmCtYGT nRmScEPhxpEFra1IgqF3Cr1rQM5h5kz3RBA8RFPEUlylaPe1InMnB6Xs0DYj4oBx 8ABtUTjuK5zA/WJ4hOQYIBSUiC0pzT3Fcc5uSw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1730984333; x= 1731070733; bh=JxBUqnp78XaIEtQ9WLxACYXRHBY+LHkPv8wtel//nvY=; b=b AqgjT4DTYLs7mNhy3vYU1iBEXODVPAGIZ/qHLZxKSV1R2eiHOwHxoSl7IR9fXTdg OqznrE7VUNSkef2jtFftVvmgeS+yi9LCamN88HXhKO3ZxAAWphxZ+NCXN/ozgm+9 kSnr0yHjG9CdUroqYjiW+Irdozf0s4n/5MABws8MDa0t5jdVGl7+Akjtu+Cq1pza OMVXCfOnzZVgz5OOKhj8U2FxF0y8bbr0IYMn9EXs+wSFAQSRzxd4f0WwKVs0yw1c xgdM9Rh+xRwzGqqm9WgB724UMefhHheKsAt7HPYwfAMQeoyymbAfgEwgvmFiyXKi XjmgDcNZpMRUHigyqIJ3Q== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrtdeggdegiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecunecujfgurhepofggfffhvf evkfgjfhfutgfgsehtjeertdertddtnecuhfhrohhmpedfffgrvhgvucevohhtthhlvghh uhgsvghrfdcuoegutghhsehskhhunhhkfigvrhhkshdrrghtqeenucggtffrrghtthgvrh hnpeeifffhudejleeikeeuhefgieegkeelleejieeggfduueehtefgtdegffeiveeuvden ucffohhmrghinhepfhhrvggvsghsugdrohhrghenucevlhhushhtvghrufhiiigvpedtne curfgrrhgrmhepmhgrihhlfhhrohhmpegutghhsehskhhunhhkfigvrhhkshdrrghtpdhn sggprhgtphhtthhopeefpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehvohhiug esfhdqmhdrfhhmpdhrtghpthhtohepfhhrvggvsghsugdqfhhssehfrhgvvggsshgurdho rhhgpdhrtghpthhtohepshihshhinhhsthgrlhhlsehfrhgvvggsshgurdhorhhg X-ME-Proxy: Feedback-ID: ic0e84090:Fastmail Received: by mailuser.phl.internal (Postfix, from userid 501) id 2871FB00068; Thu, 7 Nov 2024 07:58:53 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Date: Thu, 07 Nov 2024 12:58:32 +0000 From: "Dave Cottlehuber" To: void@f-m.fm Cc: sysinstall@freebsd.org, freebsd-fs Message-Id: <62b73206-0f14-423a-978e-7d553a9c263c@app.fastmail.com> In-Reply-To: References: Subject: Re: problem with zfs raidz3 install on an 8-disk system, can't find loader Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-4.09 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[skunkwerks.at,none]; R_SPF_ALLOW(-0.20)[+ip4:202.12.124.128/27]; R_DKIM_ALLOW(-0.20)[skunkwerks.at:s=fm1,messagingengine.com:s=fm3]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[202.12.124.157:from]; XM_UA_NO_VERSION(0.01)[]; FREEFALL_USER(0.00)[dch]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[202.12.124.157:from]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org,sysinstall@freebsd.org]; FREEMAIL_TO(0.00)[f-m.fm]; RCPT_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[skunkwerks.at:+,messagingengine.com:+] X-Rspamd-Queue-Id: 4XkhwH6wXrz4j3q X-Spamd-Bar: ---- On Tue, 5 Nov 2024, at 21:37, void wrote: > On Tue, Nov 05, 2024 at 07:09:00PM +0000, void wrote: >>Hi, >> >>the context here is * installing from usb2 stick >>* tried with FreeBSD-14.2-PRERELEASE-amd64-20241024-5ae76ff5138e-269296-memstick.img >>* also FreeBSD-15.0-CURRENT-amd64-20241031-5212b9500116-273335-memstick.img >> >>hardware is ProLiant DL380e Gen8 with 8* 4Tb seagate constellation >>attached to Smart Array P440 Controller in HBA (JBOD) mode. I don't see any general reason why this wouldn't work, people have used it in the past - https://lists.freebsd.org/pipermail/freebsd-stable/2017-September/087746.html >From inside the installer, it would be worth sharing with the list: - dmesg - sysctl machdep.bootmethod - relevant disk controller bits of `pciconf -lv` and comparing what you see when booting from h/w (not memstick.img) to see if it comes up in UEFI or BIOS mode. While dealing with these issues I would start out first with a single drive and see if FreeBSD will boot to "plain zfs" first. I'm not familiar with raidz3, but that's where I'd start off. A+ Dave From nobody Thu Nov 7 13:23:47 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XkjTR1d8Lz5bs97; Thu, 07 Nov 2024 13:24:11 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from fout-b3-smtp.messagingengine.com (fout-b3-smtp.messagingengine.com [202.12.124.146]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XkjTP5b67z4kb2; Thu, 7 Nov 2024 13:24:09 +0000 (UTC) (envelope-from dch@skunkwerks.at) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=skunkwerks.at header.s=fm1 header.b=psrmfv+6; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="R Zjoq44"; spf=pass (mx1.freebsd.org: domain of dch@skunkwerks.at designates 202.12.124.146 as permitted sender) smtp.mailfrom=dch@skunkwerks.at; dmarc=pass (policy=none) header.from=skunkwerks.at Received: from phl-compute-02.internal (phl-compute-02.phl.internal [10.202.2.42]) by mailfout.stl.internal (Postfix) with ESMTP id 23E98114010F; Thu, 7 Nov 2024 08:24:09 -0500 (EST) Received: from phl-imap-02 ([10.202.2.81]) by phl-compute-02.internal (MEProxy); Thu, 07 Nov 2024 08:24:09 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skunkwerks.at; h=cc:cc:content-transfer-encoding:content-type:content-type :date:date:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:subject:subject:to:to; s=fm1; t=1730985848; x=1731072248; bh=48KFAiqIPbglIcNuNcPAQz3343Nz1udT xrnw4aVbSQA=; b=psrmfv+6qSdZeKcknRYf2bxbc6h153kyhAesmz+pUfnu6qr3 9b2s4Jad3wUxrDMgpwYbq7BCs4dgFd5v28+e+bz4NUvrgIDCCQEZB2F7EQ3oFJlc KVXD1TilgJpqfC+EfUZq+1ce2X2lsKSQe6aTybCpGsCYYQNwclyO47+o9I5BWntj MzLrc0gZiYM78Ye8aKtDLjlp/ETT/6b7myTZ8UCFK7nUMNNznNkOiy5snkbBJCDV 1rIAb/TOWYUYRdqpYJKp2ef1/J9RZO0erIOpTu/7tlUAxTDgJxu/yCi5sZgpIirt r32DW+5Kbk0gdIsLOyHnTPb8p9WGcxQNnvR5sg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1730985848; x= 1731072248; bh=48KFAiqIPbglIcNuNcPAQz3343Nz1udTxrnw4aVbSQA=; b=R Zjoq44r2LI6TpX516h98r/So2YNBj6UR2EjFBXr4Pp1YQW3nnwfVWb46gL9j+yVN 0X/4n9ZTpxSSwznx2WksEvbIradAd9d3HO9aKrkTfAzi0IlYjVPYrJbFJLGI8eKT 1d8M/rJJHAX8vFJkd8FOnQ1uWlBooSxpde0UQnD2bo33g521PqQWX6UpqzRBw6bb GiNa/hXkJ4eaAucV7lI6LAFuSPBuyfU7T23HA/ixOCBYLtvZcMckaWnhDaeVaNKT Ffi00dAlEehUAf2Y5FYDvHTQo4PexbiO4Cfr4V0pJxQzhwjbgLmlR861Tc2sD6E3 oIarZAKc0cm3aBm7/b7rg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrtdeggdehudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecunecujfgurhepofggfffhvf evkfgjfhfutgfgsehtjeertdertddtnecuhfhrohhmpedfffgrvhgvucevohhtthhlvghh uhgsvghrfdcuoegutghhsehskhhunhhkfigvrhhkshdrrghtqeenucggtffrrghtthgvrh hnpeeifffhudejleeikeeuhefgieegkeelleejieeggfduueehtefgtdegffeiveeuvden ucffohhmrghinhepfhhrvggvsghsugdrohhrghenucevlhhushhtvghrufhiiigvpedtne curfgrrhgrmhepmhgrihhlfhhrohhmpegutghhsehskhhunhhkfigvrhhkshdrrghtpdhn sggprhgtphhtthhopeefpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehvohhiug esfhdqmhdrfhhmpdhrtghpthhtohepfhhrvggvsghsugdqfhhssehfrhgvvggsshgurdho rhhgpdhrtghpthhtohepshihshhinhhsthgrlhhlsehfrhgvvggsshgurdhorhhg X-ME-Proxy: Feedback-ID: ic0e84090:Fastmail Received: by mailuser.phl.internal (Postfix, from userid 501) id AFC9AB00069; Thu, 7 Nov 2024 08:24:08 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Date: Thu, 07 Nov 2024 13:23:47 +0000 From: "Dave Cottlehuber" To: void Cc: sysinstall@freebsd.org, freebsd-fs Message-Id: <63b1a504-7aa2-4b4c-9c0e-f5f7da2b0c4b@app.fastmail.com> In-Reply-To: <62b73206-0f14-423a-978e-7d553a9c263c@app.fastmail.com> References: <62b73206-0f14-423a-978e-7d553a9c263c@app.fastmail.com> Subject: Re: problem with zfs raidz3 install on an 8-disk system, can't find loader Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-4.09 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[skunkwerks.at,none]; R_DKIM_ALLOW(-0.20)[skunkwerks.at:s=fm1,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:202.12.124.128/27:c]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[202.12.124.146:from]; XM_UA_NO_VERSION(0.01)[]; FREEFALL_USER(0.00)[dch]; ASN(0.00)[asn:151847, ipnet:202.12.124.0/24, country:AU]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_TO(0.00)[f-m.fm]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_THREE(0.00)[3]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org,sysinstall@freebsd.org]; DKIM_TRACE(0.00)[skunkwerks.at:+,messagingengine.com:+] X-Rspamd-Queue-Id: 4XkjTP5b67z4kb2 X-Spamd-Bar: ---- On Thu, 7 Nov 2024, at 12:58, Dave Cottlehuber wrote: >>>hardware is ProLiant DL380e Gen8 with 8* 4Tb seagate constellation >>>attached to Smart Array P440 Controller in HBA (JBOD) mode. > > I don't see any general reason why this wouldn't work, people > have used it in the past - > https://lists.freebsd.org/pipermail/freebsd-stable/2017-September/087746.html Assuming this is detected as a ciss device, have you got `ciss_load="YES"` in your /boot/loader.conf per http://man.freebsd.org/ciss ? I'm assuming this is a UEFI partition layout, also. There is also a hw.ciss.verbose=2 tunable, which may provide more info if it gets that far. A+ Dave From nobody Thu Nov 7 14:44:56 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XklGd1t0pz5bx4N for ; Thu, 07 Nov 2024 14:44:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XklGd0rJ6z4sJm for ; Thu, 7 Nov 2024 14:44:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730990697; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AEM3RAeL6PoJagrKdM7fST9yFOz8ypXijAtymLviibk=; b=sxo3S3KgddBOOtgGx27y4SGo+bnbdjj4gq4HfwSU81ff8cmtYVLd7URK1M0L8I7QHhBAw2 FXTif7EX8vA8aqhEUxmy0Qr4LGPcQMB9Pyi/sa/JFWzqmZCSnjF7YoIHmXPvfqar+EpTOT UH4UsTBg/SQTuWzTfE//fm8YjLZrUbrNVXqyspbxB4BqdgRCsL1FoPGgN2LUQD9qsfRetr VOqc4JISA7mV5uTVFAr/qrK2lup5E3EnZi3yPWzpISceyZ4VdZZHGkBvQneMNvXOn4skDw dfmMUlNGwj3mxsqkQN6ke9Dm0Hw8nZnfW900fMh30jqbYDD3Ma4LKuc3ML6YRw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1730990697; a=rsa-sha256; cv=none; b=Sps91AuhID1xBc6TYBDVcJdMTACl8Yi5wQ+sc00RaDQKb/IyWfnVbvLnRqrY7T6C83nqdh fN19OVHOZCyQw2uSgSt29Wd0ITPmnWGlKP5AmdvssmCHbJK2O98v/ooh1tW1IoGN2I5yM5 XL2pIVCkZihpBKJcXeX/4KfNrE/CDVxypbrrP2FWu26CRsIDPlcqtmoS7eVFlvrZQo9F2V gMFdfXNWJWvqoOFl6Vv9NAlCci+lY7FEZoUirZnH4pJ80gH5rDjLIT0rUV8V+ZrMJvVQPp IeJMSP4pTU+rNb5sl6+yFsGP2oiWwR1Id17Iahq95A+h3arW3bD9b0BCyunw5w== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XklGd0KvnzqTj for ; Thu, 7 Nov 2024 14:44:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A7Eiu7Z056573 for ; Thu, 7 Nov 2024 14:44:56 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A7EiuDK056571 for fs@FreeBSD.org; Thu, 7 Nov 2024 14:44:56 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282449] UFS deadlock with install -S during freebsd-update Date: Thu, 07 Nov 2024 14:44:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: Unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282449 --- Comment #13 from Mark Johnston --- > Could you please try setting kern.maxphys=3D131072 in /boot/loader.conf, = then reboot and try to reproduce the problem? Have you had a chance to try this? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Nov 7 18:57:39 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XkrtJ012Fz5cPGP; Thu, 07 Nov 2024 18:57:44 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fhigh-a6-smtp.messagingengine.com (fhigh-a6-smtp.messagingengine.com [103.168.172.157]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XkrtG4FsJz4ctZ; Thu, 7 Nov 2024 18:57:42 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=CQfwYDW7; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=ANv+h9fT; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 103.168.172.157 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from phl-compute-04.internal (phl-compute-04.phl.internal [10.202.2.44]) by mailfhigh.phl.internal (Postfix) with ESMTP id 277D711401DA; Thu, 7 Nov 2024 13:57:42 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-04.internal (MEProxy); Thu, 07 Nov 2024 13:57:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1731005862; x=1731092262; bh=1gHSNj+3WB 2dXPGccRPCtUi8aWohcvBDtW6p+EKxWmU=; b=CQfwYDW7mi7sduHd7dFo/aL7eK 5G32sFdRh8dYTK7dPTqObnnkb2E6L6lIKClQ0zDX2xPPnp0+MiGMmichc2tjfwEe 61IwwJsmQ/luNcdW6gfJknf3q4n7Nar0XaD3DU2pWJkqe8Cnyf05pgPk7PpRV34P X1tmFXN5ilb7i60inwok/iDAJENtfdKvtkzCriOToxa7E+VBm2oHRJroF/Az8P8g jwTquDUZ8LWkKtKiJvGwHV2jXYoPGJzz93MVicODu1y1hez7Yus0w1GvRErx3UUm ReJ0zdpWXhW4kS8V4o5V5UQZn0xcz17fYHEdUe14Bb7TO74EzCdHRkc5FdQA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1731005862; x=1731092262; bh=1gHSNj+3WB2dXPGccRPCtUi8aWohcvBDtW6 p+EKxWmU=; b=ANv+h9fTtDP5/Zw+NkYywieoqxZHyKxezYHHSuXYh2MCC+6RRrF sGan234fnfJlyDXO09IT5746oHfQTuZ+6BSKiOVZmWKgqr2BtEebJ0k1XBHJN2/c 94+JQQfKMFCkDp7OfUAVcOgUDJv+pu6LY8/PUflKJ+AcTNzo0Ls7WPly9YU9XO5x kZSAY+D64FtvHuobZbUU5MULp1Vu/Qim6zq75t/AixThH3N5pjRVj/km3ty0t/9a SV87H2pDaf+8LTSz2x+ioom8c43meC894peHTZiG/puLmDd/AysUuweUxzpof/hO pdLz1zP5Xf7aG6Lcp3WBXBCeHJ+dbDkOt5Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrtdeggdduudekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdq mhdrfhhmqeenucggtffrrghtthgvrhhnpeduvdeugeduudefteehheffvdegteelkeekvd evffdukeeugeeuteffteefkeelhfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehvohhiugesfhdqmhdrfhhmpdhnsggprhgtphhtthhopedvpd hmohguvgepshhmthhpohhuthdprhgtphhtthhopehfrhgvvggsshguqdhfshesfhhrvggv sghsugdrohhrghdprhgtphhtthhopehshihsihhnshhtrghllhesfhhrvggvsghsugdroh hrgh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 7 Nov 2024 13:57:41 -0500 (EST) Date: Thu, 7 Nov 2024 18:57:39 +0000 From: void To: freebsd-fs@freebsd.org Cc: sysinstall@freebsd.org Subject: Re: problem with zfs raidz3 install on an 8-disk system, can't find loader Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org, sysinstall@freebsd.org References: <62b73206-0f14-423a-978e-7d553a9c263c@app.fastmail.com> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <62b73206-0f14-423a-978e-7d553a9c263c@app.fastmail.com> X-Spamd-Result: default: False [-3.57 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.975]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:103.168.172.128/27]; RCVD_IN_DNSWL_LOW(-0.10)[103.168.172.157:from]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org,sysinstall@freebsd.org]; MISSING_XM_UA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4XkrtG4FsJz4ctZ X-Spamd-Bar: --- On Thu, Nov 07, 2024 at 12:58:32PM +0000, Dave Cottlehuber wrote: >>From inside the installer, it would be worth sharing with the list: > >- dmesg >- sysctl machdep.bootmethod >- relevant disk controller bits of `pciconf -lv` > >and comparing what you see when booting from h/w (not memstick.img) >to see if it comes up in UEFI or BIOS mode. > >While dealing with these issues I would start out first with a >single drive and see if FreeBSD will boot to "plain zfs" first. > >I'm not familiar with raidz3, but that's where I'd start off. alas, not possible now as it's for production. There's another, similar, machine I can test with though. I'll try that and report back. This other, similar machine has had freebsd since 12-current. it may have been installed as ufs and subsequently zfs added for data. Need to check. The particular machine (the one with the issue reported initially) was running -current but it was using the hpa smartarray. Nothing special was needed for the install back then. The smartarray presented 2x devices as it was configured for 2* 4 hds in raid mirror. The zfs was then installed as stripe, resulting in about (and this is from memory) 7Tb space. I wantd HBA (read: JBOD) mode for this system so I had to destroy the hardware array and put the card in HBA mode. When booted, the installer saw all the individual disks and apparently made a raidz3 as configured via the installer. *Currently* the installer presents all the disks, no special driver needed. What makes me think this is a bug is that everything can be added to the zpool but when the system boots, it can't boot. To get around the installer issue, I installed onto one of the disks as ufs (da0). This proceeded normally. Now with only 7 disks for zfs I made a raidz2: % zpool status pool: data state: ONLINE config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 This appears to function normally. The issue *appears* to be that the installer doesn't install the guts of what is required to boot the array, despite configuring and apparently installing everything else. This is similar to a long-standing issue affecting installing zfs on arm64. -- From nobody Thu Nov 7 19:53:38 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xkt6q0zdvz5cVV1 for ; Thu, 07 Nov 2024 19:53:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xkt6p6wTZz4lXJ for ; Thu, 7 Nov 2024 19:53:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731009219; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xrc20W8mn9Ot51kmIjoo8UFz61t+pLla3eiVtBqz+vM=; b=ruKYYrsiiOkABbioiRLUlsbVE09V+U5n4AIdoF9TfwovOMM+8djcBwUQN7avL80TMqKCcD 80KDN1OtljwG9F/ZfKIwh7P1x4NrRf7oN+XbGYr1Amlo+5PHjC7RkZuekRaqDQm4DQF6V/ /nQCmYRPOR9v0mO0c2r+gOhNg7MH23ThxUh2G0Hq0Nox+UzjA3B55sMtDHbfGuTOSTctOJ Rumsjl0FWpP5jg0Opz54chE+//ztTeHI3hZ3BOKneNLraEY2f50J9dssEDo7p2iHqamGH/ i+n/u2FR+wYYXTwpXEAJNnJlkTvyadX0kTYYBXnaNfC3v6Y6fMsBUzo116x+eQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731009219; a=rsa-sha256; cv=none; b=eTzqnpnQRVNRKwzn2A52V6U34FTFG/qViOSSUWfcZq1oeT3JrM8G8uqYvdPnZyb5OVaLxe TP9EczrDw6cwvLe/3jrmPxjSqfPI7UwD8fan9NzZ6UbQZnD3ONrDB1EIW3FGr4SKrhIpyV Hl+MgSd9XrX5VgmOLqmydPRODvBR5/mXs11QC8uiFA5EnOfNr4bB0z+dJoo1K+QKcJwKRi a7pDlULMs3WDR4IeM0LdbX8gX1BSUcuXHMoOsR5Qo+2Q3Y/IjZby/wJ/NfLaoTPqxrBq5X b+quVVjWdyz19DrNVy9ZKYCGbjH0u/Lstfby9OKL+ALYACYckMl2hKfQIcyW/g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Xkt6p6X4CzynY for ; Thu, 7 Nov 2024 19:53:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A7JrcCd050424 for ; Thu, 7 Nov 2024 19:53:38 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A7Jrc5H050423 for fs@FreeBSD.org; Thu, 7 Nov 2024 19:53:38 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282449] UFS deadlock with install -S during freebsd-update Date: Thu, 07 Nov 2024 19:53:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: Unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: Grau_Smue@md5hashing.net X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282449 --- Comment #14 from vfs-locking --- I've set it on the relevant systems and they still work. :) I don't have a reliable reproducer, but my test VM completed freebsd-update= . I think your theory of cause would have to stand as a basis for calling kern.maxphys=3D131072 as a fix. The question of which parameter to tweak as a fix should get some discussio= n; 13 and 14 should probably go with reduced performance if it doesn't break. = 15 might go the opposite direction since a higher memory floor is not a surpri= sing change. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Nov 7 20:31:18 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XktyJ5WBFz5cYfg for ; Thu, 07 Nov 2024 20:31:20 +0000 (UTC) (envelope-from lexi@hemlock.eden.le-fay.org) Received: from fuchsia.eden.le-Fay.ORG (fuchsia.eden.le-fay.org [81.187.47.195]) by mx1.freebsd.org (Postfix) with ESMTP id 4XktyH61g2z4tVg for ; Thu, 7 Nov 2024 20:31:19 +0000 (UTC) (envelope-from lexi@hemlock.eden.le-fay.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of lexi@hemlock.eden.le-fay.org has no SPF policy when checking 81.187.47.195) smtp.mailfrom=lexi@hemlock.eden.le-fay.org; dmarc=none Received: from hemlock.eden.le-fay.org (hemlock.eden.le-fay.org [IPv6:2001:8b0:aab5:4::10]) by fuchsia.eden.le-Fay.ORG (Postfix) with ESMTP id EE1678687 for ; Thu, 07 Nov 2024 20:31:18 +0000 (UTC) Received: by hemlock.eden.le-fay.org (Postfix, from userid 10006) id B4A5D53C; Thu, 07 Nov 2024 20:31:18 +0000 (GMT) From: Lexi Winter To: fs@freebsd.org Subject: p9fs (bhyve) success report List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <60564.1731011478.1@hemlock.eden.le-fay.org> Date: Thu, 07 Nov 2024 20:31:18 +0000 Message-Id: <20241107203118.B4A5D53C@hemlock.eden.le-fay.org> X-Spamd-Result: default: False [-0.03 / 15.00]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.90)[-0.905]; NEURAL_SPAM_MEDIUM(0.87)[0.871]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:20712, ipnet:81.187.0.0/16, country:GB]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[fs@freebsd.org]; DMARC_NA(0.00)[le-fay.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[fs@freebsd.org]; R_SPF_NA(0.00)[no SPF record] X-Rspamd-Queue-Id: 4XktyH61g2z4tVg X-Spamd-Bar: / hello, as this has come up before here (and didn't make it into the quarterly status report), i just wanted to mention that i'm currently using p9fs(5) in -current to export a filesystem from the host to a bhyve VM and, after some fixes made in October, it's been rock solid for me. this is a *lot* easier than using NFS/SMB to export filesystems to a VM, and brings FreeBSD up to par with Linux as a VM guest. so, thank you to everyone who was involved in implementing and merging this. for reference, the configuration on the bhyve host: -s 5:0,virtio-9p,torrents=/data/public/Torrents,rw and in the VM's /etc/fstab: torrents /data/public/Torrents p9fs rw 0 0 i do think the manpage should be in section 4 though, so i submitted a PR for that (https://github.com/freebsd/freebsd-src/pull/1520). From nobody Thu Nov 7 20:40:09 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xkv8l1Kmxz5cZXQ for ; Thu, 07 Nov 2024 20:40:23 +0000 (UTC) (envelope-from bakul@iitbombay.org) Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xkv8k4SVjz3xNX for ; Thu, 7 Nov 2024 20:40:22 +0000 (UTC) (envelope-from bakul@iitbombay.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-20c8719fb63so1746005ad.1 for ; Thu, 07 Nov 2024 12:40:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iitbombay.org; s=google; t=1731012021; x=1731616821; darn=freebsd.org; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=CbgItrwkrAZa5qcFOS4/3fhMV7ly3d2sB5vm01zg7Cg=; b=H4YNd5ENafIZ0rdUS0JEVz2EYQSC065cPMep2PB4GX2pZw7RXrKqVca90z5AyE833W lRavxfaOgqfFqzyNGqhwYMwKYS7QHol4fhUba83WrCruxFkv4Ydcp/7yNrttIREXCZdx 1RAodGPn2GoNDLUXfhM2o5QLWVJS5DjlZquyQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731012021; x=1731616821; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CbgItrwkrAZa5qcFOS4/3fhMV7ly3d2sB5vm01zg7Cg=; b=nZyjvRGhe6+llRGpBsRwt4nwjvkQHUjlQpABTMviTwJNJkovwBmD7Z92UnRGVSFKnq OpqBdm8A6kz50efI3Q2Ohu/eAhlEojQf4vKMdPhFlIuvP0nBwDDg3eqM+/xgvDw9LCL7 olDAM2gUqes4iZstFVUTpaGFEnnJKasOwCXU2z8cK5QihBtoOo7qtRJ7DO5Z91LIEl0p esWcgIgAu1qZK9tzIBTolRKF4mmYH0wa0aitozfll55dLFEbu6S38Fc1Qq2nw2WWc7FD 3CSGgvZZ3Ae6PAUWWmeFX7nOJmu4iTnP6FhjCeSPZlghK/dD6+oGhL/w/JuERjNoD4Vm FANw== X-Gm-Message-State: AOJu0YzYjAE9G1xWBrsM1NRExFdqjxtZ4rLFjbI7YQF7BEtP48+dn9vB Fq6I4UbjuRjRTAw2f9pJ0ruVoqpR7JZ5iXr2J5AQWHz9O6xAGzXmuoC227G4IfdFPdPatJngHoI = X-Google-Smtp-Source: AGHT+IF80C8DzQGr+P/PFbGNHMeZvdq6Q/BJhZiKS5bmz+H2T23ERcGWEMmstjcgvTeBNjaBxknvWw== X-Received: by 2002:a17:902:e750:b0:20b:b93f:2fec with SMTP id d9443c01a7336-211835420ddmr1307725ad.8.1731012021306; Thu, 07 Nov 2024 12:40:21 -0800 (PST) Received: from smtpclient.apple (107-215-223-229.lightspeed.sntcca.sbcglobal.net. [107.215.223.229]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21177e6890esm16841725ad.223.2024.11.07.12.40.20 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Nov 2024 12:40:20 -0800 (PST) Content-Type: text/plain; charset=utf-8 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.200.121\)) Subject: Re: p9fs (bhyve) success report From: Bakul Shah In-Reply-To: <20241107203118.B4A5D53C@hemlock.eden.le-fay.org> Date: Thu, 7 Nov 2024 12:40:09 -0800 Cc: fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20241107203118.B4A5D53C@hemlock.eden.le-fay.org> To: Lexi Winter X-Mailer: Apple Mail (2.3826.200.121) X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Xkv8k4SVjz3xNX X-Spamd-Bar: ---- On Nov 7, 2024, at 12:31=E2=80=AFPM, Lexi Winter = wrote: >=20 > hello, >=20 > as this has come up before here (and didn't make it into the quarterly = status > report), i just wanted to mention that i'm currently using p9fs(5) in = -current > to export a filesystem from the host to a bhyve VM and, after some = fixes made > in October, it's been rock solid for me. >=20 > this is a *lot* easier than using NFS/SMB to export filesystems to a = VM, and > brings FreeBSD up to par with Linux as a VM guest. How is the performance compared to nfs?= From nobody Thu Nov 7 20:46:46 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XkvJN3vH2z5cb6D for ; Thu, 07 Nov 2024 20:47:00 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [66.165.241.226]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XkvJM4mPRz40gT for ; Thu, 7 Nov 2024 20:46:59 +0000 (UTC) (envelope-from pete@nomadlogic.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=nomadlogic.org header.s=04242021 header.b=I7zsWfDK; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 66.165.241.226 as permitted sender) smtp.mailfrom=pete@nomadlogic.org; dmarc=pass (policy=quarantine) header.from=nomadlogic.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nomadlogic.org; s=04242021; t=1731012371; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9KTz51tfUfI6cMzh7XnXTEkKLzTArHYLv2OtyiNu87g=; b=I7zsWfDKadtsFyVMU5/yjbVxhLf3IbvcSzDUg1igi8kHw4Q2RQjkD6iv1vb6EnWoPpzfuP zWp+bHxlP8XuuqiR/EGoitlVkgTptrG+r48b0hM+cn6C+6yX/mO5Nuufq4PKh2gJbZgX3Z UhKl+10ro5EHShVwtSIgz9IQzJcdsWM= Received: from [192.168.1.160] ( [47.154.20.141]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id fa3c0e0f (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 7 Nov 2024 20:46:10 +0000 (UTC) Message-ID: Date: Thu, 7 Nov 2024 12:46:46 -0800 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: p9fs (bhyve) success report To: freebsd-fs@freebsd.org References: <20241107203118.B4A5D53C@hemlock.eden.le-fay.org> Content-Language: en-US From: Pete Wright In-Reply-To: <20241107203118.B4A5D53C@hemlock.eden.le-fay.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[nomadlogic.org,quarantine]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[nomadlogic.org:s=04242021]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:29802, ipnet:66.165.240.0/22, country:US]; RCVD_COUNT_ONE(0.00)[1]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[nomadlogic.org:+] X-Rspamd-Queue-Id: 4XkvJM4mPRz40gT X-Spamd-Bar: --- On 11/7/24 12:31, Lexi Winter wrote: > hello, > > as this has come up before here (and didn't make it into the quarterly status > report), i just wanted to mention that i'm currently using p9fs(5) in -current > to export a filesystem from the host to a bhyve VM and, after some fixes made > in October, it's been rock solid for me. oh this is great news, i can't wait for this to land in a release branch. thanks! -pete -- Pete Wright pete@nomadlogic.org From nobody Thu Nov 7 20:47:59 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XkvKX5B13z5cbBv for ; Thu, 07 Nov 2024 20:48:00 +0000 (UTC) (envelope-from lexi@hemlock.eden.le-fay.org) Received: from fuchsia.eden.le-Fay.ORG (fuchsia.eden.le-fay.org [81.187.47.195]) by mx1.freebsd.org (Postfix) with ESMTP id 4XkvKX4VXYz41C1 for ; Thu, 7 Nov 2024 20:48:00 +0000 (UTC) (envelope-from lexi@hemlock.eden.le-fay.org) Authentication-Results: mx1.freebsd.org; none Received: from hemlock.eden.le-fay.org (hemlock.eden.le-fay.org [IPv6:2001:8b0:aab5:4::10]) by fuchsia.eden.le-Fay.ORG (Postfix) with ESMTP id AC68F868E; Thu, 07 Nov 2024 20:47:59 +0000 (UTC) Received: by hemlock.eden.le-fay.org (Postfix, from userid 10006) id A67CAA28; Thu, 07 Nov 2024 20:47:59 +0000 (GMT) From: Lexi Winter To: Bakul Shah cc: fs@freebsd.org Subject: Re: p9fs (bhyve) success report In-reply-to: References: <20241107203118.B4A5D53C@hemlock.eden.le-fay.org> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <77511.1731012479.1@hemlock.eden.le-fay.org> Date: Thu, 07 Nov 2024 20:47:59 +0000 Message-Id: <20241107204759.A67CAA28@hemlock.eden.le-fay.org> X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:20712, ipnet:81.187.0.0/16, country:GB] X-Rspamd-Queue-Id: 4XkvKX4VXYz41C1 X-Spamd-Bar: ---- Bakul Shah: > On Nov 7, 2024, at 12:31=E2=80=AFPM, Lexi Winter = > wrote: > > as this has come up before here (and didn't make it into the quarterly = > status > > report), i just wanted to mention that i'm currently using p9fs(5) in = > -current > > to export a filesystem from the host to a bhyve VM and, after some = > fixes made > > in October, it's been rock solid for me. > How is the performance compared to nfs?= i haven't done any specific performance testing. for the workload i'm interested in (which is mostly 50MB/s or less of largely random i/o) i haven't noticed any issues, though. From nobody Fri Nov 8 09:57:43 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlDs03nv3z5cVQn for ; Fri, 08 Nov 2024 09:57:56 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlDs03J1gz45wp for ; Fri, 8 Nov 2024 09:57:56 +0000 (UTC) (envelope-from girgen@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731059876; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=YLleXONc7i5yttBmgakKSyP4gx40ZnILWksPh6k90fU=; b=m6nyzDCd3qjUW3ZUB+NcGesv82ETccmUX0WkGiGLzp1CTUnS45FVSy41L0BINuO8DKfnVg fCcbF7FkDCG//GQdpaB9kUhXuYRAE5VZ0aGuv2gCA/w0l9cs3rs50IXNT8fv/MvuRBKhSe 0aL4LGRlD4CaJjxBIFq9sK71SC3KxqEF5Hl60lEz2j/M/d4kCMZV4VWiO6B8rbKLBAgBxx OQ4Ss8Kdcw3lsWLQzYirIiLSE00NdGJc/kjnaCa5L37LpU5F4I2Y3iH9NXKBN4mkBn8r56 fmEGlRxfjQ6OmoIr6Wm5yMESWb6wnuJj/vxPe++MYxC6Rk8iWXP2heXS4s0pEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731059876; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=YLleXONc7i5yttBmgakKSyP4gx40ZnILWksPh6k90fU=; b=j31VgI766gCRYQkLpx8owuLBdG8JtSm8gwtI5nfx2CrwJOhnSTO3R3AL34sw60Aujj9Zjz 6cn1Q2rT6suJSwPrImXuisckxWAUD3LvTDzElUUNWEPrW2pwS+Gf+iUQw7J8H7nIM5NQDo /6sGTo2UutxTQQeR7l5pstajxxrq+gL5oZFkyEHZDFiyRoGluHd4RuwaxskhUMioFhPryc reYjccAczdy1xNMnVKSWqm9gxaZFbRBO785TDP3fHoWGz196RVpOIm3mDyVHJ7KFY0jiUJ bdz/xKEA1Ff/owyrU1NNecxwPhjtLVlcJqDfZS03+o9TB/rFiNp6WD5NT/cz2Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731059876; a=rsa-sha256; cv=none; b=oHjvgWGQOHEgtjevD/hsqx3pQb0Rk6MyvxACl7pOu+GOwVumhwv07tw0eah2Y0fDmJvkkD TsY06NOVSVVtBTr+3lDTfs3O4Vn64IXQoE4AsZAPlWBaA0WrLiIR5kGcgI8jpOw3eawTWi IiODIPhai3SNkbrkUb1KsrCwd65b5Wz7ZB3yYAGDJRjgfKYbBP6h8kNss1KSwDjT/wW08W IiKSzpqyPBRaxgFNSMs1vnLATRhJJeFrsK+k4dDtWIemmCdsufEKsOQhqricEzRj8Ky0X1 nOfboRwTjQNVTo+4YI0IGHXhXLIGgH3X9cH3SICIUzZ6LMko/FvrF9sKrReK1g== Received: from smtpclient.apple (h-213-164-202-49.A498.priv.bahnhof.se [213.164.202.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: girgen/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4XlDs01812zYkM for ; Fri, 8 Nov 2024 09:57:56 +0000 (UTC) (envelope-from girgen@FreeBSD.org) From: Palle Girgensohn Content-Type: multipart/alternative; boundary="Apple-Mail=_9895E057-545C-47E4-AA1E-C2BBECE6E816" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.200.121\)) Subject: zfs snapshot corruption when using encryption Message-Id: <03E4CCF5-0F9A-4B0E-A9DA-81C7C677860C@FreeBSD.org> Date: Fri, 8 Nov 2024 10:57:43 +0100 To: "freebsd-fs@freebsd.org" X-Mailer: Apple Mail (2.3826.200.121) --Apple-Mail=_9895E057-545C-47E4-AA1E-C2BBECE6E816 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi! We se sporadical corruption in snapshots (not really files, it seems) = since we started using encryption on previously well behaved system. There a description in the OpenZFS github, = https://github.com/openzfs/zfs/issues/12014 , that seems to be spot on. = The issue is still open. I filed a bug report: = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622 Any ideas? Palle --Apple-Mail=_9895E057-545C-47E4-AA1E-C2BBECE6E816 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Hi!

We se sporadical = corruption in snapshots (not really files, it seems) since we started = using encryption on previously well behaved system.
= --Apple-Mail=_9895E057-545C-47E4-AA1E-C2BBECE6E816-- From nobody Fri Nov 8 13:50:22 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlL1H27mGz5cljC for ; Fri, 08 Nov 2024 13:50:27 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fhigh-b5-smtp.messagingengine.com (fhigh-b5-smtp.messagingengine.com [202.12.124.156]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlL1F6250z4WWl for ; Fri, 8 Nov 2024 13:50:25 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=I1CRz60H; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=CPXcvBLI; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 202.12.124.156 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from phl-compute-10.internal (phl-compute-10.phl.internal [10.202.2.50]) by mailfhigh.stl.internal (Postfix) with ESMTP id 80246254012A for ; Fri, 8 Nov 2024 08:50:24 -0500 (EST) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-10.internal (MEProxy); Fri, 08 Nov 2024 08:50:24 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1731073824; x=1731160224; bh=6OBn+qHSak LhcPUkTsgyqMSdTaB/orynDbmCjKP8m8w=; b=I1CRz60HsyFljyRenETsp52YnI wBMWstDYXAxw10US0IsxJrV9JwroCbibmE2HTNAJ7W/K6dV4xLqAq2Jyajy0ulOu wmwpI55RBAhIneZF2ZP7eaCrjbI5RYweeYT2LE26Xd9Lw8vXP7mndRpymZPyEv1b rXe9bYa2ugJDlQPHcHztKoIYMZvTKtMZkWVf+39QTG94szg5kxizM29pkL+1vnqP lbG0tjQVhQxbpKdU75fUathBHU485WAPgX8EmMtvQdJZ/9yA6wblVVmbdHu9Zdkn DJ4MLcLxmwd2TZ7MsCNbhhOo7GzNs/d+e2Tr+HLMHoxuSYbYBr5rx0+XzsLA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1731073824; x=1731160224; bh=6OBn+qHSakLhcPUkTsgyqMSdTaB/orynDbm CjKP8m8w=; b=CPXcvBLIOjHRda6FP6T3y1i6x3UAuJjMx9y+gfNmBe+GU7VSbzD 3Quxwwjt/oo50kujMLlzA41mt1Nxk0SLzCJ6MlkVx4hd6zDU84+GW4JdjAlE2xbK DmWzsy/QZRmucFYa85bDLP2gima+3QnBtNVPX6jmDo7Wt6KaYJpEbUtXqMd1Tc+Z JcsLciIYZ6w1jw8l9kznvuYthJpIt12yvWCe3IMw5FlsKwSb7qWK877ICncJfyPl TN18/D9Y1fAwUXO9Y4RXCVktJmAOa9lCxUIR3y4dUonGA1PDolO0y6ovPc20wtcL SWdJ4s/wdJAtK0C2jQ1BQ7yY7B/ZQ09YppQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrtdeigdehfecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukf hfgggtuggjsehttdertddttddvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdr fhhmqeenucggtffrrghtthgvrhhnpeeivdevueehffdtfffhvdefhefghfeikeetudffvd eitdfftdekudeileegffetheenucffohhmrghinhepghhithhhuhgsrdgtohhmpdhfrhgv vggsshgurdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilh hfrhhomhepvhhoihgusehfqdhmrdhfmhdpnhgspghrtghpthhtohepuddpmhhouggvpehs mhhtphhouhhtpdhrtghpthhtohepfhhrvggvsghsugdqfhhssehfrhgvvggsshgurdhorh hg X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 8 Nov 2024 08:50:23 -0500 (EST) Date: Fri, 8 Nov 2024 13:50:22 +0000 From: void To: freebsd-fs@freebsd.org Subject: Re: zfs snapshot corruption when using encryption Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org References: <03E4CCF5-0F9A-4B0E-A9DA-81C7C677860C@FreeBSD.org> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <03E4CCF5-0F9A-4B0E-A9DA-81C7C677860C@FreeBSD.org> X-Spamd-Result: default: False [-3.60 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:202.12.124.128/27]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[202.12.124.156:from]; FREEMAIL_FROM(0.00)[f-m.fm]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[202.12.124.156:from]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4XlL1F6250z4WWl X-Spamd-Bar: --- On Fri, Nov 08, 2024 at 10:57:43AM +0100, Palle Girgensohn wrote: >Hi! > >We se sporadical corruption in snapshots (not really files, it seems) since we started using encryption on previously well behaved system. > >There a description in the OpenZFS github, https://github.com/openzfs/zfs/issues/12014 , that seems to be spot on. The issue is still open. > >I filed a bug report: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=282622 > >Any ideas? No idea but i'd like to ask, in your context: do the snapshots get created, and if so, can you restore from them? In other words, does the 'corruption' fix itself, with zfs self-healing? -- From nobody Fri Nov 8 14:29:22 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlLtR1Xdsz5cKGg for ; Fri, 08 Nov 2024 14:29:35 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlLtR12kwz4cfW; Fri, 8 Nov 2024 14:29:35 +0000 (UTC) (envelope-from girgen@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731076175; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=rSnhNhyUDViyB/vzHhW4ZtjDLXGOARwBclTTB1Co+9w=; b=xDXlolgo9RaeAe2KijKWwB9PSpI6klA4H4ygLNZmU9rXjSkOJBa2oh3TqrnHJX5c5v3wk6 mZZjvxJHtjl9a3K4uMzhTtMvHMADHdCrn3n6Ukqaj3uhHuUl2w4MYGBX9eHPtG3iTajluM Jk3LtIcJX1vDE4rxqN605VbPn+BzBxfvCjrk5kkOhqDEsFUcnefV0riR5MpdpE9a1my+y8 UoySnhnr/s8/jXri15zDkuxP2op93o57cvoJkS5+QWyl/iG7Lou0CMbYysC8SOhcmP5/5J BalEOAMcN7PiEw5FBaaYQtuV88nYtw3JAPTwSa81q1EMylxpQ9slUH0mvrAz9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731076175; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=rSnhNhyUDViyB/vzHhW4ZtjDLXGOARwBclTTB1Co+9w=; b=UnhsUR4UOUb2xeknf8h0Fh2oDZ1GcdvB6YXVlM82J/KZ+KERbrueTbAZaOg0lJlokpqaCO O+pnFRZ6VQylCoTjeoR6wt0I0fg/LqZXacNmbomL4yDnk4IVN4qktSgSKW01eKVpvWhmfV ALzP4ijxVs8QBL/NXE7dHrjLxehPm2bZ8CUqhNe+1FBTQ/rS9BZrFCxly15DDsXPOSS5YA mJmxK8mH229MyTRx8dzBEpj/dK4AUhVRrQ4ABnv+OYrJuIp2XwUiy+UeedmX0KdAXd31Ws i435dRkK70eCNqmcHlWEocAEv1at5j3Cu3TE5P4W0EQM/koPnQQ3oyGAn/J1ug== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731076175; a=rsa-sha256; cv=none; b=yJtkyKWEcckO6TU04Lh/FP74FYjIyIwrNW8HVqGOGzaehBlmn3PnorbJhQsDkzVw+U6L2k 8miNrYtYRqPQ9FD7KdQ5QUiiRa8jVS7OqQQjtKrb/2n1KqmVQrOfidGI/PBIC7036GRKlD lAdc+kKUztFkXcv5ki9XB5xFE/iHYdm9p+DJx0lxjbMVKkXekdzQPwvX3ahDWozPoLRD61 DpzC98ajbJdXadmHZGA2C93dducp7Kop6VX6PFNajaSFhAxgYIKEpwLdqUoHhmgMx5RQ8+ 255gKfDnHc8Ip4ALGUDIBIMv8apYmu996sScZm3HoW98rFTzQFfFO7PvtEGvew== Received: from smtpclient.apple (visitor.bahnhof.se [213.136.33.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: girgen/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4XlLtQ5QQ6zfCB; Fri, 8 Nov 2024 14:29:34 +0000 (UTC) (envelope-from girgen@FreeBSD.org) From: Palle Girgensohn Message-Id: <3E85AAAE-8B1E-47C7-B581-E3D98AB03907@FreeBSD.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_6FFFCB03-1F59-4ADF-8B24-40751C8D519A" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.200.121\)) Subject: Re: zfs snapshot corruption when using encryption Date: Fri, 8 Nov 2024 15:29:22 +0100 In-Reply-To: Cc: freebsd-fs@freebsd.org To: void References: <03E4CCF5-0F9A-4B0E-A9DA-81C7C677860C@FreeBSD.org> X-Mailer: Apple Mail (2.3826.200.121) --Apple-Mail=_6FFFCB03-1F59-4ADF-8B24-40751C8D519A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > 8 nov. 2024 kl. 14:50 skrev void : >=20 > On Fri, Nov 08, 2024 at 10:57:43AM +0100, Palle Girgensohn wrote: >> Hi! >>=20 >> We se sporadical corruption in snapshots (not really files, it seems) = since we started using encryption on previously well behaved system. >>=20 >> There a description in the OpenZFS github, = https://github.com/openzfs/zfs/issues/12014 , that seems to be spot on. = The issue is still open. >>=20 >> I filed a bug report: = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622 >>=20 >> Any ideas? >=20 > No idea but i'd like to ask, in your context: do the snapshots get = created, and if so, can you restore from them? In other words, does the = 'corruption' fix itself, with zfs self-healing? I cannot run `zfs send -I fs@previous_snap fs@problematic_snap`, I get warning: cannot send 's@problematic_snap': Input/output error` Removing the snapshot fixes the problem.= --Apple-Mail=_6FFFCB03-1F59-4ADF-8B24-40751C8D519A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

8 nov. 2024 kl. 14:50 skrev void <void@f-m.fm>:

On = Fri, Nov 08, 2024 at 10:57:43AM +0100, Palle Girgensohn = wrote:
Hi!

We se sporadical = corruption in snapshots (not really files, it seems) since we started = using encryption on previously well behaved system.

There a = description in the OpenZFS github, = https://github.com/openzfs/zfs/issues/12014 , that seems to be spot on. = The issue is still open.

I filed a bug report: = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622

Any = ideas?

No idea but i'd like to ask, in your context: = do the snapshots get created, and if so, can you restore from them? In = other words, does the 'corruption' fix itself, with zfs = self-healing?

I cannot run `zfs send -I fs@previous_snap = fs@problematic_snap`, I get

warning: cannot send 's@problematic_snap': Input/output = error`

Removing the snapshot fixes the = problem.= --Apple-Mail=_6FFFCB03-1F59-4ADF-8B24-40751C8D519A-- From nobody Fri Nov 8 19:20:58 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlTLk5dXhz5cgDs for ; Fri, 08 Nov 2024 19:21:02 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fhigh-a4-smtp.messagingengine.com (fhigh-a4-smtp.messagingengine.com [103.168.172.155]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlTLj3PyPz4BpK for ; Fri, 8 Nov 2024 19:21:01 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=CtLcGwtb; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=DWuQyUk+; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 103.168.172.155 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from phl-compute-04.internal (phl-compute-04.phl.internal [10.202.2.44]) by mailfhigh.phl.internal (Postfix) with ESMTP id 8077D11400EC for ; Fri, 8 Nov 2024 14:21:00 -0500 (EST) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-04.internal (MEProxy); Fri, 08 Nov 2024 14:21:00 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1731093660; x=1731180060; bh=FQGlT2K7H0 UjM1BAB5a4NKjM2ATL0jkcWXNqGKs1c8A=; b=CtLcGwtbr9ASw8MkIFMjUn58X6 1K5I4YZKmnhTcVKK0PNj8A+z4SPpNfmNheS9I6Z5p+4ooSUzFdiT8Y7ExS4p+Jzp AGIfoYDPI6g8dnlEVU+8LZhS6TAAnFo+REnryqv2F1REwAr8hF0vKxPlLGNTfPpy pvq0Y6Ll3Hi1DUEleIzkPZzEppjNEwlV6JdwDVLB2mjkqx+M9Hrtt6tsh0FXduWz OM/hJ8NuJpH0tR3Lk17rKj+56bn40cJYuxTQb3NBRVrtm/K5/1aUv2yI2Gf3Lddr z6zQejw5p4WDkQn+++HUrOwFtOG5MJitEWACjsvX68+I9hkM3gCLZkroDr+g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1731093660; x=1731180060; bh=FQGlT2K7H0UjM1BAB5a4NKjM2ATL0jkcWXN qGKs1c8A=; b=DWuQyUk+1VJzirrIk1Me+bcCYmPLrI2baLaG3xu9c6hjIkrX0s5 sPR/vOLPF7JXwQErOSNLK3/FzwdB3IMhHCx0Fs330PN2KxsgeXsr7RwNjk3Jw2is 1gXuQYq0SH79zxCbhiboBhoBhOar/lQKEneVbOGXM3yt2fTkeqdOGsX2qGcPmbrU rfXBFe+yPQvNvlj6fk5FlieA7WMN3F3Q0w6Y9TP+hKSSSXVc4yD6vfdfSzX55wDu YQczB+xHdFzc5tc2w+g5l8yoY26LgkePGdaXJzLowDaA9TBz13FGu5Riy73B8d3g JUt4XyyhS8IWFqx9ZdQw9VrFMqiK2VbadtA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrtdeigdduvddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuf fkfhggtggujgesthdtredttddtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhm rdhfmheqnecuggftrfgrthhtvghrnhepkeeluddvlefhieelfefggffhffektdehleelgf dugfdvgeekjeejuddtheehgfeunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghm pehmrghilhhfrhhomhepvhhoihgusehfqdhmrdhfmhdpnhgspghrtghpthhtohepuddpmh houggvpehsmhhtphhouhhtpdhrtghpthhtohepfhhrvggvsghsugdqfhhssehfrhgvvggs shgurdhorhhg X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 8 Nov 2024 14:20:59 -0500 (EST) Date: Fri, 8 Nov 2024 19:20:58 +0000 From: void To: freebsd-fs@freebsd.org Subject: Re: zfs snapshot corruption when using encryption Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org References: <03E4CCF5-0F9A-4B0E-A9DA-81C7C677860C@FreeBSD.org> <3E85AAAE-8B1E-47C7-B581-E3D98AB03907@FreeBSD.org> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <3E85AAAE-8B1E-47C7-B581-E3D98AB03907@FreeBSD.org> X-Spamd-Result: default: False [-3.60 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:103.168.172.128/27]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[103.168.172.155:from]; FREEMAIL_FROM(0.00)[f-m.fm]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; MISSING_XM_UA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4XlTLj3PyPz4BpK X-Spamd-Bar: --- On Fri, Nov 08, 2024 at 03:29:22PM +0100, Palle Girgensohn wrote: >I cannot run `zfs send -I fs@previous_snap fs@problematic_snap`, I get >warning: cannot send 's@problematic_snap': Input/output error` >Removing the snapshot fixes the problem. What an odd problem. Like, it can write but not read. What zfs version? On 13.3 I'm using: zfs-2.1.14-FreeBSD_gd99134be8 zfs-kmod-2.1.14-FreeBSD_gd99134be8 on 14-stable: zfs-2.2.6-FreeBSD_g33174af15 zfs-kmod-2.2.6-FreeBSD_g33174af15 on 15-current: zfs-2.3.99-31-FreeBSD_gb2f6de7b5 zfs-kmod-2.3.99-31-FreeBSD_gb2f6de7b5 Is the encryption you're using is the GELI based whole-disk one (which has been around iirc for a few years) or the relatively recent zfs encryption that works per-filesystem. This is why I'm asking the zfs version. I thought the latter is relatively quite new, and I've never heard of it working on 14.0. The way I'd go about trying to address the issue would be to start from the lowest layer and work up. On a raidz2 array (the card in JBOD mode), I can identify all the drives as da0-7, so can use smartctl -x to query them all directly, looking for, in particular: Reallocated_Sector_Ct Reported_Uncorrect Current_Pending_Sector Offline_Uncorrectable If your disk arrangement is connected via something like HP Smart Array or similar, you'll need to look at the man page for smartctl for the exact syntax to query the disks behind the card. -- From nobody Fri Nov 8 22:11:23 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlY7Y24ywz5csTW for ; Fri, 08 Nov 2024 22:11:37 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlY7Y1nmjz4S0w; Fri, 8 Nov 2024 22:11:37 +0000 (UTC) (envelope-from girgen@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731103897; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=9c5aMOhvpS8lgbbOUg2UzpDwzwNg+YSf33P+P5GmR3g=; b=tpnviEGFUtFKNjeGda61PjAu83M7AFmwOI8jfdJE1t54shVnS7d8wR342xZ9LMSRoSYswk /qirg5aAaxQazx3oA1QK+DKxGyvBxsE7LSasctW49suCBZ+VCJ8Rm7ywkukWCugjn7Km1Y yJv+13ll2mSm39EU/wgUC39JxwUvdpitXJa34cqAXgqGq4OBbC2omgevEIZpdplBcKuoiA 1fRoGAdRkw9Mm9Gm0h8NQLVzQeBx1CHN0sLfaRWiwMTckpMt7n6HThSG9EkFnb/7w9U2YF 5ZBc7AavHut/sSy4DBTyNn5NFzGjA0+938d8qgF4sIqBzql40OxnJ+VuRbsHug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731103897; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=9c5aMOhvpS8lgbbOUg2UzpDwzwNg+YSf33P+P5GmR3g=; b=tGu2sLHsZcD1NAcOSf9w854s533QJuLWa21BrBLzAwLfTFI/6FaooxooNfeAELFmKSb93X FVvbm65vceYO5ka+YabcWrl+b9Zeg+oN6ceH3VC4y5SA62SRyV5ag2To7rimKHShmQkGcx 0QaZ+NQ+XaiXHhn8Niqx7640XNxiHBio7M6ZC4f+mf385fmYFdtSq9hgaJmfI5KbCMW55n hJzrn69pvXgR4OSXW8C+3BSMK+ezTZWJMO+ezRpvFnYWVtAbQHikZNpKxrD6FnK65rD1WC Vm8O3l6P+uFwQ7+oh9zCZhlAu5hN7LpqL00VF/VJ2VC41lXRqyvvluHIDrsFpA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731103897; a=rsa-sha256; cv=none; b=u7d446mLLnXz9wyPR+BWSL8k+H2voWVicwdz/SfTx7QOLyQz3Z4R6Ln0Hog42rltrIHtFp CPBdaSS+DFNuTFxRt8+409yabUV6KFAU91gi52fHNLNPKeB0R0XoU1aAoE0VCkOzutdIy4 KeaS8y723nE1cFW+wgbM691vuPfffmNYKRm1zZM7i2B8AehSJrmuqWeHMOuytdcBzUTId4 2q/+8iTmHHMe0Tj9I1S22R+WJ1sgTPaUs+1R6dk68FBrKB5U5+ZNU9MjhAj6KEd0DWxmCk 2EthKQ5KPYqQzldQNdgJlORl568oCxZ0f4gcemfs8MBzpdrt6lxTs9iY3liaUA== Received: from smtpclient.apple (h-213-164-202-49.A498.priv.bahnhof.se [213.164.202.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: girgen/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4XlY7X5cPtz155R; Fri, 8 Nov 2024 22:11:36 +0000 (UTC) (envelope-from girgen@FreeBSD.org) From: Palle Girgensohn Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_59E36AA6-CDF3-40DA-93B9-25F2EB7D1973" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.200.121\)) Subject: Re: zfs snapshot corruption when using encryption Date: Fri, 8 Nov 2024 23:11:23 +0100 In-Reply-To: Cc: freebsd-fs@freebsd.org To: void References: <03E4CCF5-0F9A-4B0E-A9DA-81C7C677860C@FreeBSD.org> <3E85AAAE-8B1E-47C7-B581-E3D98AB03907@FreeBSD.org> X-Mailer: Apple Mail (2.3826.200.121) --Apple-Mail=_59E36AA6-CDF3-40DA-93B9-25F2EB7D1973 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > 8 nov. 2024 kl. 20:20 skrev void : >=20 > On Fri, Nov 08, 2024 at 03:29:22PM +0100, Palle Girgensohn wrote: >=20 >> I cannot run `zfs send -I fs@previous_snap fs@problematic_snap`, I = get >> warning: cannot send 's@problematic_snap': Input/output error` >> Removing the snapshot fixes the problem. >=20 > What an odd problem. Like, it can write but not read. What zfs = version? > On 13.3 I'm using: > zfs-2.1.14-FreeBSD_gd99134be8 > zfs-kmod-2.1.14-FreeBSD_gd99134be8 >=20 > on 14-stable: > zfs-2.2.6-FreeBSD_g33174af15 > zfs-kmod-2.2.6-FreeBSD_g33174af15 >=20 > on 15-current: > zfs-2.3.99-31-FreeBSD_gb2f6de7b5 > zfs-kmod-2.3.99-31-FreeBSD_gb2f6de7b5 >=20 $ freebsd-version -ku 14.0-RELEASE-p10 14.0-RELEASE-p10 dunno how to present in the above fashion? Binary install, not from = source. > Is the encryption you're using is the GELI > based whole-disk one (which has been around iirc for a few years) or > the relatively recent zfs encryption that works per-filesystem. This = is why I'm asking the zfs version. I thought the latter is relatively = quite new, and I've never heard of it working on 14.0. The latter, zfs per-filesystem zfs encryption. I haven't heard any = horror stories about it, albeit it's quite fresh code. > The way I'd go about trying to address the issue would be to start = from the lowest layer and work up. On a raidz2 array (the card in JBOD = mode),=20 yes, JBOD > I can identify all the drives as da0-7, so can use smartctl -x to = query them all directly, looking for, in particular: >=20 > Reallocated_Sector_Ct > Reported_Uncorrect > Current_Pending_Sector > Offline_Uncorrectable >=20 > If your disk arrangement is connected via something like HP Smart = Array > or similar, you'll need to look at the man page for smartctl for the = exact > syntax to query the disks behind the card. Na, it's JBOD alright. I see nothing weird with the disks. Palle= --Apple-Mail=_59E36AA6-CDF3-40DA-93B9-25F2EB7D1973 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

8 nov. 2024 kl. 20:20 skrev void <void@f-m.fm>:

On = Fri, Nov 08, 2024 at 03:29:22PM +0100, Palle Girgensohn = wrote:

I cannot run `zfs send -I = fs@previous_snap fs@problematic_snap`, I get
warning: cannot send = 's@problematic_snap': Input/output error`
Removing the snapshot fixes = the problem.

What an odd problem. Like, it can write = but not read. What zfs version?
On 13.3 I'm = using:
zfs-2.1.14-FreeBSD_gd99134be8
zfs-kmod-2.1.14-FreeBSD_gd99134= be8

on = 14-stable:
zfs-2.2.6-FreeBSD_g33174af15
zfs-kmod-2.2.6-FreeBSD_g3317= 4af15

on = 15-current:
zfs-2.3.99-31-FreeBSD_gb2f6de7b5
zfs-kmod-2.3.99-31-Free= BSD_gb2f6de7b5


$ freebsd-version -ku
14.0-RELEASE-p10
14.0-RELEASE-p10

dunno = how to present in the above fashion? Binary install, not from = source.

Is the encryption you're using is the = GELI
based whole-disk one (which has been around iirc for a few = years) or
the relatively recent zfs encryption that works = per-filesystem. This is why I'm asking the zfs version. I thought the = latter is relatively quite new, and I've never heard of it working on = 14.0.

The = latter, zfs per-filesystem zfs encryption. I haven't heard any horror = stories about it, albeit it's quite fresh code.


The way I'd go about trying to address the issue = would be to start from the lowest layer and work up. On a raidz2 array = (the card in JBOD mode), 

yes, JBOD

I can identify all the drives as da0-7, so can use smartctl -x to = query them all directly, looking for, in = particular:

Reallocated_Sector_Ct
Reported_Uncorrect
Current_= Pending_Sector
Offline_Uncorrectable

If your disk arrangement = is connected via something like HP Smart Array
or similar, you'll = need to look at the man page for smartctl for the exact
syntax to = query the disks behind the card.

Na, it's JBOD alright. I see nothing = weird with the disks.

Palle= --Apple-Mail=_59E36AA6-CDF3-40DA-93B9-25F2EB7D1973-- From nobody Sat Nov 9 01:09:49 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xld595X6bz5d44y for ; Sat, 09 Nov 2024 01:09:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xld592jcWz4fx6 for ; Sat, 9 Nov 2024 01:09:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731114589; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cSvRpJ5KG3/FvpR0SPXhjwIH2H3B/bNwabwL6Z6BDV4=; b=g2u+IOMAaGgIj8aiGI4RlwE3qO8g/acYy6dIO3oKWpnE/RkXR+QK7dHeYLFYKk8lHF7ySA ZphrbBOl5OnoiLBXovT/O+53h5SVCcLHY0u5iKA07oRT6YsKYwcX0Y27H+D+QVwGlOADXa XQtHspnYrQ0NJlCvt1/UNq1lLNkfpJyYRwRKsInwk+qHa7OVuCFymz/qH39ExOMbz6NhZa j240H4DpQ9UdtYeKMo0KTy9x5ELOBcopdbWRehwJ7Pw59meGgghPjBaqmv7A6S6ZoUzSMG VHqLUzI4gPLYqKClikBiWimBl1+Hg6Dh0z96sYtHRZQgcn2l/UOGo0YBfTn0Zg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731114589; a=rsa-sha256; cv=none; b=VjZAmfs2ZdNT/Ihw+14QYWkoNnJPAadnQAR/tZ2mJaJqhfZBwHnXNr6RTAKu3/9PoWR8IV KuhfeGvTWjZpeGmUfHOzNVeo56+esIJhm9I08k+MpOfLHq/YJPHu0GcZl9CS115xmep5+F SiWvGf2xgKUNpejnaAK5Up9nZ7bZGIIt2Souz4p7S+uarln+PJtz/ZJwhj8b5UreyD6Lqt bgci+op7A2xllEPuMM0i/+tBDh88f15/HES1W9AzrkxKlKGaIVAKsMrroxgJ9X4VjalHTj oNYoqM+Q6+NXMiEuXsQ0LpLtLEmCz2krzt1Ip1jFvOoAxOMSfrUhnbKiWivcQw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Xld592KtjzsQ9 for ; Sat, 9 Nov 2024 01:09:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A919nwM059019 for ; Sat, 9 Nov 2024 01:09:49 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A919nuh059018 for fs@FreeBSD.org; Sat, 9 Nov 2024 01:09:49 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282622] zfs snapshot corruption when using encryption Date: Sat, 09 Nov 2024 01:09:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Nov 9 01:45:31 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XldtM2FZ9z5d6P6 for ; Sat, 09 Nov 2024 01:45:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XldtM0s7Mz4n11 for ; Sat, 9 Nov 2024 01:45:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731116731; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3fdCVRyMmmc25qWdAEYL117AnYsVQtEPW8K996v1nno=; b=wq4Qu/4HvtaS3bnpnV/HqKtvtnHINLfYTwfV2F0c27mcQC2oV60TU+A8woaBKIDVicBAvu Lb7gRUmUKrVw8uEwb+ZP7KfgagzMUdGekhwDvTSrCiFc2/D8NwPbDYK792dWfu/Xobi9gb +OdVxcwdCf94wK2uRqCBcUfTHsn3NU7nZwdD5MisqNQswMDUQp3gAyCSL4EUVRDEUDzfgc P625k9QG/RyQJAaqzssteKEBjyTDZUnZMok2QgloK2M6ylnUW+G592bgEGY0LTHNLGUV+t upv3pxlN6y6Ifh9zUnK23vvSWx1wuL1J0dpozmsneHnLgVYFWXW3hdxZEudgrA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731116731; a=rsa-sha256; cv=none; b=OzTb7OfMQcuMTI2rxgg+Ew1888vSwL/Yi8mmuksLP52Gt5SR6QWogbCZpvv8hv0xDWbubu 4oKxhx5JZXftzgzR298t3Q7yInex0RecG2We7u7xMFYPi78Np9RSsDNd2fTj6xaKZcW8gA B+vHPtk9EObyACdSRZC/aaAyBNssbt9NmdPEdncXw9UrEcmxRRvU/XzCWMd3+d33U64sss FKEZVm84nbGX23oKUGtqpSEPe4jF93OpCkJiBp5pQwUkEyy31WTX+z/gfILSTDsCItv3Ia Pvv+uELfF3epdb+spqgBseqdteagFQfsdUHd8h8KkgjVqX4DE1ELEOpg721vEA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XldtM0KqHztjK for ; Sat, 9 Nov 2024 01:45:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A91jU0f017522 for ; Sat, 9 Nov 2024 01:45:30 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A91jUif017521 for fs@FreeBSD.org; Sat, 9 Nov 2024 01:45:30 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282622] zfs snapshot corruption when using encryption Date: Sat, 09 Nov 2024 01:45:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ft@hemlock.eden.le-fay.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622 Lexi Winter changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ft@hemlock.eden.le-fay.org --- Comment #3 from Lexi Winter --- i've run into this about a year ago on 13.something, and it's still present= in 15.0. a workaround to avoid removing the snapshot is to reboot, which will cause = the snapshot to work again: the issue seems to be purely with the data in memor= y. there should probably be a warning somewhere in the manual that ZFS encrypt= ion is not actually usable, or perhaps it should even be locked behind a sysctl. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Nov 9 01:59:35 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlfBg5rybz5d739 for ; Sat, 09 Nov 2024 01:59:39 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fout-a5-smtp.messagingengine.com (fout-a5-smtp.messagingengine.com [103.168.172.148]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlfBf6BG0z4qyc for ; Sat, 9 Nov 2024 01:59:38 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b="U6T2w2U/"; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=hiMLXjp6; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 103.168.172.148 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from phl-compute-09.internal (phl-compute-09.phl.internal [10.202.2.49]) by mailfout.phl.internal (Postfix) with ESMTP id 85BD31380368 for ; Fri, 8 Nov 2024 20:59:37 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-09.internal (MEProxy); Fri, 08 Nov 2024 20:59:37 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1731117577; x=1731203977; bh=GYWr5PU40o ETdnsxpNlF1PDDq08AFcxlhzdy1VQnVz8=; b=U6T2w2U/rGp2GxWLGjODTL2bbd +iLT4jB+OXJwBn0R/87TPOY+Xz0gPvptPPPN5YqOGUGu4nTgAABhEZISTl1JjAF8 exA/z1EjtHejtL/Adn73Ohub39qS9xLJKalWLeB0JOdePsuFcN8Gutvx73kOgwMG HyALoV7U87n2hMRiKI7AeaDU4LzlsoI4qg6TyN4ukXAQEgAwdS7+Zl+tyMWSGf/7 zaip13BBiPbfMASPUaiBqXj2BOSh/zOV/K+AZu/LxrsJ3iwR5rCl7KVC9OnSX8bi ElEtU/VVM2WUUhfWxkKNkk/g2Mv+jk2AesIoOPpp5JlC7Z9ld1v/7kIJdaqA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1731117577; x=1731203977; bh=GYWr5PU40oETdnsxpNlF1PDDq08AFcxlhzd y1VQnVz8=; b=hiMLXjp6bF1I8cUbYAddFUHXfhSgGipTmDmls/+4++CWQhDAUt6 aCwGxtZivHkk2Ek7lpetMHJ8+MGah6MUxYcYL17OBDKxdlM5AhSjL7kKRdr/GwMF 4iVf9lENFBmMPFDmO79t6zLz2ccUH+tskz6qoi1G7TLBZLHw3IhZoJi0qf1FyItM fuhuXuK+xnDkD/oAP9FzRV3uQobfNG8SNLykrs9yDoyLw6psW+TxKDsugnoeZvTn 8+aCEaAdTMXvNk7UXVIAa9D2U97EifABSWHdMuQY3LbFofngg7R10LXF/ee+k4ts Cm/V2NEuaLALaIBmYcDqeihJbmHP89R9aIg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrtdejgdegudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukf hfgggtuggjsehttdertddttddvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdr fhhmqeenucggtffrrghtthgvrhhnpedthfehueetjeduffekhfdthfefkeegtddvieetve fgjedtgfeffffhueeiledtieenucffohhmrghinhepfhhrvggvsghsugdrohhrghenucev lhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehvohhiugesfh dqmhdrfhhmpdhnsggprhgtphhtthhopedupdhmohguvgepshhmthhpohhuthdprhgtphht thhopehfrhgvvggsshguqdhfshesfhhrvggvsghsugdrohhrgh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 8 Nov 2024 20:59:36 -0500 (EST) Date: Sat, 9 Nov 2024 01:59:35 +0000 From: void To: freebsd-fs@freebsd.org Subject: Re: zfs snapshot corruption when using encryption Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org References: <03E4CCF5-0F9A-4B0E-A9DA-81C7C677860C@FreeBSD.org> <3E85AAAE-8B1E-47C7-B581-E3D98AB03907@FreeBSD.org> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-3.60 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:103.168.172.128/27]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[103.168.172.148:from]; FREEMAIL_FROM(0.00)[f-m.fm]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; MISSING_XM_UA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4XlfBf6BG0z4qyc X-Spamd-Bar: --- On Fri, Nov 08, 2024 at 11:11:23PM +0100, Palle Girgensohn wrote: >$ freebsd-version -ku >14.0-RELEASE-p10 >14.0-RELEASE-p10 > >dunno how to present in the above fashion? Binary install, not from source. % zfs version >The latter, zfs per-filesystem zfs encryption. I haven't heard any horror >stories about it, albeit it's quite fresh code. I'm wrong about native zfs encryption not being on 14.0. Seems like it is. https://www.freebsd.org/releases/14.0R/relnotes/ I'm reading this bit: ### ZFS Changes OpenZFS has been upgraded to version 2.2. New features include: block cloning, which allows shallow copies of blocks in file copies. This is optional, and disabled by default; it can be enabled with sysctl vfs.zfs.bclone_enabled=1. scrub error log (zpool scrub -e) BLAKE3 checksums, which are fast, and are now the recommended secure checksums corrective zfs receive can heal corrupted data vdev and zpool user properties, similar to dataset user properties. ### Then again, 14.0 is EoL. >Na, it's JBOD alright. I see nothing weird with the disks. >> Reallocated_Sector_Ct >> Reported_Uncorrect >> Current_Pending_Sector >> Offline_Uncorrectable Were all these values '0' ? Using truss might give a clue about the failure: # truss -o /tmp/truss.out zfs-send-commands make sure the outfile is made in a place with lots of spare space. -- From nobody Sat Nov 9 02:08:50 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlfPH0dYGz5d7XF for ; Sat, 09 Nov 2024 02:08:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlfPG6fSCz4rdn for ; Sat, 9 Nov 2024 02:08:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731118130; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mR3jnrX49yoBvcdQWknMlHdg2jZkESAxc5ZyK7QvL2k=; b=hNNA0XMlwa2bQeDlhfZLueD+ZJ9OPmw0ziUusGmgKygJ/dV4N7tDi/VsTs99RBRNVMz0o1 kzgldPY1rHI/OKT4/1iOcfEG/aIWgPx+pgnFJt7KjyUqmcPPHs815TCptW5E6b1NsrLXs9 CnV2c3TfH4ZwErg8yyBR4MM/AGV00nC8UowL9OxpDSV1u3D5AsSVpukPSfTTawqQ4jKf9U sTwYTj91B2ey7xY2Wq2PVf01t/LAxopkXF32vJdTQeD3CDz+E0iqBPZVGviZpcVvqsU92j iNj2vxNjOXYmrLD+ZRxL54cmF5jW1ZgwzMKLj7GtMEKLkWEeaYaOnSYEqOXi9Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731118130; a=rsa-sha256; cv=none; b=kpjK/irB9HHsrV6MFsUObYiKkEa9cJAGT29A8LMIJS1d3ROU22KaRMYH8xZJQ2zcPiYb3G 81aXt3rUMnw4iSCzDb2NyiOtRsSg/3bNuSD6PPlKpGbjyBeujdFLxt+Ko1TPOBBlWa+AZ2 n/BvUSdCLgWIDjyEKqDWDPmovZoD6YIpzvOo0taepGkst4vh/d40zML5MGwuPMs6B4hs4t BYjwHacZ/0S8gNLOfnVtrSDgiSN9F2lbs1vGJFCBs67EWzJI58HuQ2jXGOjsy4vM5ATPHp Otc1GXwktXWhXmxZ1cjn4EFsEz0+heGOZnxh6STk5coObRGny1vHsrS0uLcc6A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XlfPG677cztq0 for ; Sat, 9 Nov 2024 02:08:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A928o7N051211 for ; Sat, 9 Nov 2024 02:08:50 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A928oGC051210 for fs@FreeBSD.org; Sat, 9 Nov 2024 02:08:50 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282622] zfs snapshot corruption when using encryption Date: Sat, 09 Nov 2024 02:08:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: void@f-m.fm X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622 void@f-m.fm changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |void@f-m.fm --- Comment #4 from void@f-m.fm --- (In reply to Lexi Winter from comment #3) > i've run into this about a year ago on 13.something, and=20 > it's still present in 15.0. That's interesting. can you reproduce the error reliably? > a workaround to avoid removing the snapshot is to reboot,=20 > which will cause the snapshot to work again: the issue seems=20 > to be purely with the data in memory. How much memory? Any zfs sysctls? vfs.zfs.arc.max? > there should probably be a warning somewhere in the manual that=20 > ZFS encryption is not actually usable, or perhaps it should=20 > even be locked behind a sysctl. In my context (15-current) it can be used but haven't used it yet=20 for snapshot/restore (am using tar for this). --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Nov 9 03:56:21 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlhnP21ySz5dFTk for ; Sat, 09 Nov 2024 03:56:25 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fout-a6-smtp.messagingengine.com (fout-a6-smtp.messagingengine.com [103.168.172.149]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlhnN3V7zz46ny for ; Sat, 9 Nov 2024 03:56:24 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=I2hS1P9A; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=nNGYopUf; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 103.168.172.149 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from phl-compute-09.internal (phl-compute-09.phl.internal [10.202.2.49]) by mailfout.phl.internal (Postfix) with ESMTP id DD6FB1380434 for ; Fri, 8 Nov 2024 22:56:23 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-09.internal (MEProxy); Fri, 08 Nov 2024 22:56:23 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1731124583; x=1731210983; bh=I0ae42txvt TSQwOF+vRKd3h53+7jUOTHrjiRIypk1vA=; b=I2hS1P9AIZfKnso8/LbGrJX1ke RDN+ZvIzd3F2zvZb62Tis4RwXhiBwoTeSWAQHAzvVmU9bco8AOAHVU7CwgpKmcXq OJkbUpqwVn2flEE1xiLYVCBQydBQUp+iA9S6PoHOHpPcf4rjbBALtNYb64RbRF3P ANg3gUMcwhQUpZpZhL2bHkQQ8M5pgvz6gk0qiH0x/vwMFk4xr5lXwfhWHDyZGKoZ 4jxZSGqm/NPx/xUJG1laK4e8UanmJxwHbJrUOIUFf3vNKEltascU4Koy7GE74AKN +FKvfd/Y9IyWSbs5eejC7yZI1RWcsV/NNJ/UgXtghlT0JqzWEyR+JHKX2NIg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1731124583; x=1731210983; bh=I0ae42txvtTSQwOF+vRKd3h53+7jUOTHrji RIypk1vA=; b=nNGYopUf6NkE4nvFMulLa/E4tkBiw470vup3u2+pFccDZTP+zFn wFIaUaljl24OwqNmojjJAvLP+B0N/1HnmXb40GsywQpmO1qlQnsp7/grMeB96azQ VkxtpUpirCS+pWXsY/h/Gq1rfieddJ26veo3NxCB09fElP4fdHz7jwuyey816hp/ wVC1CwYHAlcHP+nobpYfKfPLOPDFcOMGgzPNJMTkQ8hLepxSLkLKpPabTTDyh75v Z+lKM4+IqXXjcQWPj1Tf2AAb9VZO4P/J32h2X/cfxhEUPNl8tb1zoY9pyoxK3QSS pITv2HSnmJwYjE0uZQhmscAWYXwuBofm5Bg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrtdejgdeiiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukf hfgggtuggjsehttdertddttddvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdr fhhmqeenucggtffrrghtthgvrhhnpeeftdfgtefhfeekjeehhfehteetveejieeffeduge ehgeefleefgeegveevudegvdenucffohhmrghinhepghhithhhuhgsrdgtohhmnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepvhhoihgusehfqd hmrdhfmhdpnhgspghrtghpthhtohepuddpmhhouggvpehsmhhtphhouhhtpdhrtghpthht ohepfhhrvggvsghsugdqfhhssehfrhgvvggsshgurdhorhhg X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 8 Nov 2024 22:56:23 -0500 (EST) Date: Sat, 9 Nov 2024 03:56:21 +0000 From: void To: freebsd-fs@freebsd.org Subject: Re: zfs snapshot corruption when using encryption Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org References: <03E4CCF5-0F9A-4B0E-A9DA-81C7C677860C@FreeBSD.org> <3E85AAAE-8B1E-47C7-B581-E3D98AB03907@FreeBSD.org> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-3.60 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:103.168.172.128/27]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[103.168.172.149:from]; FREEMAIL_FROM(0.00)[f-m.fm]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; MISSING_XM_UA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:151847, ipnet:103.168.172.0/24, country:AU]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4XlhnN3V7zz46ny X-Spamd-Bar: --- Looking further at this, there seems to be many reports of the same sort of problem you're reporting: https://github.com/openzfs/openzfs-docs/issues/494 https://github.com/openzfs/zfs/issues/11679 https://github.com/openzfs/zfs/issues/16623 https://github.com/openzfs/zfs/issues/10019 (I'm going to turn off zfs-encryption on the few datasets where it's enabled, until it's properly fixed; thank you for reporting this). -- From nobody Sat Nov 9 10:20:52 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlsK02ktXz5bw32 for ; Sat, 09 Nov 2024 10:20:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlsK00zptz4fDm for ; Sat, 9 Nov 2024 10:20:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731147652; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BQ/oDp2VUUuH+59tLicoKtCBDb1ok8WFkc48deY6Aq8=; b=iQ50lFYYzsZbvBKXH0Ds2Rzl79WjxME81aT4LIQLesJUexCoPcXU1ojCj+0Cogunj2tKS6 +a9n0iXzJ5BgAJ41XT1NU9vblePxV5WOrToCc1p2w5eVkasIlZ0GWhKyKIjmxqS1zZ6qEH XgPNgI2dSSb4PJR/k60UqINYmhNV092gmrEdTphGH/AjspSbskngXBF2mxG798OjJM4Ii3 JQLwSs+CWLOsGx0I3bYcZpyg1Klzps+5uC2TdRANiJc48zuVheM5t64q1hzOwRoYEUBwf0 DUELhJxNFrdICO6db355Bpjqtv7IbOx7EqtMvcJUj9qYeupbu5IzPFTMZqO/5A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731147652; a=rsa-sha256; cv=none; b=RbH0vM/ss8xU5H067of+cRkrdCGfm7cUAoYs9oArdUXkRlo3apxdToCp2kagU8hTrouzO3 uhXPfQFHnyZHqydFKNQfOY1MQty+hKN3R9L6evz894I2lu/ggziIKTxBWzZ2MLDAapFvzg oa3v/9XEInEd7BiAxBSKgY3t/gb1qaFqWx9X95KxZ3Lkeezp39IDo2r1a6K/IGtcMxsZDl 4ms524kFfcc1ztch2UBvdBSvJ0pwg5fWgouziz72rXz8hT1WLN1IDN0mIXchSuT3ZT8shq +qcBDa6Cl3efcuTVWKa2yUo0fiMcfrAEudkrrQzAgKmDGhP1/kVLjJhjHr3A4Q== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XlsK00S6Mz18cK for ; Sat, 9 Nov 2024 10:20:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A9AKp4b051642 for ; Sat, 9 Nov 2024 10:20:51 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A9AKpWq051641 for fs@FreeBSD.org; Sat, 9 Nov 2024 10:20:51 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282622] zfs snapshot corruption when using encryption Date: Sat, 09 Nov 2024 10:20:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ft@hemlock.eden.le-fay.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622 --- Comment #5 from Lexi Winter --- > That's interesting. can you reproduce the error reliably? when ZFS encryption was enabled, syncoid (part of sanoid) would reproduce t= he problem every few days. more snapshots would gradually become inaccessible= and 'permanent errors' listed in zpool status would grow over time until the se= rver was rebooted. i've since disabled encryption on all filesystems (other than the pool's ro= ot filesystem) and the problem no longer occurs. > How much memory? 32GB. > Any zfs sysctls? vfs.zfs.min_auto_ashift=3D12 > vfs.zfs.arc.max? vfs.zfs.arc.max: 0 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Nov 9 12:00:55 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlvXR2PJGz5c3LG for ; Sat, 09 Nov 2024 12:00:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlvXR1L6fz4pFD for ; Sat, 9 Nov 2024 12:00:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731153655; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jXYNuLKTIyhfAjZHQarLqARqNK8F9bGVN1DdPh9Quv4=; b=fSeSmAlCXMeWHvaAtnKqp3LOJf49sX7DsRUgCckO+mFBi2KdHIkvNu8VwaJo3mU+karXQm JjjhXEXH1nPLz7gOreUCr3eXhgaD2fwLIHIy5ekXmZ1thKZUCPgKV95T36ya3LLHXAZF/2 e8JF5I5p2matnuA8VDH48GZnGi1FTy7Lxao5FCvRpBCPhhAe+aXseDx9vmSYuMrPKIB9VE VVtW8eRRDY+ToiHjgd6lD/tghzNUj7Sid+n5YgvM4iyQE/hv8nNgFzCaw4oHFVbFw9HBMr mp2qIhFXisJh+McuwExdRkCr6BdcWG2ajQFUq7izYv1YIv/zJ/f4EoZG/ORESg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731153655; a=rsa-sha256; cv=none; b=fR0f6qYMtINoUiKEB1Vy+kcvdZjhqooEygbvpTWUtczwfjo+cadZmw4IaRRkGVuARgtZIb mRuNJ76s724MtkDwgCnrmAL69XAEc+1TJJcKvJ12OyIqXuQYTE5O0n6uSWFztE6uv6a7AL qvITnx2DI6W9GTUDguzng3L9OdPvTIvnLXkezcdKM16PjafLs9kRFUy2LZCSCPyEX/Zt4e Sme9I5t2VfXFPvbcPh6KOzJkc8imODuWAoZCGoLXwEax7qKjU4WZVtcfrrs8cqypxk7nWi I1QlTV0Enoc9M6oyPhuBDerYHM4RPJmhOVcC9/n6UeToRHXEV0MF1zopU/lTIQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XlvXR0rDhz1ByQ for ; Sat, 9 Nov 2024 12:00:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A9C0smF005177 for ; Sat, 9 Nov 2024 12:00:54 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A9C0sOq005176 for fs@FreeBSD.org; Sat, 9 Nov 2024 12:00:54 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282622] zfs snapshot corruption when using encryption Date: Sat, 09 Nov 2024 12:00:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: girgen@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622 --- Comment #6 from Palle Girgensohn --- destroy the snapshot and run zpool scrub -e=20 is sufficent for me to get rid of the pool error. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Nov 9 12:03:18 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlvbB4yzxz5c3WC for ; Sat, 09 Nov 2024 12:03:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlvbB1tT6z4pR2 for ; Sat, 9 Nov 2024 12:03:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731153798; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g0pZQk2dTHC6pk1qrvQgQwVvFE79h90G4Oe4hEj1Soo=; b=heJClEMM2ufZTPLUbVmNnM1vseDU8JSA3EklTi4L5kXg6J/3t+4sE34ncL9h4A1rr76lhH 3X/LAApKpqJSpZjk71mWKXJWbKUGU0BkPFBdHy5FuqGSp5k7z/CK7ZnYJXIhlDIFlRELS5 xSD3MbYvCNjPSqTfeV2phoYlQfqgQmLCzA9xD6v9C0sDbklKCl3rnPFPDqlQLFvOdw03ji ODGTf/6vWbNlYnOkBQa/h2LLLE0ApU7bJMHyWi1a28Hrn5pKJ7u/Z9BNSY8vWxUbAsld3Z 3nKYLZrMiVgXmTPrFen5MaehsKmu5Ur0JnauS7K89gDNMODHWoklxNUv1f3qWw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731153798; a=rsa-sha256; cv=none; b=tbaEZwqf895WOJ4jlfsjnltUdRctFTq4KCuq1DaBy3i6rr1IeXRV6WHzYqDrLXnMxqzXJf SVBXuZWDwPGfD+7ULCnPxTpgdNJzXhBhwy59A3P6XGSwTrIipx72xXQgmfkeLUQia4nNR6 l8a/7sZlGJ/OgzTwfj4cLqWkVEaROc8pR4SPgxVYVctvy45Oqy57CVh0fMdjMedC6oLHe3 SYEy2X+ValgPYtfDo8FZMc0vn45GORVmwXDrT2y1CTI29cGN2sNyAne2CJBhIkQJSmsyQF h8jRaHlG1sxstjgeQsyPazpZTBi6VBJTe2hGSLRxkX896hW8Jnc4IJblv+sK8Q== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XlvbB1Q5yz1C03 for ; Sat, 9 Nov 2024 12:03:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A9C3Ii1015904 for ; Sat, 9 Nov 2024 12:03:18 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A9C3I37015903 for fs@FreeBSD.org; Sat, 9 Nov 2024 12:03:18 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282622] zfs snapshot corruption when using encryption Date: Sat, 09 Nov 2024 12:03:18 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: girgen@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622 --- Comment #7 from Palle Girgensohn --- Adding this parallel discussion https://lists.freebsd.org/archives/freebsd-fs/2024-November/003806.html Looking further at this, there seems to be many reports of the same sort of problem you're reporting: https://github.com/openzfs/openzfs-docs/issues/494 https://github.com/openzfs/zfs/issues/11679 https://github.com/openzfs/zfs/issues/16623 https://github.com/openzfs/zfs/issues/10019 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Nov 9 12:06:21 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xlvfy6h2Lz5c3rj for ; Sat, 09 Nov 2024 12:06:34 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xlvfy54Z5z4qpV; Sat, 9 Nov 2024 12:06:34 +0000 (UTC) (envelope-from girgen@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731153994; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g4acKkrxIx1aDYcgvsAwS/+AzAJdD2cAhG+q9xb9Hik=; b=D5BrpLN4kvETP/Y+zNjQtbzjD+yBiq0+d1rrmt3dlnD3UH2QwxROxwIDFOHNSwxNWCMnVa 9MQN0jc6MeS6hbE9TcGhSi5o1OyrUTxmw0ivoITuB7wiTHAbkzJQhXoz0cG9GezIqLAVCp wj6LWUhITzR2SJXhcY9unyaY21JIdVJKIdsDoIcXfMRexYpsRa5zZnGvBq7boaY8sF2eBM Bt+TpHxv5eno6A+ek16UcvoiCk5n5FSdHUrcMrNjwEuTcnOckRkXR/SpKUE/Ox+WW1eFRz fEcylglRMW9spNN33C79s2INfmcEGowcyWblsEN85lXDguITKMR2hwvjU8ZxuA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731153994; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g4acKkrxIx1aDYcgvsAwS/+AzAJdD2cAhG+q9xb9Hik=; b=shFBGrAy7/pwNHVZ5i8dqDCNl+83VfskKvCub6j/CaQ2ULacrq/1iLTTRFqkNkgZVm3+Dq lS/n1UUzjwY+SJxlVqeSttFiqi0b8lUJmeC+v/I308+KhTy6Z1v1iPWGuk0HxuvF9bikMM l0Y7ZTuAoBb3L83wCb+8WQXoVPgEtyZl2fnhGLWUjOE+fepTED0ufaM4WX9PcDVQ49BC/t KdgVa+yBAqEtDlipwzLD7EW93kgkEyHtYRdRCcfFpbL5yyCUBuRqlf0/yM1hvimM/YP6I/ SnuOIIq7KnTV2qphbHZsh6PSyQCwMyLQUfglZ5yUee6+XieZhrCKvq8sle4jlw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731153994; a=rsa-sha256; cv=none; b=CO5Q+IC0BPCvPA1rTs02lYiAQdkkX/9nRxTfiQXkx+BWXKfQ8H/6E44mqgeVG0ZfemAYZU dD93OkACXnvxF3fvTQlVvt6K0sATTmjh7v/UQLQ4iapM8CdLXctHvza5jZklWDa8CiWD/0 rCT8FMXB8JG3iJYMJStCeHze8T6N5VnqwsIdPegMcWLdDfMJjW1dcHEC28QlWEfe/Ch+bh c7e13Bw7Hwjj/rl/fc0PxQX3UnkTZc5Stn8n08S7JxziiKf7wH9F3BBIulnH4sWy2xVSrB MgeCzEb3lwA7Btig88F//qdsK3aA8iNHXkmaD/ySvw5hQv0vuqLR8TXC3yhzQA== Received: from smtpclient.apple (h-213-164-202-49.A498.priv.bahnhof.se [213.164.202.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: girgen/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Xlvfy27Z9z1Mwg; Sat, 9 Nov 2024 12:06:33 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Content-Type: text/plain; charset=us-ascii List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.200.121\)) Subject: Re: zfs snapshot corruption when using encryption From: Palle Girgensohn In-Reply-To: Date: Sat, 9 Nov 2024 13:06:21 +0100 Cc: freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <03E4CCF5-0F9A-4B0E-A9DA-81C7C677860C@FreeBSD.org> <3E85AAAE-8B1E-47C7-B581-E3D98AB03907@FreeBSD.org> To: void X-Mailer: Apple Mail (2.3826.200.121) > 9 nov. 2024 kl. 02:59 skrev void : >=20 > On Fri, Nov 08, 2024 at 11:11:23PM +0100, Palle Girgensohn wrote: >=20 >> $ freebsd-version -ku >> 14.0-RELEASE-p10 >> 14.0-RELEASE-p10 >>=20 >> dunno how to present in the above fashion? Binary install, not from = source. >=20 > % zfs version >=20 >> The latter, zfs per-filesystem zfs encryption. I haven't heard any = horror stories about it, albeit it's quite fresh code. >=20 > I'm wrong about native zfs encryption not being on 14.0. Seems like it = is. https://www.freebsd.org/releases/14.0R/relnotes/ >=20 > I'm reading this bit: >=20 > ### > ZFS Changes >=20 > OpenZFS has been upgraded to version 2.2. New features include: >=20 > block cloning, which allows shallow copies of blocks in file copies. = This is optional, and disabled by default; it can be enabled with sysctl = vfs.zfs.bclone_enabled=3D1. >=20 > scrub error log (zpool scrub -e) > BLAKE3 checksums, which are fast, and are now the recommended secure = checksums > corrective zfs receive can heal corrupted data vdev and zpool user = properties, similar to dataset user properties. > ### >=20 > Then again, 14.0 is EoL. >=20 >> Na, it's JBOD alright. I see nothing weird with the disks. >=20 >>> Reallocated_Sector_Ct >>> Reported_Uncorrect >>> Current_Pending_Sector >>> Offline_Uncorrectable >=20 > Were all these values '0' ? I don't see the props at all. >=20 > Using truss might give a clue about the failure: >=20 > # truss -o /tmp/truss.out zfs-send-commands >=20 > make sure the outfile is made in a place with lots of spare space. At present I don't have an error, had to remove the snapshot and run = zpool scrub -e, to get the zfs send propagation working. We see these = error popping up every other day, so I'll get a truss once we get the = next problematic snapshot. Box is updated to 14.1, FWIW. Palle From nobody Sat Nov 9 12:45:15 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlwWc2cGTz5c6DD for ; Sat, 09 Nov 2024 12:45:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlwWc1W0bz4spM for ; Sat, 9 Nov 2024 12:45:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731156316; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bW8bfBpiQJChDFTuNC4CJVr6I87R7Irvei13yQdSick=; b=XuHDRE4RkR3TCB+jeoz2ptW0wp0dCdnvyRTO6xr13w846SwIG4+n+Geekwh8ANmAdg8vlm iqbSJQAQX0Ou0/XgY0fQaC8zatLXiKyym8dK74yWZlRI5YyfKtmfpDarx/mfPMMci+VU1r G4FZUJQk5E43/O3dMb3vmpyxaZFMMZ57iJKYBp5uJtdNTBom7EdmNm65NShO6maqfKToXw Ij/sGLziCHLdo2BNCtt4/nqc2ByqPwH8aW1xI+5OgSog+hxbdv9tlyenHr5dIU/Ko539+/ BvVuQ8WPlooc1nc9Bh3eSXz61QdlNbJJwyNwp9QBZnSiQe/Ey0ltJNjukKwpPQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731156316; a=rsa-sha256; cv=none; b=aQQhXq4YrUBTA8+fLg8RqE9mYLxg52JoejaNc70FPa8T0i7TfQVrHa3C5xqxI3inTRwySp thk/m2cKJxBjyK/7nqe97K6O7cgNyPR8N+lNLkDOa7xViJs7jFEjQVymiRCcBj2sL1VnIy jrAypqiQ3in9/nKrm2ZqaTUQIrkiJDr+zFvQgZRlfPJ0K3R4NFSzEEUyVWKT2+5dVe7O1d J7A1sqZYWWd5Kd3XvGNHIX1/u829pbQ2zokLJ6ab/3tk9JQfikiH4PumLsmDOyOTw6ep5m 5V6LjMJwEUMV4FsAv3IFVBQ4RzrVFXhQtf2ND2z6f2DFg2g/Vnp5nRTVi4eIig== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XlwWc10PFz1DCJ for ; Sat, 9 Nov 2024 12:45:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A9CjGQP083941 for ; Sat, 9 Nov 2024 12:45:16 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A9CjGVc083940 for fs@FreeBSD.org; Sat, 9 Nov 2024 12:45:16 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282622] zfs snapshot corruption when using encryption Date: Sat, 09 Nov 2024 12:45:15 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: 000.fbsd@quip.cz X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622 Miroslav Lachman <000.fbsd@quip.cz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |000.fbsd@quip.cz --- Comment #8 from Miroslav Lachman <000.fbsd@quip.cz> --- Just a data point. I am running FreeBSD 13.3-RELEASE-p7 amd64 GENERIC with ZFS native encrypti= on: # zfs get encryption,encryptionroot tank0/vol0/remote_backup NAME PROPERTY VALUE SOURCE tank0/vol0/remote_backup encryption aes-256-gcm - tank0/vol0/remote_backup encryptionroot tank0/vol0/remote_backup - Under tank0/vol0/remote_backup there are 42 descendant encrypted filesystem= s. There are about 10TB of data - rsync backups of remote machines. Each day t= here are new snapshots for each encrypted FS. (1671 currently present) This machines is in production from 2023-03 without any issues described in this PR. Key is loaded manually by zfs load-key tank0/vol0/remote_backup --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Nov 9 12:47:06 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlwYk4S4Qz5c6MM for ; Sat, 09 Nov 2024 12:47:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlwYk3LQpz4tb2 for ; Sat, 9 Nov 2024 12:47:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731156426; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MFMOowqgMrIVf5wblg6mEPpFeJ9Vm/LU/pazZT98bMo=; b=OUyguEV/InLuHZG3tx04TX/W+x4facml4J2SOOYcnXC6KISxAxM4ShFtdqkJbSYk4pUwoh eVBMseEpuwqPNBRukjC3Kx35Iqvws0ChexZPEipZexrt1WUJNKtM1kpanrNWf3/1hue0p9 cjAGUsd0jh+kheREj/aUWem86ha3Ws7aDTUQClgcQ/MhfGAKga9/WcKgcxYEV0GFj4QtQF e4Y4aybhg2PnFeYBcfITyOxwnbLt1FXFeQn7J4ND/UEvmFMk4jTdIJH6AnBnNN9+VLGvsL mCTlFPv9B7kOw+shHEReCd4e3UDENLPRMzGFGHfWLEO7Ci+EG6nOFLSDysoo3Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731156426; a=rsa-sha256; cv=none; b=HfG8aNpeZH4qvcEw+fB12Ob1UHd+JoC6JkZnOM1v7k6W4r0RQGJCl1CNz1gtFoFsDFAM6T ngvTvxkymfb+jzMM7KVrQmvFAlarnb0Uct8vqqa25KRDgBLskQF/XYQHX5ffPG9zoIFMfK 3TLNpNAte1Y5LvUT6R9ROMcugvPrOidm4vZ7jDTWKf/3E1sjMYZEypODR2Riya1HLVZvWd qsjaaVHH2Xl96WXLvR0MDeBKGfJ6Tj/vMWKUzEw+BoAM3SISwtJ2anhUwuXzX2nvvRxzhW hszFQEe8vzD2gKUcCYztCO5czyOY6kN7lt4uwK9N+4c1fLGY6RDo2/LEWN976A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XlwYk2jzdz1CyN for ; Sat, 9 Nov 2024 12:47:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A9Cl6XK084997 for ; Sat, 9 Nov 2024 12:47:06 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A9Cl66J084996 for fs@FreeBSD.org; Sat, 9 Nov 2024 12:47:06 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282622] zfs snapshot corruption when using encryption Date: Sat, 09 Nov 2024 12:47:06 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ft@hemlock.eden.le-fay.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622 --- Comment #9 from Lexi Winter --- (In reply to Miroslav Lachman from comment #8) > This machines is in production from 2023-03 without any issues described = in this PR. do you send snapshots *from* this machine to another machine? this is what triggers the bug described in this PR. normal local filesystem access or receiving snapshots will not trigger the bug. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Nov 9 12:50:25 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlwdZ2P8Vz5c6FB for ; Sat, 09 Nov 2024 12:50:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlwdZ1J4Jz4tyQ for ; Sat, 9 Nov 2024 12:50:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731156626; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b3greYTS7t1YMeXDMhF+FgURXaGtBd+srAm1QcmmybM=; b=FLLmCoYaNronu5creTtgM9i8h3+U+HB/S3HIo2JksVs57YQRA6ueeTfK/phS0an4Ei0T7y nMAyzRk6DShJnX5kEHcjKEz2qSTSahsGQG0a1e76HmGldEOHG6y790ybH4ULWrUIj5Mv03 wK3f1o0jaDCTOSL0l3UPC6XIQDPUnsYYCsEA/6fXpC8C530vCZ5de1Nh5l64Rsw8dKHA7a ScN9jRH5p9PgyaMLFRfvVvUhMO2KRk6TLiqtdxmyTtANcWmtJvEDvcu6aTYpS+BuTpw5mN CiakdL4KffRQtmXIaQhUDa50bsUvF/w0XqSyX/NOvRkO4ryNbdTEY9dMdjOQIA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731156626; a=rsa-sha256; cv=none; b=iWOEU7I9GJqtDYIXSOlCnqJzmC9YiLygGGoeHEthNz/aW2AxFmOjBHwZxh7xZZuA5zEiNo nq/pe6x0DYSh4GzdhM4sMVRSjPYt9nWE1Bvvd2aiHcAHNSIORhJ2P9GAHJ8xh2bkg+FGCo em3F9Crk2aOBt9V0n1PjlkCgEuKPntFjsSMi+QOcWygKbmyWH0WnwVYowR94B3+MnWRpeX PjdblN+O53NZyo3MmTyxx2icmgvA2ss7jKXBJTIIroOm6XaXa37QVteqyuQcs2SAGrjJqf AT4uQLDoXBhcKK8h8bSmUa3iES1BshiWQg0oexllEF2jNt/S1oeqV9e48Z7UPA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XlwdZ0TH2z1CyR for ; Sat, 9 Nov 2024 12:50:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A9CoPxP087043 for ; Sat, 9 Nov 2024 12:50:25 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A9CoPRP087042 for fs@FreeBSD.org; Sat, 9 Nov 2024 12:50:25 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282622] zfs snapshot corruption when using encryption Date: Sat, 09 Nov 2024 12:50:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: girgen@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622 --- Comment #10 from Palle Girgensohn --- (In reply to Lexi Winter from comment #9) The error is triggered by zfs send -I fs@previous_snap fs@problematic_snap > /dev/null so yes, zfs send has problems. No files are corrupted, AFAICS. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Nov 9 12:52:33 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xlwh11M2Cz5c6LB for ; Sat, 09 Nov 2024 12:52:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xlwh10J6Yz4vWt for ; Sat, 9 Nov 2024 12:52:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731156753; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EzVNYUwF57MlLdvYmh6FG1wH9ep/aYu57fAFoREW9Xo=; b=nwgx6anWSP8cFGLxgSSqsxJbNbHw7+ElTaSuDv1iW8+AQ1z1QV7MeE7X7oYy2tMGLubMz7 GtAwMadScux3KjjPlIYcbBMjsLrpq4K01o3ZXK65FT1guLYmqZNzy6/JUa5Imx4d/eyhSb olqkrAIIt/b65/n+FdJayq7k9Xyrxz5uGENQACnZKHGpiZBecotBgZ8TB3cxob0YilTw3N UiqFINiIhx1GhGL+JyLkMS4pyMn6XTENIQBVs5j4s7YFfCgNPKm+6Cd58Ee3iOefam7VEU fh1Kvy3lmeaj0mrYUBnsPxN7SLp2gRk9jYLdA9Rixo24XC9lvKcdG46qfKG9hA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731156753; a=rsa-sha256; cv=none; b=FjGx3dNG63p55+4bGs59pCKunWP4lk0tAIgkYWYybMWkbU97kEW9ewAfOt2Sbdjktqp40g X//K/NOPs6UmdzykL8O4I0dmGC6sCCdkMBER3c9iuwJmn+r0M1YIqFDJz+O0TiwqFa8SKl LBUn+etdN/l0KibtcpN0a/NsgrfMCLZ5UMzygcBdGltxT0LYZ2Poa+jFp2Un00abOFCLG/ HS7Vpk/3bzyUZCLl9MyS8hefGYlK2Y8GbT/Vdujozdfa73qAMDPnFS4j4asBBhaIlciGtt LLf/yqqWrXkZOwQQnGi6DwtsM6T1DBmv80VjznJBazt141TBbGpDXrX1IZiaGg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Xlwh06zzbz1Dxs for ; Sat, 9 Nov 2024 12:52:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A9CqWTl095911 for ; Sat, 9 Nov 2024 12:52:32 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A9CqWPY095910 for fs@FreeBSD.org; Sat, 9 Nov 2024 12:52:32 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282622] zfs snapshot corruption when using encryption Date: Sat, 09 Nov 2024 12:52:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ft@hemlock.eden.le-fay.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622 --- Comment #11 from Lexi Winter --- (In reply to Palle Girgensohn from comment #10) > No files are corrupted, AFAICS. this is also my experience (i.e., the errors vanish on reboot), so things c= ould be worse.=20 however, it's not really ideal to have to reboot a production system every = few days :-) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Nov 9 14:24:08 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xlyjj0byXz5cDMb for ; Sat, 09 Nov 2024 14:24:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xlyjh248kz44yZ for ; Sat, 9 Nov 2024 14:24:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731162248; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WAg9+pyHaFjTizDJa8KNjRQLMciCjPmV2Mhk7aZyau4=; b=Xr0spnwhbVh7m4VIfGObNUOeQTuLB4J/dkgXTPuHZCxpCw/eizKXC3xulQgyvoltgNefbU maAl0UlnSIBu1b8AVGaAvdJt+qCiF5TjVEngmP8aRNWDyXOZ/UJm2m5yPjJ3HzaorJJafj BBV9AbIVQ2Eunokmmq03VNCvmT7Cu48/5iVkXidAdnhn98d6y28Kiei2lfOahL1LiK7+k6 f4BcpkXPBCc741nv49Uar6j9piqGCBXEjlIbJa8agJbIWHeto6HA2mz8O4+xZlfoXo4Fco eNiwx+N2/LAs7inm2Lhm4Yp/u1FaPlTYju2SVwj+mouMve15u3MDNyCGfk6aUQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731162248; a=rsa-sha256; cv=none; b=jQOipCOEhTQRxlq9xmC/xO73orT1GwNLGVQeYU3HPT78WxGEFdZs5wIsFlRh393nDp8cvq gDBpB4z2xtJKGi1DZ8LnQqCECs7RYm2SpCLjdWO/ovSeHdCISRqK4VYqXbQNCe6Td+nCMg bsrG9VtI5JVWDzVYAxMUTiv8g4sDfHpL3y/oWaETzarqbq6G6wSCiNBffCwSehUDN34AL5 ARhMXzjitYAv7+c6MgBc9S7qH6aJ8MjL2qFGA5gux+Ulq1I9R+uE4FwekDjIY0IirYIL15 dEmDeNXLsljW701WvIqRlZ6TAZXU/RM8vbjJMU/jrIU5DgabQBB0p65wYJ17ow== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Xlyjh1ZSCz1H6q for ; Sat, 9 Nov 2024 14:24:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4A9EO8UY042486 for ; Sat, 9 Nov 2024 14:24:08 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4A9EO869042485 for fs@FreeBSD.org; Sat, 9 Nov 2024 14:24:08 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282622] zfs snapshot corruption when using encryption Date: Sat, 09 Nov 2024 14:24:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: girgen@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622 --- Comment #12 from Palle Girgensohn --- (In reply to Lexi Winter from comment #11) zfs destroy snapshot zpool scrub -e tank is sufficient for our case. I tried rebooting (acutally failing over to the other node, since we use dual channel disks and disk boxes). The snapshot w= as still broken. scrub -e fixed it though. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Nov 9 14:43:55 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xlz8q57qBz5cFXj for ; Sat, 09 Nov 2024 14:44:11 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-yb1-xb29.google.com (mail-yb1-xb29.google.com [IPv6:2607:f8b0:4864:20::b29]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xlz8p5wMDz46CR for ; Sat, 9 Nov 2024 14:44:10 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cedro.info header.s=google header.b=ZLb51DRW; spf=none (mx1.freebsd.org: domain of tomek@cedro.info has no SPF policy when checking 2607:f8b0:4864:20::b29) smtp.mailfrom=tomek@cedro.info; dmarc=none Received: by mail-yb1-xb29.google.com with SMTP id 3f1490d57ef6-e30eca40dedso3108334276.3 for ; Sat, 09 Nov 2024 06:44:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1731163450; x=1731768250; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=0pDQ3Mn7uSJciDfWor4vo5dIaq8UPaSuCMEVUmwBsjQ=; b=ZLb51DRWTSmG5EQrZSb3P8sZ9qTES64EpzBYVrJTu2cpQNatfOhEkzoyfcAPFO9Paf tp56NQ5yrz/Byji1QQmOyuJkuQ0XTPnvyC6NZvUikYCAoATmwuIRS19vO8+82AvVqBOt Rsb7vN5ab6KxBm4iecQ9OOhfeLuStQ4K6kWicO6jhvdJyNbWFuuqjEX8z32K9/DtnUvS Dm1eMX2Jg0OnuXY3uml1XjUGPFcttrRodrWt0NvUNUHnrLhdfDR2B/66YPj2zFOg48E3 6zuOQPYdjX3NuUumBjd6xXecIi5U1+YrJcISwMWsXZXIQZTmrbeBGe/CTH5tlUBOFSNo refQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731163450; x=1731768250; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0pDQ3Mn7uSJciDfWor4vo5dIaq8UPaSuCMEVUmwBsjQ=; b=wzX/QdVSuhEeIWFa40wUzXfAk0w1d21uDff2msaIbCu/vsUGowJxNI9kyD8H+LuuNg E9uQKXqBZemPb0l6YrWDtIRdpuOFNzkwKhpPrOAG9nP5vUsqzyKim1v9cBwJIzTrPnGD xoUhojRc/PCNTxG2XX+n0nohLfYppLhuDShk5IZYAwYxBYaArxe0gyJMoo3vIHD3iUio TZ+W7drLbmaJ87MbltI2yRdcwtYMa+9h1AjbZ3nBcKTssGLKczpYIs0ZxKkfiHmN1AcO GgnBOz6A8e6LgRzFmIVlBsMXBEJ+ENpp63hve0OFrRrliBoGS8q66nB3eFPHgQ3wftpE X2JQ== X-Forwarded-Encrypted: i=1; AJvYcCXl1I27t8WNaRn6y0GNEFvqoE4CHz/ZQzMwg9gk8oOkKwFofSaNkEprQajezqEMpTSntc2vV6U7DH3h@freebsd.org X-Gm-Message-State: AOJu0Yznzh4wb9Lk5e7/ZM9gFUh94VVvrzED//GxZbRXfJXaI6IfBTG3 H3acaef3pUNGPBQWSwqV4Xhjo8LSydVJPR6MjpEnwL6FdQaKutJ4wp6Gz04JWQ== X-Google-Smtp-Source: AGHT+IEk11WG2zYb3+C3vJMRZnFPZ2coIXn6v52PQdR6Tws5czPsfQ7bqPBKTGT7iI88IzeUcadJrw== X-Received: by 2002:a05:6902:2186:b0:e2b:9967:6352 with SMTP id 3f1490d57ef6-e337f85f106mr6489062276.20.1731163449817; Sat, 09 Nov 2024 06:44:09 -0800 (PST) Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com. [209.85.128.175]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e336ef20282sm1205011276.14.2024.11.09.06.44.07 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 09 Nov 2024 06:44:08 -0800 (PST) Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-6e377e4aea3so26201917b3.3; Sat, 09 Nov 2024 06:44:07 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCX3r+K4ShiJXLT5GGoxY+Utw/H2t4Vnef7WtHJGu6nF8odrhNi6a6sBi4wyVZl3m6X8Xf8GnTxeq0nt@freebsd.org X-Received: by 2002:a05:690c:c93:b0:6ea:7b00:4aa5 with SMTP id 00721157ae682-6eaddf81a01mr74864447b3.33.1731163447001; Sat, 09 Nov 2024 06:44:07 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <3293802b-3785-4715-8a6b-0802afb6f908@app.fastmail.com> <620f4e82-aa69-4af0-bd22-b21203ab8745@app.fastmail.com> In-Reply-To: From: Tomek CEDRO Date: Sat, 9 Nov 2024 15:43:55 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: nvme device errors & zfs To: Dave Cottlehuber Cc: Warner Losh , freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_DKIM_ALLOW(-0.20)[cedro.info:s=google]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[cedro.info]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[cedro.info:+]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; R_SPF_NA(0.00)[no SPF record]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b29:from,209.85.128.175:received] X-Rspamd-Queue-Id: 4Xlz8p5wMDz46CR X-Spamd-Bar: --- On Tue, Nov 5, 2024 at 1:47=E2=80=AFPM Tomek CEDRO wrote= : > On Tue, Nov 5, 2024 at 12:20=E2=80=AFPM Dave Cottlehuber wrote: > > On Tue, 5 Nov 2024, at 11:10, Tomek CEDRO wrote: > > > Magician software can upgrade firmware and perform other checks, work= s > > > on Windoze macOS and Android: > > that will be difficult, I don't have an nvme capable thing for any > > of those. > Look here :-) > https://download.semiconductor.samsung.com/resources/software-resources/S= amsung_SSD_990_PRO_4B2QJXD7.iso Dave did you solve the issues with firmware upgrade? I am waiting for black promotions to get one and I saw on the forums and the reviews there was initial batch with faulty firmware that caused quick data degradation so this will be most likely also cause of your problems, you definitely need to firmware upgrade :-) --=20 CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From nobody Sat Nov 9 14:45:38 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlzBn5zlMz5cF9m for ; Sat, 09 Nov 2024 14:45:53 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-yb1-xb31.google.com (mail-yb1-xb31.google.com [IPv6:2607:f8b0:4864:20::b31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlzBn1Ny0z46yj for ; Sat, 9 Nov 2024 14:45:53 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cedro.info header.s=google header.b=idvEuyOj; spf=none (mx1.freebsd.org: domain of tomek@cedro.info has no SPF policy when checking 2607:f8b0:4864:20::b31) smtp.mailfrom=tomek@cedro.info; dmarc=none Received: by mail-yb1-xb31.google.com with SMTP id 3f1490d57ef6-e2e340218daso3270067276.0 for ; Sat, 09 Nov 2024 06:45:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1731163552; x=1731768352; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=l0Gp4Vp8Cv6/zjlqv9JA8NbfY+qlBej/T0GEzjgaUFE=; b=idvEuyOj8GpOsnFCuehHOUjjBQHRunJUBecdLckyjMDUoOqHo+w1k6G19RuXMNCW9O ANyK9g1kHyoNN9MidQRONL3NwuJCrn04/xhOp6UpukwiqdfK4egeAyPuEjrOX4G75quG FAC8b9MyQh9Bezx8Mnf94utXHgq3L1DRyp0typMzXAk15mHr2adQd5MSpbThQQsqEBDd GDiLMnx+3fvtUiZxlCKS6Pc9IbFJHgj1XpXMqc9OTPK00mVOqEViFzLT1cpTZP0u+0Jb hQ+p5UivFyg5DgSs3qeLhiCxS/5+yRtIkKpe4iR84faR7AzdzJxh8Xtj1JYVEWdiqURW LKsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731163552; x=1731768352; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l0Gp4Vp8Cv6/zjlqv9JA8NbfY+qlBej/T0GEzjgaUFE=; b=d5U1eoAQbxuz6Xpudc057XSsaddHNrgGxo21+3BD4I/e/IQLM/P+47gtBQbAY+JmDb ja+Kpauym7AcJRJBxnUpo5p00+iX8V5QIdmZroWldvd9RR9AskNoIZPrwIv4z7fxX3zB TbHs0QxTbb+sEsraSbrZyp3E+LJcvthXwGO3yLFC/WWSXKyCuglwKVsnnsIFI4EKB8PS mi4DM2ggbeHp6BcHsHtJn/vDfhNCE6e2u2VPXTOlWfWUX7AaEAbk7NCTOvKID6LMlteN C7aQEiDJf2xEq0/X9z0R0BTSZ51aA/Ep3dTryeEnMHYJvHY4A7USHCq/ZfOv2eSaDRM+ yfFA== X-Forwarded-Encrypted: i=1; AJvYcCWaw5wj2G6tjQxwXANz3m7GrMgj27EaKTdZnbNby3N1OxBbJutwwGDaNcEbXedUgC6FvCtBcUp2aTQU@freebsd.org X-Gm-Message-State: AOJu0YydZ6+mohRIS+x9x6hhitpWaUh7K/RNPCbVHusBmPt5Q1A8zL4f PWHO7C30jygC+6rVcZuNHYmx9kft9mSXotH5t/AD+r9ZEeWq0YSBqLWm10v2XA== X-Google-Smtp-Source: AGHT+IHWFlmY5dMl1EWPPFfh6Vd2WrCCFRXovh76+ub/rv3TIsL+PQsYSm6W9lJFmvNLa6S+6vMlRg== X-Received: by 2002:a05:6902:2d07:b0:e30:e39b:9d6f with SMTP id 3f1490d57ef6-e337f843d2fmr7365267276.8.1731163552324; Sat, 09 Nov 2024 06:45:52 -0800 (PST) Received: from mail-yw1-f174.google.com (mail-yw1-f174.google.com. [209.85.128.174]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e336ef20282sm1205762276.14.2024.11.09.06.45.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 09 Nov 2024 06:45:51 -0800 (PST) Received: by mail-yw1-f174.google.com with SMTP id 00721157ae682-6ea85f7f445so32921447b3.0; Sat, 09 Nov 2024 06:45:50 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCWJ+EwuTcB5lVWnQb6qWLJfeffElF3LvaZWZWesi+7tPhtIa7FRibBs6Q8rOU/UCP9tANKwyKSJmGOt@freebsd.org X-Received: by 2002:a05:690c:7343:b0:64b:5cc7:bcbc with SMTP id 00721157ae682-6eadde50e75mr63319577b3.32.1731163549644; Sat, 09 Nov 2024 06:45:49 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <3293802b-3785-4715-8a6b-0802afb6f908@app.fastmail.com> <620f4e82-aa69-4af0-bd22-b21203ab8745@app.fastmail.com> In-Reply-To: From: Tomek CEDRO Date: Sat, 9 Nov 2024 15:45:38 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: nvme device errors & zfs To: Dave Cottlehuber Cc: Warner Losh , freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_DKIM_ALLOW(-0.20)[cedro.info:s=google]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_SPF_NA(0.00)[no SPF record]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MIME_TRACE(0.00)[0:+]; MISSING_XM_UA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b31:from]; DMARC_NA(0.00)[cedro.info]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[cedro.info:+] X-Rspamd-Queue-Id: 4XlzBn1Ny0z46yj X-Spamd-Bar: --- On Sat, Nov 9, 2024 at 3:43=E2=80=AFPM Tomek CEDRO wrote= : > On Tue, Nov 5, 2024 at 1:47=E2=80=AFPM Tomek CEDRO wro= te: > > On Tue, Nov 5, 2024 at 12:20=E2=80=AFPM Dave Cottlehuber wrote: > > > On Tue, 5 Nov 2024, at 11:10, Tomek CEDRO wrote: > > > > Magician software can upgrade firmware and perform other checks, wo= rks > > > > on Windoze macOS and Android: > > > that will be difficult, I don't have an nvme capable thing for any > > > of those. > > Look here :-) > > https://download.semiconductor.samsung.com/resources/software-resources= /Samsung_SSD_990_PRO_4B2QJXD7.iso > > Dave did you solve the issues with firmware upgrade? I am waiting for > black promotions to get one and I saw on the forums and the reviews > there was initial batch with faulty firmware that caused quick data > degradation so this will be most likely also cause of your problems, > you definitely need to firmware upgrade :-) Ah and the folks mention it will live one year only on the old firmware, so if you can replace them I would also suggest that, and then firmware upgrade new ones right after unboxing :-) --=20 CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From nobody Sat Nov 9 14:53:25 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlzMl43T1z5cFb4 for ; Sat, 09 Nov 2024 14:53:39 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlzMl36QXz47Hq; Sat, 9 Nov 2024 14:53:39 +0000 (UTC) (envelope-from girgen@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731164019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=wt/6ETNb4hiY98j8e4q5PNXg7SSOUPHrPGIEIIO6nL8=; b=H/YKD8yux6YOx2SbXB3bwe7iZH89NsvBfIHYXX0pYfq55zqs+fn4pzxsuo8aJucClAAxu+ TMh0C+CGOKi/Y7zsiI0Q5MA45+Oex3fzUNLCpvS/NQekWq0ksvQyjoO2+Q5ab2el6yjFKI 1zseehTR7kc+5r5KJX1bW04qXSzk9xj02HI+mAuQjlHsBflDxKP2oWpHe08TOyx0HYCGiu VxCPIzIBx1aHVszOoS5si8nH6lW3hiu+4hBcQTK6CzxnrEDMpEd2TMKyMYlivmH9JrNS1+ +QrboJY74c2R1zlmqQ/zOViDmyq3BixKwpiQro9IoY3efQvVyfsUAAUAuabW/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731164019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=wt/6ETNb4hiY98j8e4q5PNXg7SSOUPHrPGIEIIO6nL8=; b=gqziXZuBzI+piFAb5DUJjJVG+xI7uTcxLKt941sLP1n+yy09UzRxEfaL0IaBh1nbySUhUf Tx7PjlosLQ9CjNnTgoM76DvNyslXV5BJ5yklSVbna2qsbnFBXpB0OQz6CGgYLJPeIMzP23 siw4jZikzLY7EM14ToF1j6qX6znAdZjNLgUe43KWHGN5g4r0Pau/Pft+IzHARAQOlhGt9F g+eTv20MnUzWDsYmWKK7kpQF2r5sPXmIM2MHrfijUCfENqg2g/S+K7JAbPBkybWKj2Hwdj 0QXASpET9z08AQ9biqwEQBXHEQgnkl3p62B4OyNtnHpUs7rISy7++DJRpEVA1Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731164019; a=rsa-sha256; cv=none; b=jCJt7ajcSfwYl0yWLkbgDaSvqbX6FuqjW0IUwLEhw8USAzhOTtCyRMrKUPfUWlcLHUUXIy IvY8BDkjb2Fjx7P3m+aj+b4MAyge7qk94XyYNHcZafe+1el/mtYgN0ATcSVw/D5ZXZCuVJ adqjf9lJ1QEeDUmGPTbxd5R3v3bNrJ9CQyEXmBLKmx3Rw/c9v4hGeq5UXEgXlitZGtPi1H hzx3ewQtRu/afV4UhQHjWuZPNHRFseNi3jRshUCZCE7yTlCqLDooLwb1i9wfefPGtmy6cm Jfsw6KC3sUZn0puxuH8Jibo8yKWuc2dWQvgOKT2vu+pdrByLUed4nR1JdF/Ydw== Received: from smtpclient.apple (h-213-164-202-49.A498.priv.bahnhof.se [213.164.202.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: girgen/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4XlzMl0YXwz1Qjf; Sat, 9 Nov 2024 14:53:38 +0000 (UTC) (envelope-from girgen@FreeBSD.org) From: Palle Girgensohn Message-Id: <6F51D3D6-D9A3-46D8-94F2-535262F43EF4@FreeBSD.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_DEC8DB13-34BA-4D65-AD39-965F9DA2C2CE" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.200.121\)) Subject: Re: zfs snapshot corruption when using encryption Date: Sat, 9 Nov 2024 15:53:25 +0100 In-Reply-To: Cc: freebsd-fs@freebsd.org To: void References: <03E4CCF5-0F9A-4B0E-A9DA-81C7C677860C@FreeBSD.org> <3E85AAAE-8B1E-47C7-B581-E3D98AB03907@FreeBSD.org> X-Mailer: Apple Mail (2.3826.200.121) --Apple-Mail=_DEC8DB13-34BA-4D65-AD39-965F9DA2C2CE Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii > 9 nov. 2024 kl. 02:59 skrev void : > > % zfs version Ah, of course. $ zfs version zfs-2.2.4-FreeBSD_g256659204 zfs-kmod-2.2.4-FreeBSD_g256659204 Palle --Apple-Mail=_DEC8DB13-34BA-4D65-AD39-965F9DA2C2CE Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

9 nov. 2024 kl. 02:59 skrev void = <void@f-m.fm>:

% zfs = version

Ah, of = course.

$ zfs = version 
zfs-2.2.4-FreeBSD_g256659204
zfs-kmod-2.= 2.4-FreeBSD_g256659204

Palle

=
= --Apple-Mail=_DEC8DB13-34BA-4D65-AD39-965F9DA2C2CE-- From nobody Sun Nov 10 07:05:23 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XmNx10RVyz5cMTG for ; Sun, 10 Nov 2024 07:05:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XmNx03mYDz4QqK for ; Sun, 10 Nov 2024 07:05:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731222324; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZwV/DTtiRIkkcn4Gzje/T5UQiykr9x/eawbYkYgLz5E=; b=RhOGi5r7Tv5MBISa5oErTfSLNrrJs+mFzm/snSB7gODOYEzbCFWXSdh7nj1eLBg84qFsih LqYkyKViUhG3+ol0jD2Nx8cyC5ctnJxy5kYqWfqui6ZL7aBzhZdbqUfwktGYf8dW0c4Jj/ HMB3JqigIbUJ/4LFKULs4AiE2wBk22FjC1ELURQOJFS+550uHVHjxGR3EYmEYbeFVvocJb AiJsEl/cK9tPgfvxzl7YA+WhWzexVFNtx7B5QDrvWUTIUrnFJLpXYnw0EXU4YBJRzK23/g AU6mNEwcTKTmAm9eEMemS9YLJV2Pf4xoQw6pKLQo0p7bR4Yp0Ab6ppLS8VZ+EA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731222324; a=rsa-sha256; cv=none; b=IeYCPkBGD2Znmde7J640jh7rdtBphdGrutCUsoHGhBRHrjOro8Jm44vcGc80swly+Lm3zM /RX4A1gf/iMzX3DIjPh3KnVTTtR3xp6CGPt1G+YpBRX01ZRu253poCatU0h6q9KM3zo6OK tK7PaiH5i3ijj/FhPT2JIIJHTi4pgZ+NLtTvBifxpJ5OzaT8pA+PIGU+Zv5xz3lhsIJ+U6 JW4WikTWlBakgUJiiZxRd9j/zt9v/nzrDVCPfbnrK+1cdRI3a52WDh6vCcyY4vhopB6F+r aNTre9dEz8rxBlvA3cqIB1/rep4zS5vM9W88E42egU7FRuVP426z9x0ZraXuNQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XmNx03NxczYGV for ; Sun, 10 Nov 2024 07:05:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4AA75Oh7030302 for ; Sun, 10 Nov 2024 07:05:24 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4AA75OZj030301 for fs@FreeBSD.org; Sun, 10 Nov 2024 07:05:24 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282622] zfs snapshot corruption when using encryption Date: Sun, 10 Nov 2024 07:05:23 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: 000.fbsd@quip.cz X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282622 --- Comment #13 from Miroslav Lachman <000.fbsd@quip.cz> --- (In reply to Lexi Winter from comment #9) No, not sending snapshots to other machine. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Nov 10 21:00:28 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XmlSX46Ptz5cGTk for ; Sun, 10 Nov 2024 21:00:28 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XmlSX1dB3z4btd for ; Sun, 10 Nov 2024 21:00:28 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731272428; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=62ISZTqpFj/BeMS1ck7RxJPnecXZyWOdi9+XCfoHXlY=; b=H1Q4b2a3S8a32ffqsVKyotFG8dnzyvju4JQ18rW2BnWcepBVoe88f1QsK+IK8l6TbE4wz7 sG/P4IPkz/eKa5o4aITZwe5/kc4JZ23BR6pAKMUOeaLoz8dvnTUN391Ehwt19rq1NoHbXN Mkf3D4YcgfJRW1Jf3fv0y4UQSUebIC5ZcIvqWSisBmq4bhpDUETV2Q2Frv2CWOsPFY6yix exdZhUg3g8WRxaj0DdNswQxbaJ6KklcpGibuGWYc5mP+f2bzFtBEFp0WG7ni6jQ41ZP4iq F46kI0uf0LQE8MCsx+O3oCnz5DQKMPqkeiu4FVB4k1sfI6KFhLOVrAu9xrtMIw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731272428; a=rsa-sha256; cv=none; b=vqPpGtapXksq6tYvPLd6KQs2kA9yKpJq5K/D1+81kFpNULVjQqNpSzux9fslz5eexMhyLI YdmJCi4+Kpd5StSo4iWjfXb73ohAG9XJIOEJEPlgSXh4/CVQIbthxL8jeeOmxwZIpxw2aC Ud+h7tdg9c4B8lUUAoE+LkwZK1Wo0AC9txe0gpaHvZy7rTZQreKjM+0SPDnLBxZCWqzKGU PyJnW79mawXVad5BjxxEWewsNLcLfNJYda1tkT0DCZnC4v4hjuzzWoqQMxTTXpNRgr8TxF ezFnlX6u5SbUI2CFqZE1aNAsLlEDgBYlyJevipiqSwhQR2sY0vAk9HAcKUmMiA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XmlSX1FJrzxQF for ; Sun, 10 Nov 2024 21:00:28 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4AAL0SMC038000 for ; Sun, 10 Nov 2024 21:00:28 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4AAL0Sop037999 for fs@FreeBSD.org; Sun, 10 Nov 2024 21:00:28 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202411102100.4AAL0Sop037999@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 10 Nov 2024 21:00:28 +0000 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17312724280.a5e1Be4A9.35304" Content-Transfer-Encoding: 7bit --17312724280.a5e1Be4A9.35304 Date: Sun, 10 Nov 2024 21:00:28 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit Open | 271384 | zfs_load is not suitably documented Open | 226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 = 3 problems total for which you should take action. --17312724280.a5e1Be4A9.35304 Date: Sun, 10 Nov 2024 21:00:28 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit
Open        |    271384 | zfs_load is not suitably documented
Open        |    226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 =

3 problems total for which you should take action.
--17312724280.a5e1Be4A9.35304-- From nobody Sun Nov 10 22:35:27 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XmnZ75lYcz5cNC2 for ; Sun, 10 Nov 2024 22:35:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XmnZ735vlz4nG8 for ; Sun, 10 Nov 2024 22:35:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731278127; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BEz7mUuX403uc4awzp2VoK6iIvjJnQTVl0gsBwtWRgo=; b=iYKLKFjL1qB7Vkx7SF+y8zAo74jEWOky7cr7+uzaLgNan5e7VKeuDwSwZUpEEtNzuWTJ9A eAl/Yg0/W4UD7ksiNKp93QnccZEBiZeF+49RdgceGw54fuvWXzk6KFVxEw8QOf2OCpfByb YvibIlHKi8F26HvdxAmQYi986Lco+nsU1rtF8SOxukuNlOkKDY+wzIpFzyvvT/58Q6rl1N pxvdGCTqpK6rs46ZXY1jYsEX0ugpiQZHCzRPwqXzbw+yhO0HsdrQMOJ4oNOTxZMjd1U+Pc cPaaCmT+KQDOEqt5+tA4o2cuu3AmermPBpO64ois1X0GrdlVyz7tNe851dsOcw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731278127; a=rsa-sha256; cv=none; b=YZi031GGU9Tt1cOpCYY2RIhJFKz+FWi8kkV/dePR5o6t9fF8evVQHRNtWnCHmbhUcYeSGw rHuKR/D7vIea5iNmL+j5fRiEICyxYtd/sXHIk5cxE4TpQ2vRYckZtEDab7iDt/DyDV5cmh StTLMq8wLuZ5LX/52b/JKuzXkPP1zEBoxooRy4IwHXzHwsAeZTlvlDm5X2Ttyu7dJwlmu2 1+g49pgB+U25ENOfu+lMFreEwmQFGKNK9Cb0g4SEKRlqXszM4MbT9I7CdfwVmu3rur6SNm ln/bpR6JwHq3vYkNfygAhRE+o+ZauNem5lnYNs/svud52lH0k++dzkhGMrgvdw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XmnZ72hTVz11RX for ; Sun, 10 Nov 2024 22:35:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4AAMZRlw097356 for ; Sun, 10 Nov 2024 22:35:27 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4AAMZRFW097355 for fs@FreeBSD.org; Sun, 10 Nov 2024 22:35:27 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282449] UFS deadlock with install -S during freebsd-update Date: Sun, 10 Nov 2024 22:35:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: Unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282449 --- Comment #15 from Mark Johnston --- Looking at this again, I think the problem isn't with the general runningbufspace mechanism. There's a specific code path in the SU+J implementation where we can deadlock when hirunningspace is small. What happens is, we prepare to write a data buf, claim space for it in the runningbuf total, then call bufstrategy(), which in turn might dispatch jou= rnal I/O asynchronously, which can block on runningbufspace. Normally this happ= ens in the context of the softdep flusher, which is exempt from the runningbufs= pace limit, but it can happen in a user thread as well: _sleep+0x1f0 waitrunningbufspace+0x76 bufwrite+0x24a softdep_process_journal+0x728 softdep_disk_io_initiation+0x79b ffs_geom_strategy+0x1f0 ufs_strategy+0x83 bufstrategy+0x36 bufwrite+0x1da cluster_wbuild+0x722 cluster_write+0x12f ffs_write+0x41d When hirunningspace =3D=3D maxphys =3D=3D runningbufspace, this recursive b= ufwrite() call will block forever. I suspect this bawrite() call in softdep_process_journal() should be gated = on (curthread->td_pflags & TDP_NORUNNINGSPACE) !=3D 0. The softdep flusher wi= ll continue to issue async writes, but if a user thread is forced to flush jou= rnal records in order to initiate I/O, it'll do so synchronously. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Nov 10 23:21:09 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XmpZs67Prz5cQTk for ; Sun, 10 Nov 2024 23:21:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XmpZs50zVz4rJy for ; Sun, 10 Nov 2024 23:21:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731280869; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4R5gt7SrCJpIL19cA+d7yafOaCl++NkCFSjdPpG/jHE=; b=uq8nEOKRMCztws/jxNbPrLri3kFiM0/HYJYLlpDatbSGA0ZQO1vig0NJwSAzXxuojCBE5w DrKPrMu/T26X5NFXENxOxfu4Z622eF4YFsu4EquFd8rE7YMJfkhcL+a8H831hMCGaGUSNc E7Ry10gra4kPBRhTHTQyXSE4yScdV643oohu1jHh3c1ygT/PtwY+HdxHncOxCWTw/99cdM lokO2KQXk4JnNHV4SBvqxU59qh5I+ZI7xeV9PVY96Y9CkVEmXbj2Pf2qs9+uSFAsGSStvs /KR4jfYB16noKAuyPE1jniy5/K833zJUh5OBzwDXp806VgKPNJrxFYBr6hg22Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731280869; a=rsa-sha256; cv=none; b=pxc+Iiizpgc9GgyXNV8GC649V0Ud68EBxVj13pR6RDzKyLzFVoqRchrhh8LDEb0bL45mcA nl4fI+yHBTXlDxy5Kxa6IpefZsE1CbAB1dAhTCuvgjZJlwiAl2ZVDeQ9ESxCbcQsL7iClF 3D2Vz/FOoL4ZZtDmfUnxK8p6F5hF2cvjNoOVRhmz115HPTKjXRy84JswpSHA10IwQvnvlt dCmmhdNpcoFaTjpqxVgt5QPOYGLDCd/a+5C3mV8MO1edMziyPcHmjs6zLKYsRee3y3XnUR hx4jF4vmcXGVe/NBYqkN0B+VToNg750UIw6Gkv+QShww1k2bv0U89y7+P1okYQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XmpZs4cw2z11mW for ; Sun, 10 Nov 2024 23:21:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4AANL9Xe063895 for ; Sun, 10 Nov 2024 23:21:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4AANL9f0063894 for fs@FreeBSD.org; Sun, 10 Nov 2024 23:21:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 282449] UFS deadlock with install -S during freebsd-update Date: Sun, 10 Nov 2024 23:21:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: Unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282449 Konstantin Belousov changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kib@FreeBSD.org --- Comment #16 from Konstantin Belousov --- (In reply to Mark Johnston from comment #15) This is worth a try. But I suggest to try a more invasive attempt then: convert all async writes into sync if there is not enough running space and the thr= ead is not exempt from the running space accounting. --=20 You are receiving this mail because: You are the assignee for the bug.=