From nobody Thu Jun 8 05:11:48 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcC4S4w6Xz4bXbD for ; Thu, 8 Jun 2023 05:11:52 +0000 (UTC) (envelope-from rebecca@bsdio.com) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcC4R5dqfz4GxJ for ; Thu, 8 Jun 2023 05:11:51 +0000 (UTC) (envelope-from rebecca@bsdio.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdio.com header.s=fm2 header.b="O0O/qNDq"; dkim=pass header.d=messagingengine.com header.s=fm1 header.b="r99dEBh/"; spf=pass (mx1.freebsd.org: domain of rebecca@bsdio.com designates 66.111.4.26 as permitted sender) smtp.mailfrom=rebecca@bsdio.com; dmarc=none Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id B8A8D5C00AB for ; Thu, 8 Jun 2023 01:11:50 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Thu, 08 Jun 2023 01:11:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdio.com; h=cc :content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:message-id:mime-version:reply-to:sender :subject:subject:to:to; s=fm2; t=1686201110; x=1686287510; bh=oD 6xldeLN7QfscQFB0oruziGRqB0Q4sqN6CZa+2e0jo=; b=O0O/qNDqKLdpKSj4QS /0E06Znml3sBk1epySK0jLHyjUhvwbqGh5WgbquNh+DTUHMSmrOAjzIUuPCVk6OY 268q8garYwBzJhYWDPmxJXDWtLxDDxXLMhK/SpLcTVIqdNZieUF2IKUvL7R53Mwr dMREbtyZgUpGf+Bg4d51IBrmov5pQ5O1jPWzUSSs2p4w4cHv1n6Zy9oTwkpYGYZA QvOAk2Y6qxt7/ZnMx9skYPpjVCCRiUEVPGtD86U6zNghA1VQvqrQKT7C9IBfUSLj jnbVcx0eTTGn2LpfSCEYZyElOBwJjldYwyFxEB4cPkXWRvR++OlP6z/oLluBQtdn edng== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:message-id:mime-version:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; t=1686201110; x=1686287510; bh=oD6xldeLN7Qfs cQFB0oruziGRqB0Q4sqN6CZa+2e0jo=; b=r99dEBh/fzgJyx8qKxNjM0tcfhaLy SmkCzvgpCSBViRinhQnOjUgcelslLVqc8YvkoTr8eB8vEN68sLmEZETE8HWV51gH 3fyzdNV3nNF8L2oiz4dNMm2e5l+RtG0dIsuMFx78IDp5lQ0Rj90z0oiNmGcgB7OH JvpvTtFBfEkGlMVFDgUf+CQfJN2Ktb1siEXEO6XYIKS+yYOUJN0WLi+6EpWKqKpN HjJr6HnRnVhUhcR0oOQR6F5Dm/S9+KIADLaUMO9Dr++3H/cA5vvZ2JmQSykKJh8F YmmNIGpByyW+BshqG1ezB+AnXepvkrFrvebhhKLgVLo/Jl4y/8H0AIcHg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrgedthedgleduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfvffhufgtgfesthekredttdefjeenucfhrhhomheptfgvsggvtggt rgcuvehrrghnuceorhgvsggvtggtrgessghsughiohdrtghomheqnecuggftrfgrthhtvg hrnhepieefhfffhfetleffgfefjeeuudetlefhieehleeffeejheevfeeileetfefhtdeg necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomheprhgvsg gvtggtrgessghsughiohdrtghomh X-ME-Proxy: Feedback-ID: i5b994698:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 8 Jun 2023 01:11:50 -0400 (EDT) Message-ID: <5b52fc08-fb5a-900e-b98c-817a4ab79846@bsdio.com> Date: Wed, 7 Jun 2023 23:11:48 -0600 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Content-Language: en-US To: FreeBSD CURRENT From: Rebecca Cran Subject: Seemingly random nvme (nda) write error on new drive (retries exhausted) Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-4.63 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_LONG(-1.00)[-0.998]; NEURAL_HAM_SHORT(-0.93)[-0.931]; R_DKIM_ALLOW(-0.20)[bsdio.com:s=fm2,messagingengine.com:s=fm1]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.26]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.26:from]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.26:from]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdio.com:+,messagingengine.com:+]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[bsdio.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; TO_DN_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4QcC4R5dqfz4GxJ X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N I got a seemingly random nvme data transfer error on my new arm64 Ampere Altra machine, which has a Samsung PM1735 PCIe AIC NVMe drive. Since it's a new drive and smartctl doesn't show any errors I thought it might be worth mentioning here. I'm running 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n263139-baef3a5b585f. dmesg contains: nvme0: WRITE sqid:16 cid:126 nsid:1 lba:2550684560 len:8 nvme0: DATA TRANSFER ERROR (00/04) crd:0 m:0 dnr:0 sqid:16 cid:126 cdw0:0 (nda0:nvme0:0:0:1): WRITE. NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=98085b90 0 7 0 0 0 (nda0:nvme0:0:0:1): CAM status: CCB request completed with an error (nda0:nvme0:0:0:1): Error 5, Retries exhausted nvmecontrol identify nvme0 shows: Vendor ID:                   144d Subsystem Vendor ID:         144d Model Number:                SAMSUNG MZPLJ6T4HALA-00007 Firmware Version:            EPK9CB5Q Recommended Arb Burst:       8 IEEE OUI Identifier:         00 25 38 Multi-Path I/O Capabilities: Multiple controllers, Multiple ports Max Data Transfer Size:      131072 bytes Sanitize Crypto Erase:       Supported Sanitize Block Erase:        Supported Sanitize Overwrite:          Not Supported Sanitize NDI:                Not Supported Sanitize NODMMAS:            Undefined Controller ID:               0x0041 Version:                     1.3.0 -- Rebecca Cran From nobody Thu Jun 8 06:24:55 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcDj31SWlz4blWh for ; Thu, 8 Jun 2023 06:25:11 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcDj268Q7z4Pwg for ; Thu, 8 Jun 2023 06:25:10 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-973bf581759so51504966b.0 for ; Wed, 07 Jun 2023 23:25:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20221208.gappssmtp.com; s=20221208; t=1686205507; x=1688797507; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=AK4dKdh+r1YH5KhdtFWfl0jV1UMeKyPqkc3jttwW3RE=; b=J4iX2z6ghzfzJfKoOtMKF6wTtYa7k0AnRT0c8w/M8r8kdZarkedXU3Fcgokl1m08hz Sef31dZT3PsmlH4AvNbErELA1sMrLvQotMXVgDZ135cpEeOce4NziBWRVBUT54w8NPD9 ZlJKHrjuOhiu+CRwGWaXqrYf1lTKNVqt/JQsDIzdEt5VncpyxRNcd4hN/S+ctYLmMrJO 5wwKb8jpoCzeM57PDU7/T2KP1Ff01TZoLVZ8SdPHM3bY/+BC55M4q9aZdK7uuiIzAPB2 b15M8o0t5Wd9Prfx5v8syphu3HY+2S8nX33AXItr8756hnHqt0YAOE+EXG8+2RB3P052 9Vaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686205507; x=1688797507; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=AK4dKdh+r1YH5KhdtFWfl0jV1UMeKyPqkc3jttwW3RE=; b=Xv7K5Mry3tn8lHCGVmdSRoHRTmOlqKcrVNUpkvvHesEIuHrtlcyiXO3wmzu1NIuUWH Ioe+8hTjzZk2vvprGWfO6wRM75TZElnKuENbsgXjJ3PCdQn6TNfk0NwPUTJwMXucQbn+ 19PLmutraA6ZC6m2OgEi8deyus9Cfp1MHj3t5ONGg8ianSsUekISB04Gqkxs/ZpqBd0T VHKfTGyxyIdWMD6wlfkVP05/eyMbc1/RcfqgC9nXZdE4F9/8PGwKQDW1MQUcrN9bqUwv /sSAt90ts3LevUgJ95SNeLzs1XQxoJoWsmK1MzaXRJCIrvMRN2tRvXDsPqtb7UwBJzfL OBJA== X-Gm-Message-State: AC+VfDxZ2YHguocDx3+exv05tE8r93T2PmgNV0HYqrKNzkRLQ9rh7ayI GGes4BwO+F5IMeI2MEmLmbVHgm8B3MMJCKNznvlna7eLNUI6gz+F X-Google-Smtp-Source: ACHHUZ5xTsZPkhjfpIrV+7dO39mG/fOiDF4kZNTbOkHHBY0lxIIF6uAGjil3ijLJikLrNX0bmzZZaoDFPvvY5PwD8PQ= X-Received: by 2002:a17:907:9718:b0:978:8685:71cd with SMTP id jg24-20020a170907971800b00978868571cdmr4593922ejc.71.1686205506891; Wed, 07 Jun 2023 23:25:06 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <5b52fc08-fb5a-900e-b98c-817a4ab79846@bsdio.com> In-Reply-To: <5b52fc08-fb5a-900e-b98c-817a4ab79846@bsdio.com> From: Warner Losh Date: Thu, 8 Jun 2023 00:24:55 -0600 Message-ID: Subject: Re: Seemingly random nvme (nda) write error on new drive (retries exhausted) To: Rebecca Cran Cc: FreeBSD CURRENT Content-Type: multipart/alternative; boundary="00000000000041c28005fd9850ff" X-Rspamd-Queue-Id: 4QcDj268Q7z4Pwg X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --00000000000041c28005fd9850ff Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Jun 7, 2023 at 11:12=E2=80=AFPM Rebecca Cran wr= ote: > I got a seemingly random nvme data transfer error on my new arm64 Ampere > Altra machine, which has a Samsung PM1735 PCIe AIC NVMe drive. > > Since it's a new drive and smartctl doesn't show any errors I thought it > might be worth mentioning here. > > I'm running 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n263139-baef3a5b585= f. > > > dmesg contains: > > nvme0: WRITE sqid:16 cid:126 nsid:1 lba:2550684560 len:8 > nvme0: DATA TRANSFER ERROR (00/04) crd:0 m:0 dnr:0 sqid:16 cid:126 cdw0:0 > (nda0:nvme0:0:0:1): WRITE. NCB: opc=3D1 fuse=3D0 nsid=3D1 prp1=3D0 prp2= =3D0 > cdw=3D98085b90 0 7 0 0 0 > (nda0:nvme0:0:0:1): CAM status: CCB request completed with an error > (nda0:nvme0:0:0:1): Error 5, Retries exhausted > > > nvmecontrol identify nvme0 shows: > > Vendor ID: 144d > Subsystem Vendor ID: 144d > Model Number: SAMSUNG MZPLJ6T4HALA-00007 > Firmware Version: EPK9CB5Q > Recommended Arb Burst: 8 > IEEE OUI Identifier: 00 25 38 > Multi-Path I/O Capabilities: Multiple controllers, Multiple ports > Max Data Transfer Size: 131072 bytes > Sanitize Crypto Erase: Supported > Sanitize Block Erase: Supported > Sanitize Overwrite: Not Supported > Sanitize NDI: Not Supported > Sanitize NODMMAS: Undefined > Controller ID: 0x0041 > Version: 1.3.0 > PCIe 3 or PCIe 4? So the only documented reason for this error is if we setup the memory wron= g such that the drive couldn't start a transfer from the specified address. This seems weird to me... But in the prior paragraph it talks about other types of aborts that need software intervention. If this is a transient error, then maybe we should retry it as part of the data recovery. Unless this do not retry bit is set. which it isn't. I wonder this is retried 5 times or not before generating the error... Warner --00000000000041c28005fd9850ff Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Wed, Jun 7, 2023 at 11:12=E2=80=AF= PM Rebecca Cran <rebecca@bsdio.com<= /a>> wrote:
I= got a seemingly random nvme data transfer error on my new arm64 Ampere Altra machine, which has a Samsung PM1735 PCIe AIC NVMe drive.

Since it's a new drive and smartctl doesn't show any errors I thoug= ht it
might be worth mentioning here.

I'm running 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n263139-baef3a5b5= 85f.


dmesg contains:

nvme0: WRITE sqid:16 cid:126 nsid:1 lba:2550684560 len:8
nvme0: DATA TRANSFER ERROR (00/04) crd:0 m:0 dnr:0 sqid:16 cid:126 cdw0:0 (nda0:nvme0:0:0:1): WRITE. NCB: opc=3D1 fuse=3D0 nsid=3D1 prp1=3D0 prp2=3D0=
cdw=3D98085b90 0 7 0 0 0
(nda0:nvme0:0:0:1): CAM status: CCB request completed with an error
(nda0:nvme0:0:0:1): Error 5, Retries exhausted


nvmecontrol identify nvme0 shows:

Vendor ID:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 144d
Subsystem Vendor ID:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 144d Model Number:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 SAMSUNG MZPLJ6T4HALA-00007
Firmware Version:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 EPK9CB5Q
Recommended Arb Burst:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 8
IEEE OUI Identifier:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 00 25 = 38
Multi-Path I/O Capabilities: Multiple controllers, Multiple ports
Max Data Transfer Size:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 131072 bytes
Sanitize Crypto Erase:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Supported
Sanitize Block Erase:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Supported Sanitize Overwrite:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N= ot Supported
Sanitize NDI:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Not Supported
Sanitize NODMMAS:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 Undefined
Controller ID:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0x0041
Version:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1.3.0

PCIe 3 or PCIe 4?

So the onl= y documented reason for this error is if we setup the memory wrong
such that the drive couldn't start a transfer from the specified addr= ess. This seems

--00000000000041c28005fd9850ff-- From nobody Thu Jun 8 09:28:19 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcJmR2By7z4c6qf for ; Thu, 8 Jun 2023 09:28:23 +0000 (UTC) (envelope-from rebecca@bsdio.com) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcJmQ75hvz3jXx for ; Thu, 8 Jun 2023 09:28:22 +0000 (UTC) (envelope-from rebecca@bsdio.com) Authentication-Results: mx1.freebsd.org; none Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 05F8D5C012F; Thu, 8 Jun 2023 05:28:21 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Thu, 08 Jun 2023 05:28:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdio.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm2; t= 1686216501; x=1686302901; bh=hDUDq3omg/hQQhPwTWfa4t/nhzUsCXKYGpt LVHgFyhk=; b=2DilElwsCB7LmJC2eAuhob9pZl/wAwVapnqQUPXFtbXvu9HsLzF 6GN8m6QezHuCegFFmtgUxeRkM/r59gx7gE6Gv4UZM4PykRl780JrI2zSkjyFkr/f GUUMYLPN7pX7aiOvfcCuPQqUaPp0KP4+CgHS31Ysvjwynorffi8MBFdLpq4EdVjD +EJriHqUxLAiQokSTXgoVQBD7qbhrwq2tq426Rvft3KD+8TUUBzQzNm0QxrowJUX v4EPHFHqURFSucBNgCUX1vHgaANbpIhWuPMCnB68Q10LcJhDDKkrRIhQDp7G6W9h uj81tqOhLg/nbL8PIfYu1Tt6tdMzzr74I0w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1686216501; x=1686302901; bh=hDUDq3omg/hQQhPwTWfa4t/nhzUsCXKYGpt LVHgFyhk=; b=CwR5Y6yziuJUsK5JEr7xRaoc8a2UNY7hfWWnT9vnrKF5yu668Ix MMeACwEKLdweM4ByJRSGH8mfIdHhjjCOfS0oK243U8uNMGfoRzNtBWwebgclcUyP okGgbSKhUM4MAoiCqgdQ0H9eQGSs7lBjtArF5aiopvRim2kfrreW+4R+DY6/Gr+B 6jLWXUlv7adxDnYpQpJicOttJzNSqY94vxZALnsKovpZI8XKC9EcYol6ebxnwTJa gAmqze04QEjQR70LWHltWtU+/ZlZJR5Rb2wxrh65Sl0LEBnnpExNEsXFKzldK84b gc058gewH6NnZ4grOW3GG+gHgtE0ZwdzqAg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrgedtiedgudeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeftvggs vggttggrucevrhgrnhcuoehrvggsvggttggrsegsshguihhordgtohhmqeenucggtffrrg htthgvrhhnpeehleekgfdutedugeehgeffgffhteefhffhteetfeevhefgvdejhedthfff ieffleenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe hrvggsvggttggrsegsshguihhordgtohhm X-ME-Proxy: Feedback-ID: i5b994698:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 8 Jun 2023 05:28:20 -0400 (EDT) Message-ID: Date: Thu, 8 Jun 2023 03:28:19 -0600 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: Seemingly random nvme (nda) write error on new drive (retries exhausted) Content-Language: en-US To: Warner Losh Cc: FreeBSD CURRENT References: <5b52fc08-fb5a-900e-b98c-817a4ab79846@bsdio.com> From: Rebecca Cran In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4QcJmQ75hvz3jXx X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 6/8/23 00:24, Warner Losh wrote: > PCIe 3 or PCIe 4? PCIe 4. nda0 at nvme0 bus 0 scbus0 target 0 lun 1 nda0: nda0: Serial Number S55KNC0TC00168 nda0: nvme version 1.3 x8 (max x8) lanes PCIe Gen4 (max Gen4) link nda0: 6104710MB (12502446768 512 byte sectors) -- Rebecca Cran From nobody Thu Jun 8 10:25:36 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcL2k6D5Sz4bHvX for ; Thu, 8 Jun 2023 10:25:50 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-qk1-x730.google.com (mail-qk1-x730.google.com [IPv6:2607:f8b0:4864:20::730]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcL2k4H3sz3sYV for ; Thu, 8 Jun 2023 10:25:50 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; none Received: by mail-qk1-x730.google.com with SMTP id af79cd13be357-75d4dd6f012so39797285a.2 for ; Thu, 08 Jun 2023 03:25:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1686219949; x=1688811949; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=UGK5UMjSiW1lTrPUKN/3d6JLh2mgS/2VBAGb/z7L2GU=; b=aqRY/3HNF1fbYlB92/OnqgU2WqOErAPqdP0RWE7ofPhAOi+jvoUdwEACFS+tLTDTtC SfS1CmQZiyMfAiGhJxfiOqQMMe5kQ/BaSRLQRVlGMpvnXnnJjqYGfTHqb3ztWJOK0ZkX boldA7QJAYtHhbjXoERbRj2o9ahekPrqVHN5s94XgTRl3jdVIHRihQfLdzquoQZQyBJ/ 9MNfowkPS1QGYzO8+aJ5olwvn9reBvg/IAb9EnTRu/F5JItVnQ8+u6UzpTp7ihSWT7r3 C1rxKOpk1H48zqKo98kf6uhkJPoKgZPAuZADDSPMyBLctOK6LYcKsgdc7tB8vuMhDdBR YtOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686219949; x=1688811949; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=UGK5UMjSiW1lTrPUKN/3d6JLh2mgS/2VBAGb/z7L2GU=; b=j2HUqcRSejxeBuD2KC0490xjcJ+8VULAhcjIbfM7lJK4qGxI6McH6yNT9h++srHrTC hyanlx7GLNnDUi9MBmgl3Khb6ZodBxtmT2OvduXGui9N/5QvAzuGfKFSDBK+jckliWF0 j0esA8d1abR3EZMl8uXSHJdNKqs9l2bxrcRhEuSCkOT50fywbY4FSWGCz74Wq0g44RsX VUF0EPmJcCS+qccVjwh/IBs5JWDKfvRmM3vkBJX3ZHCaypKCFcsop8S7uzrSESKz8uL0 dkTciLfSkT89f/0htubv0TDVKfBsyg9EScrxe9HMk5kN9spCNXxYT3y8J/3PRrprMuZG 3mDA== X-Gm-Message-State: AC+VfDwIufu39MDgckbXxHOA6UBdXlKmIJEqhzW8lfwBKbNxS0UUBmpB +IuFjdgxjD6E1vvMXx/G49IUuWar46xqJRCgXdI= X-Google-Smtp-Source: ACHHUZ4UQBa5FOekGuXZyRkqTF/0ps1TlOoRLBjL1o0g+od6Y27yY9HJ6qjCOocA5GLx5SI+w3HxJA== X-Received: by 2002:a05:620a:a4d:b0:75b:23a0:de9b with SMTP id j13-20020a05620a0a4d00b0075b23a0de9bmr5004593qka.25.1686219949556; Thu, 08 Jun 2023 03:25:49 -0700 (PDT) Received: from mail-yw1-f178.google.com (mail-yw1-f178.google.com. [209.85.128.178]) by smtp.gmail.com with ESMTPSA id a189-20020a818ac6000000b00552df52450csm285152ywg.88.2023.06.08.03.25.49 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 08 Jun 2023 03:25:49 -0700 (PDT) Received: by mail-yw1-f178.google.com with SMTP id 00721157ae682-565eb83efe4so3757527b3.0 for ; Thu, 08 Jun 2023 03:25:49 -0700 (PDT) X-Received: by 2002:a81:8450:0:b0:565:b22c:4165 with SMTP id u77-20020a818450000000b00565b22c4165mr10395444ywf.11.1686219948817; Thu, 08 Jun 2023 03:25:48 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <5b52fc08-fb5a-900e-b98c-817a4ab79846@bsdio.com> In-Reply-To: <5b52fc08-fb5a-900e-b98c-817a4ab79846@bsdio.com> From: Tomek CEDRO Date: Thu, 8 Jun 2023 12:25:36 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Seemingly random nvme (nda) write error on new drive (retries exhausted) To: Rebecca Cran Cc: FreeBSD CURRENT Content-Type: multipart/alternative; boundary="00000000000010051505fd9bad0a" X-Rspamd-Queue-Id: 4QcL2k4H3sz3sYV X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --00000000000010051505fd9bad0a Content-Type: text/plain; charset="UTF-8" what filesystem? is TRIM enabled on that drive? have you tried disabling trim? i had similar ssd related problem on samsung's ssd long time ago that was related to trim. maybe drive firmware can be updated too? :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info --00000000000010051505fd9bad0a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable --00000000000010051505fd9bad0a-- From nobody Thu Jun 8 11:34:55 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcMZW0LVCz4bPNw for ; Thu, 8 Jun 2023 11:34:59 +0000 (UTC) (envelope-from rebecca@bsdio.com) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcMZV14w3z420X for ; Thu, 8 Jun 2023 11:34:58 +0000 (UTC) (envelope-from rebecca@bsdio.com) Authentication-Results: mx1.freebsd.org; none Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id B6C495C00B9; Thu, 8 Jun 2023 07:34:57 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Thu, 08 Jun 2023 07:34:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdio.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm2; t= 1686224097; x=1686310497; bh=uuc1CMcOtN1zQnscgVFS4m0iDOwGtuUzv8X KOyLXAmk=; b=rwEu+0MdY+3AO2pM+9lXpuZUfD/Wg3P1BtJJvr9J5vVUWxq1zgL Hd6Oo49TMmi3kcn2YhaErvp/9ck+EcD+foF5okq62hA22OqMKqqcjWjB+81venyp wxZs/4cJu3ltruyokVlFWyZ7CZPl95gxJO9KPsbZmGjXLN9ZnVvJWIvJiDwpACia GhADgHIQTemWfypoYVKrjPUeY9Y+/xu6RBqLMlkzIPUQGPj+S0VQl0FX+xnpp9lF TroXEWdAZC0E4dMiP7ju9bp5HVonm9QT+RCiJAcuswZ8mSomTJUvVm74cDsb9wof towx4vBy5rFODXVPradoljoJHYHvCcIVPPQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1686224097; x=1686310497; bh=uuc1CMcOtN1zQnscgVFS4m0iDOwGtuUzv8X KOyLXAmk=; b=AfbyYpkWcyJ9KAXqqimHaKR6xjvrz0a9luDjMMHzC0oI/9hIhDb BvGbpMvboFI9qclHnR+VwS7So9wOZBFSKsZLG5LCikyeYDSpAB+hDZ93IcCc91lO pmFhTInjtiWQg0lVsmt0rvBpFFTt/rTcLakDqLzztzL9qxKX060RfE7tyba1JOzQ tQe9JPxDlPezDurwDMbS+uR1klHlS4Zplyub5ekwSs2pP9hOtG/2Y6Shoaa+mIuj YbvDj2D2P8+oCDgOf1VZpKgmeUlBLFjiO0g9VoH7UX663f4kMEncL64FrrXjnzZP 2gzqosADHTCQ8qzy4SHqhYoh4Prcb7Shaag== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrgedtiedggeduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeftvggs vggttggrucevrhgrnhcuoehrvggsvggttggrsegsshguihhordgtohhmqeenucggtffrrg htthgvrhhnpeejtdduudffvdeggeekleelgfevleduledufeegfedvfeduhfeggfegueev heettdenucffohhmrghinheptggvughrohdrihhnfhhonecuvehluhhsthgvrhfuihiivg eptdenucfrrghrrghmpehmrghilhhfrhhomheprhgvsggvtggtrgessghsughiohdrtgho mh X-ME-Proxy: Feedback-ID: i5b994698:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 8 Jun 2023 07:34:57 -0400 (EDT) Message-ID: Date: Thu, 8 Jun 2023 05:34:55 -0600 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: Seemingly random nvme (nda) write error on new drive (retries exhausted) Content-Language: en-US To: Tomek CEDRO Cc: FreeBSD CURRENT References: <5b52fc08-fb5a-900e-b98c-817a4ab79846@bsdio.com> From: Rebecca Cran In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4QcMZV14w3z420X X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N It's ZFS, using the default options when creating it via the FreeBSD installer so I presume TRIM is enabled. Without a reliable way to reproduce the error I'm not sure disabling TRIM will help at the moment. I don't think there's any newer firmware for it. -- Rebecca Cran On 6/8/23 04:25, Tomek CEDRO wrote: > what filesystem? is TRIM enabled on that drive? have you tried > disabling trim? i had similar ssd related problem on samsung's ssd > long time ago that was related to trim. maybe drive firmware can be > updated too? :-) > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From nobody Thu Jun 8 11:48:22 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcMth15ZTz4bQCb for ; Thu, 8 Jun 2023 11:49:00 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcMtg6XCFz449t for ; Thu, 8 Jun 2023 11:48:59 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-3f6d7abe934so3149505e9.2 for ; Thu, 08 Jun 2023 04:48:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20221208.gappssmtp.com; s=20221208; t=1686224935; x=1688816935; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=hVXuI9kWQdfyJlipfVXLc7tfyHiP89BwCYZrTYQdzQ0=; b=sw3TmdD8yIjXmOtFEsOfXJJEhacKlsakWZ6h+HeIxnVyEb4gqjS1bqaCQsTV988bor 2lA8JmCAu+mDdsNvhOVDhYb/cgGCwPtJ7nsvKKfXPvrP2W+KtKViWr8wk471jm+LO+lr 3VvttmqmlYrZ9Gc4u8swt3jqgMeRlyoPoH7XW7dizK83hfCdASrfWaoX6gBMJX1Qfhz8 SC0pc8wXXlMhiOcWhfPmRcfa5aFlthY5qdINWO1xdWjzaImfL1dzvJ+EVHwtRK4vnGds igf0kgkDYyMyfIGfbOMoOf7fPz0qAOahFnZomVNiQtWOnmWSP8/Z9ISOdcBSwU3Kw58v i1ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686224935; x=1688816935; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hVXuI9kWQdfyJlipfVXLc7tfyHiP89BwCYZrTYQdzQ0=; b=HhGEal6hFik3gnqtjBnVJpz+Woe8tZssU8OBkPAbAl1tFvhHItMJEmMAVhIvR3Mh23 Fn2P1WRi9Qw6NB48ZPw4RSmfQeMn8qOoI5TgnmnItJb+bPCV3xD8eQWfqBRmdXuJ8wnO pLFEA9h6BX8GFIc+LLPbWWHalR+UDubTBcZf7AlG/mZwG5AvPP4mD3TBZYpACp0uR+WZ Ih+pBOL78vEstI7aTxWyWgrL/rKiAlgqxBsr3//3KpejE6Yc8rTtPVy9t1bOYHod7cli Iv6V4mnQt5uwLohyzFwrDbN3yJSBgz8IXc1H2fSgWVS4B8++CIaE8JdTaicgqOb4To4/ MpSQ== X-Gm-Message-State: AC+VfDx7E9yiATXBXS8cOUU+m2JtkDgcdg8bnAzE0Srxcdi//0/w7mnX /5CszaBVkkb1WGZB6JQhKbrJMwqO3Lw3Qdp3lGTOVFQdObdFq4Hs X-Google-Smtp-Source: ACHHUZ6nlxId60sTpIJU6RSR6uVZQE1N2H20qdmCzBUAHNYT9EQQ0xLiFIumCmf9f5NBjjC2THv40Ytf1E8EUojZ1EU= X-Received: by 2002:a17:907:868f:b0:96a:5bdd:7557 with SMTP id qa15-20020a170907868f00b0096a5bdd7557mr9110617ejc.70.1686224913943; Thu, 08 Jun 2023 04:48:33 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <5b52fc08-fb5a-900e-b98c-817a4ab79846@bsdio.com> In-Reply-To: From: Warner Losh Date: Thu, 8 Jun 2023 04:48:22 -0700 Message-ID: Subject: Re: Seemingly random nvme (nda) write error on new drive (retries exhausted) To: Rebecca Cran Cc: Tomek CEDRO , FreeBSD CURRENT Content-Type: multipart/alternative; boundary="00000000000001e0af05fd9cd59f" X-Rspamd-Queue-Id: 4QcMtg6XCFz449t X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --00000000000001e0af05fd9cd59f Content-Type: text/plain; charset="UTF-8" On Thu, Jun 8, 2023, 4:35 AM Rebecca Cran wrote: > It's ZFS, using the default options when creating it via the FreeBSD > installer so I presume TRIM is enabled. Without a reliable way to > reproduce the error I'm not sure disabling TRIM will help at the moment. > > I don't think there's any newer firmware for it. > pci gen 4 has a highter error rate so that needs to be managed with retries. There's a whole protocol to do that which linux implements. I suspect the time has come for us to do so too. There's some code floating around I'll have to track down. Warner -- > > Rebecca Cran > > > On 6/8/23 04:25, Tomek CEDRO wrote: > > what filesystem? is TRIM enabled on that drive? have you tried > > disabling trim? i had similar ssd related problem on samsung's ssd > > long time ago that was related to trim. maybe drive firmware can be > > updated too? :-) > > > > -- > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > --00000000000001e0af05fd9cd59f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Thu, Jun 8, 2023, 4:35 AM Rebecca Cran <rebecca@bsdio.com> wrote:
It's ZFS, using the default options when cr= eating it via the FreeBSD
installer so I presume TRIM is enabled. Without a reliable way to
reproduce the error I'm not sure disabling TRIM will help at the moment= .

I don't think there's any newer firmware for it.

pci gen 4 has a hig= hter error rate so that needs to be managed with retries.=C2=A0 There's= a whole protocol to do that which linux implements. I suspect the time has= come for us to do so too. There's some code floating around I'll h= ave to track down.

Warne= r

--

Rebecca Cran


On 6/8/23 04:25, Tomek CEDRO wrote:
> what filesystem? is TRIM enabled on that drive? have you tried
> disabling trim? i had similar ssd related problem on samsung's ssd=
> long time ago that was related to trim. maybe drive firmware can be > updated too? :-)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

--00000000000001e0af05fd9cd59f-- From nobody Thu Jun 8 14:00:03 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcQnz3WPsz4bktN for ; Thu, 8 Jun 2023 14:00:07 +0000 (UTC) (envelope-from rebecca@bsdio.com) Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcQnz1cNhz4LSF for ; Thu, 8 Jun 2023 14:00:07 +0000 (UTC) (envelope-from rebecca@bsdio.com) Authentication-Results: mx1.freebsd.org; none Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 625AF32002D8; Thu, 8 Jun 2023 10:00:05 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Thu, 08 Jun 2023 10:00:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdio.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm2; t= 1686232804; x=1686319204; bh=iXhv2JvZy4Pqu+Y5tWX+uZWLKZ+Ryh6FjVl 5KCpO/Bw=; b=BBdiM9SeXe0WlZLxVO4uPxVdmLoPH0OxMv9NmuBlZFkAeBPcYi+ MVofmV8I6XxlOJbEfwvLPhTxa7dT0m2lfx/r/8EiG/MbnWcnvL1SL0eP3pvp43wl /+sRBfc/k9dVb79xxKt3+T3Rvbx4SdrGLWMXoP5FWY6Dyo960Hx9Xsf7yTw1z0OW FMixcm/ysaTBOcEAPZrKGgYKHXFq9YuzkOH4b95S5y1cMMOhVGssOw7GVuJ4XrOm Gl+Kv2sRb+Vb+rCxlv/OrjHW+XGtTFaBiUmHXgbDmJObWxJNrliXIBbLwaatiV+3 W6IWOOVCLRqKB8dtR75C7tZb2pkaL3+ymoQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1686232804; x=1686319204; bh=iXhv2JvZy4Pqu+Y5tWX+uZWLKZ+Ryh6FjVl 5KCpO/Bw=; b=SVIyYnSuqA4/HUWTDe6/1aYKz418/oDiT4QV/vKKsg4rFlhmnRo /nhE5UDVKCLBHNEOis1ehQv9LifbBQsbAJrUJb+KVP/t2CDoCAXt0ysYjA7hcJED PmgWqPbmWRlYVF84Zu79TNUO3Vxjc6oKLvwjyZwqOx/qkD9412TSrllUzxNDW1B/ 25AdC59Yerf644/bIs+Qocy3p2LPhqZtdkLYzWk32tpljd2y+AvvUYl5owPciQ4g 1KEfQ+fvXEgd8gB+7712MzdH46GCBZ0KWd3bDqvWJEJNnlcupHfmt5X5QKvzAPON 7R60VS8CZQYwCuQY/Q26AoWGWLc5hB3n96Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrgedtiedgjeduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtkeertddtfeejnecuhfhrohhmpeftvggs vggttggrucevrhgrnhcuoehrvggsvggttggrsegsshguihhordgtohhmqeenucggtffrrg htthgvrhhnpeegfeegveduheejgfduffefhfehleehiefghfetvdejvdelhfeukefhhfdv geehveenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe hrvggsvggttggrsegsshguihhordgtohhm X-ME-Proxy: Feedback-ID: i5b994698:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 8 Jun 2023 10:00:04 -0400 (EDT) Message-ID: Date: Thu, 8 Jun 2023 08:00:03 -0600 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: Seemingly random nvme (nda) write error on new drive (retries exhausted) Content-Language: en-US To: Warner Losh Cc: Tomek CEDRO , FreeBSD CURRENT References: <5b52fc08-fb5a-900e-b98c-817a4ab79846@bsdio.com> From: Rebecca Cran In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4QcQnz1cNhz4LSF X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 6/8/23 05:48, Warner Losh wrote: > > > On Thu, Jun 8, 2023, 4:35 AM Rebecca Cran wrote: > > It's ZFS, using the default options when creating it via the FreeBSD > installer so I presume TRIM is enabled. Without a reliable way to > reproduce the error I'm not sure disabling TRIM will help at the > moment. > > I don't think there's any newer firmware for it. > > > pci gen 4 has a highter error rate so that needs to be managed with > retries.  There's a whole protocol to do that which linux implements. > I suspect the time has come for us to do so too. There's some code > floating around I'll have to track down. Thanks. I dropped the configuration down to PCIe gen 3 and the errors have so far gone away. nda0: nvme version 1.3 x8 (max x8) lanes PCIe Gen3 (max Gen4) link nda1: nvme version 1.3 x4 (max x4) lanes PCIe Gen3 (max Gen4) link -- Rebecca Cran From nobody Thu Jun 8 16:11:15 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcTl312d2z4bvrd for ; Thu, 8 Jun 2023 16:12:47 +0000 (UTC) (envelope-from grembo@freebsd.org) Received: from mail.evolve.de (mail.evolve.de [213.239.217.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.evolve.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcTl20cDhz4c6g for ; Thu, 8 Jun 2023 16:12:46 +0000 (UTC) (envelope-from grembo@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 213.239.217.29 is neither permitted nor denied by domain of grembo@freebsd.org) smtp.mailfrom=grembo@freebsd.org; dmarc=none Received: by mail.evolve.de (OpenSMTPD) with ESMTP id d5fe161c for ; Thu, 8 Jun 2023 16:12:37 +0000 (UTC) Received: by mail.evolve.de (OpenSMTPD) with ESMTPSA id bd398912 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 8 Jun 2023 16:12:37 +0000 (UTC) Date: Thu, 8 Jun 2023 18:11:15 +0200 From: Michael Gmelin To: "freebsd-current@freebsd.org" Subject: CURRRENT snapshot won't boot due missing ZFS feature Message-ID: <20230608181115.7c97ea19.grembo@freebsd.org> X-Face: $wrgCtfdVw_H9WAY?S&9+/F"!41z'L$uo*WzT8miX?kZ~W~Lr5W7v?j0Sde\mwB&/ypo^}> +a'4xMc^^KroE~+v^&^#[B">soBo1y6(TW6#UZiC]o>C6`ej+i Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWJBwe5BQDl LASZU0/LTEWEfHbyj0Txi32+sKrp1Mv944X8/fm1rS+cAAAACXBIWXMAAAsTAAAL EwEAmpwYAAAAB3RJTUUH3wESCxwC7OBhbgAAACFpVFh0Q29tbWVudAAAAAAAQ3Jl YXRlZCB3aXRoIFRoZSBHSU1QbbCXAAAAAghJREFUOMu11DFvEzEUAGCfEhBVFzuq AKkLd0O6VrIQsLXVSZXoWE5N1K3DobBBA9fQpRWc8OkWouaIjedWKiyREOKs+3PY fvalCNjgLVHeF7/3bMtBzV8C/VsQ8tecEgCcDgrzjekwKZ7TwsJZd/ywEKwwP+ZM 8P3drTsAwWn2mpWuDDuYiK1bFs6De0KUUFw0tWxm+D4AIhuuvZqtyWYeO7jQ4Aea 7jUqI+ixhQoHex4WshEvSXdood7stlv4oSuFOC4tqGcr0NjEqXgV4mMJO38nld4+ xKNxRDon7khyKVqY7YR4d+Cg0OMrkWXZOM7YDkEfKiilCn1qYv4mighZiynuHHOA Wq9QJq+BIES7lMFUtcikMnkDGHUoncA+uHgrP0ctIEqfwLHzeSo+eUA66AqzwN6n 2ZHJhw6Qh/PoyC/QENyEyC/AyNjq74Bs+3UH0xYwzDUC4B97HgLocg1QLYgDDO1v f3UX9Y307Ew4AHh67YAFFsxEpkXwpXY3eIgMhAAE3R19L919nNnuD2wlPcDE3UeT L2ytEICQib9BXgS2fU8PrD82ToYO1OEmMSnYTjSqSv9wdC0tPYC+rQRQD9ESnldF CyqfmiYW+tlALt8gH2xrMdC/youbjzPXEun+/ReXsMCDyve3dZc09fn2Oas8oXGc Jj6/fOeK5UmSMPmf/jL+GD8BEj0k/Fn6IO4AAAAASUVORK5CYII= List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-2.08 / 15.00]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.98)[-0.979]; MIME_GOOD(-0.10)[text/plain]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:24940, ipnet:213.239.192.0/18, country:DE]; TO_DN_EQ_ADDR_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[grembo]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; R_SPF_SOFTFAIL(0.00)[~all:c]; TO_DOM_EQ_FROM_DOM(0.00)[] X-Rspamd-Queue-Id: 4QcTl20cDhz4c6g X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Hi, I didn't dig into this yet. After installing the current 14-snapshot (June 1st) in a bhyve-vm, I get this on boot: ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2 (booting stops at this point) Seems like the boot loader is missing this recently added feature. Best Michael -- Michael Gmelin From nobody Thu Jun 8 16:20:12 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcTvm0fZMz4bwqZ for ; Thu, 8 Jun 2023 16:20:20 +0000 (UTC) (envelope-from gjb@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcTvm0C4cz4fg2; Thu, 8 Jun 2023 16:20:20 +0000 (UTC) (envelope-from gjb@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1686241220; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=0yCNj3K51chIlVvYxjshcbYolz5BVYDpBP4O+UZqTx0=; b=olbyYNIM/9fgFfUYIlV7++h29sRdOBCUzrqcKcIwNXXEY+bMMjX+ArNA7AzC9c+w5spymh YStwdPX+9o5MQZkgmm5q2XR4Bdhdo7HfcDZQFT9IdNN7aMrXZgzqek0b+v54z1pI8QglLy j86haUjy8gzhR9FvRbZOUlKFUJTFbrSjrJiut/FnBqth+M+YtD+jGzNVLPribi/d77IwrT hvMdTqPuBfJG2qPaIK68bm/j8WEn7QSjMImVC+FVFVeB0+cILjB4PEBHUrft6o2aLPyw8C DoC4hmQelB7+p5Zk2h0bg4RgKimzguAkGixIekikgekg7kMIEQh7+phAFG/Qcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1686241220; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=0yCNj3K51chIlVvYxjshcbYolz5BVYDpBP4O+UZqTx0=; b=K1sexgL03/mnk/SrDDmBfpwvIUrv2BOt//+XAWuSpIP/Peryed1G5Itp9jVqxDKP5oWUO6 phFyRHKD0x//hfPpL5Ye9WESzkmJSCUDD+FWmW05spWfpuUDp5KGlQ0/nsm99qSY8eSiHs b7qwnV3UMOq5SV/YLjSdoZXTXL/JYlEcwpmjJVGMxqQJPNYFA4cPehqys4m2iqLmG/3GT4 KEFuzZUYusg7CtWbOEgCu6jXVVKvE+CZlspD28SbJShEizzFSIfabTVE068S/7+XqsUGJ3 543fXqBpGH1XTGjGAgPfVAGA98g+1Wx7UunCFW9766OnkgW0hJgnuS74gQeKKA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1686241220; a=rsa-sha256; cv=none; b=O2ALGlmzsIeBSbCEa3tWN9n9HqZuLAZB8EnqIZdMxS06tIZ1dervg1ZC4ugl8MF+PsWPDP ar2OyQmKw3IRhprjNP5Yi5dWbek0CyuT8Y4Hfjz5hycNdKHBCB3SZ0bpy0KZ+im831YWnw uHFA93nCX0xwW9Dml3gbzWwesDAB3cKTA7873tLNQY+BPovAmBaevLGrGKZxVCstLU7kSg 68sYy1b1bjZsf5fbWsw8szPkUzTKL+UQukbj35LIG350mUYlYX5RewFn+ErzriuBw6cprQ yTAk+HJQEjIcQg/aB9ADN5ixs5MFGUItQ+q9W1+95400H+lT8NQq6lot9G7r+Q== Received: from FreeBSD.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id B40414B7C; Thu, 8 Jun 2023 16:20:19 +0000 (UTC) (envelope-from gjb@freebsd.org) Date: Thu, 8 Jun 2023 16:20:12 +0000 From: Glen Barber To: Michael Gmelin Cc: "freebsd-current@freebsd.org" Subject: Re: CURRRENT snapshot won't boot due missing ZFS feature Message-ID: <20230608162012.GV1219@FreeBSD.org> References: <20230608181115.7c97ea19.grembo@freebsd.org> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="c1qHkdEbEbCG94PZ" Content-Disposition: inline In-Reply-To: <20230608181115.7c97ea19.grembo@freebsd.org> X-ThisMailContainsUnwantedMimeParts: N --c1qHkdEbEbCG94PZ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jun 08, 2023 at 06:11:15PM +0200, Michael Gmelin wrote: > Hi, >=20 > I didn't dig into this yet. >=20 > After installing the current 14-snapshot (June 1st) in a bhyve-vm, I > get this on boot: >=20 > ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2 >=20 > (booting stops at this point) >=20 > Seems like the boot loader is missing this recently added feature. >=20 Can you try today's snapshot? They are propagated to most mirrors now. Glen --c1qHkdEbEbCG94PZ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEjRJAPC5sqwhs9k2jAxRYpUeP4pMFAmSB/7wACgkQAxRYpUeP 4pNqCg//fwnE3fraFAVJ55PkoR+wsF5ohFducGMPJB3ON/dG8/WjWcTpsw0JvKA9 bTTWODXe514aYsCItOdSJ/UdC4xpiUfnxHFwov+Rl86qSJDqjcX7dWx7nHi4/FSH 2w2QkVXXGdBJguIqqQ5a1ar8qIQHwznpyCIKy4VWGh7g4DGGcQfV1nDMEbldah8L 7oT5akfvY8lZFqAcDRasgHld7PbDhVflvsIk0vdWZZSYyo+u9sU+NESJhtJTK1Lj gpIwoLRZiOLzp0Qj8XXe9FAdAFZBKLJz1Y2fh3ZleSmTK22mERbcNpikXKaiUp+H nzlGbAyJKF6ut03iFzz8BUvF8t9rIWqFN4F81T1qgRKnjlX1tWqc0F6G4T81PSjb mWXrd+gtL0xHwHmjD/5ENzR75mxcZGZpQiuQAp0u3cJhd6/36EqxhUPVWtrLYBXF K5xdDHfTpiHMYMvZHqbj2DfTBe/Fmtl1yMw5U/i7I4CuQVHZgzR12aXucXoLx/Uf XNfS5zdpvKCqhifz9G4nRucma718Bv6dWNwNTYW1lUBCOicmm2iax8d6Eoxiz6dY +YcC1YWZybtOnfilMbS1io0Xt95AOZr7xiMY5LW6CP+wk6QDAAhwrI0aTN/ckWds 8MwCPvSmyEC6lX0zBkTL3Vej7oLdIb9AiYzjX2N5jMaxnb/kXpU= =jkw9 -----END PGP SIGNATURE----- --c1qHkdEbEbCG94PZ-- From nobody Thu Jun 8 16:28:14 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcV4y68kfz4bxln for ; Thu, 8 Jun 2023 16:28:18 +0000 (UTC) (envelope-from yuri@aetern.org) Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcV4y34h5z3Dws for ; Thu, 8 Jun 2023 16:28:18 +0000 (UTC) (envelope-from yuri@aetern.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=aetern.org header.s=fm1 header.b=wRli7BvT; dkim=pass header.d=messagingengine.com header.s=fm1 header.b="J INtNZ+"; spf=pass (mx1.freebsd.org: domain of yuri@aetern.org designates 64.147.123.19 as permitted sender) smtp.mailfrom=yuri@aetern.org Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 03709320096B for ; Thu, 8 Jun 2023 12:28:16 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Thu, 08 Jun 2023 12:28:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aetern.org; h=cc :content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm1; t= 1686241696; x=1686328096; bh=UFdbI9e3xbnJ/nz23SzjKIJ87cdC/CTyDej RoXYrnAk=; b=wRli7BvTA2JfYlfUcd10lI5MnivLlatCTJHgZJbQbHxtW9Mbxj5 menh08oF1x+XiAuPs8aneHVhDNtNW09HPMnsODGWCVQ55p1gJyxNxUvFS7oIEO6d NVMjWAyJhG2LlmnCl/Vwj20aBxWkQE0jJf85etypT5s28jtIQkorIgsnf1hJ4Yuq pBjxrdJIItgCFKI2hhDD5fzlRHMHHjF9pvbOlJ9MVrOF/740U+TaxgBtATZuXkaG 0VIrhA3Oz4Jkw2IprOf0Vm+xrTSh9e3gC7R30sfnPtOHgDB5Bj6P+LEGX+F6SRRk cwwvYJOVT9fiKZ9DtDNFTJVi8nmWIn2/ESA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1686241696; x= 1686328096; bh=UFdbI9e3xbnJ/nz23SzjKIJ87cdC/CTyDejRoXYrnAk=; b=J INtNZ+O9LCV2t5X529F9zcc734P2JeDsRTNH2vIEq/PgZkxkaMw3O7MSGOpzLTRI JaBoOeisaMg9B8ZlNHitJ+7itbE9UW0NogjXlyoymoQ89eDMUuVcrxpcXjEsvPj3 OKbEyotUoCY2NyiClrF+RoGWY80DsfwnTmEfhle5u5F5pyelfHkjnelpnrsfHffo J+s/uf/toBNUNWuQgdxaC4ufIzbbtW8mUlKCCdFRmMcWy6xtDa4EghAk1KfD+A13 jdKDb9OoxHicQK5wvdsW6Pl6XSj6cmfoTvZUt6u5YHVLd9QrWh54Wz9niO15XtAV f+RlhqHvIIdrsExSQqGKw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrgedtiedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepkfffgggfuffvfhfhjggtgfesth ejredttdefjeenucfhrhhomhepjghurhhiuceohihurhhisegrvghtvghrnhdrohhrgheq necuggftrfgrthhtvghrnhepvedvgeeltdevkeduhfejgeegffetheeujeelhfekgeffte ehveevjeefhffgffdtnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghi lhhfrhhomhephihurhhisegrvghtvghrnhdrohhrgh X-ME-Proxy: Feedback-ID: i0d79475b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 8 Jun 2023 12:28:15 -0400 (EDT) Message-ID: <46816dca-c4a9-6e12-6057-f7fcd642eafd@aetern.org> Date: Thu, 8 Jun 2023 18:28:14 +0200 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.11.2 Subject: Re: CURRRENT snapshot won't boot due missing ZFS feature Content-Language: en-US To: freebsd-current@freebsd.org References: <20230608181115.7c97ea19.grembo@freebsd.org> From: Yuri In-Reply-To: <20230608181115.7c97ea19.grembo@freebsd.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4QcV4y34h5z3Dws X-Spamd-Bar: / X-Spamd-Result: default: False [-0.40 / 15.00]; R_DKIM_ALLOW(-0.20)[aetern.org:s=fm1,messagingengine.com:s=fm1]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.19]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; local_wl_from(0.00)[yuri@aetern.org]; DKIM_TRACE(0.00)[aetern.org:+,messagingengine.com:+]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US] X-Rspamd-Pre-Result: action=no action; module=multimap; Matched map: local_wl_from X-ThisMailContainsUnwantedMimeParts: N Michael Gmelin wrote: > Hi, > > I didn't dig into this yet. > > After installing the current 14-snapshot (June 1st) in a bhyve-vm, I > get this on boot: > > ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2 > > (booting stops at this point) > > Seems like the boot loader is missing this recently added feature. Are you sure it was June 1's? I saw this problem on: FreeBSD-14.0-CURRENT-amd64-20230427-60167184abd5-262599-disc1.iso ...but it was fixed since (for me, at least): FreeBSD-14.0-CURRENT-amd64-20230504-4194bbb34c60-262746-disc1.iso From nobody Thu Jun 8 16:30:20 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcV7N36nrz4by72 for ; Thu, 8 Jun 2023 16:30:24 +0000 (UTC) (envelope-from yuri@aetern.org) Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcV7N0t27z3G8s for ; Thu, 8 Jun 2023 16:30:24 +0000 (UTC) (envelope-from yuri@aetern.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=aetern.org header.s=fm1 header.b=mzQgktEK; dkim=pass header.d=messagingengine.com header.s=fm1 header.b="H thR3vr"; spf=pass (mx1.freebsd.org: domain of yuri@aetern.org designates 64.147.123.19 as permitted sender) smtp.mailfrom=yuri@aetern.org; dmarc=none Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 0AD9A3200979 for ; Thu, 8 Jun 2023 12:30:22 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Thu, 08 Jun 2023 12:30:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aetern.org; h=cc :content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm1; t= 1686241822; x=1686328222; bh=DmuBVeubIRGYdo4kE3TRd+e69YYt2Z2E4+y JDWSzTqk=; b=mzQgktEKBBnDR9r+6htvQ5HtqH9isbA950648IGbLaU9aTR2cpG LozqMKJesJemRcfEQ8FMKrs+DitMdT9Sy8MhsHteNNGC24InpO/cXY3K/mb4qk5K 2+EkaIseA+WEAJWSNuWNKDlk+2OLa6MLMwRs41qdv+qckhLYT/mVO/jHilP+YECg DRodIrjBri84XTyjGm1NTLzucOsD0qIC5vywR3HRjdXpqVrN0o/fuIvD9YFwTuw7 JHvpCWyBk8aGk5MB1Ba2nUdYH32/peoLakdMdjbkys1GOE+QlZ1+OG2OknMnjdRS eERUg/2nT4699qrwq9ANML67QQq+PA6Tgww== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1686241822; x= 1686328222; bh=DmuBVeubIRGYdo4kE3TRd+e69YYt2Z2E4+yJDWSzTqk=; b=H thR3vr4eie6cs2Ncted/sXTVwwQ2w11w1u0K0Bh3/9ImH6uBJ+D1EsWJ06tyXn4X ISFDeNxawdXQo8ti8KhBk6Bqkc0P3JXxcbZkJCyqRFjzr3QIG22a07KN0sJrEU7q n+CwmqR8K4kY3Qfv8bM6be1+EKWqJw+KkMWH29Fxm97J+dVRJY+V4o3sCFl1pqBM JGqqbB6Ww8UJYEhqHt03iubEyYZ0NIqT8Ca2bMiSTc3nt9i8DwAxUnLGKJrnQnb/ RMylam0fA7cwxhXHFLLMo6XPmB4AWn/nH37Yrs4qLj1ec3nkI/TREWqdF53c8ZZI ua9VkguYggBBdxU2Ebitw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrgedtiedguddtvdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepkfffgggfuffhvfhfjggtgfesth ejredttdefjeenucfhrhhomhepjghurhhiuceohihurhhisegrvghtvghrnhdrohhrgheq necuggftrfgrthhtvghrnhepleeiiedvjeegtdffveegteevvddtueefieeufeefjedtue duiedvieduiefhieffnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghi lhhfrhhomhephihurhhisegrvghtvghrnhdrohhrgh X-ME-Proxy: Feedback-ID: i0d79475b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 8 Jun 2023 12:30:22 -0400 (EDT) Message-ID: <0b352a7d-0365-5c6b-7831-cebd62b9725d@aetern.org> Date: Thu, 8 Jun 2023 18:30:20 +0200 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.11.2 Subject: Re: CURRRENT snapshot won't boot due missing ZFS feature Content-Language: en-US From: Yuri To: freebsd-current@freebsd.org References: <20230608181115.7c97ea19.grembo@freebsd.org> <46816dca-c4a9-6e12-6057-f7fcd642eafd@aetern.org> In-Reply-To: <46816dca-c4a9-6e12-6057-f7fcd642eafd@aetern.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4QcV7N0t27z3G8s X-Spamd-Bar: - X-Spamd-Result: default: False [-1.50 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.19:c]; R_DKIM_ALLOW(-0.20)[aetern.org:s=fm1,messagingengine.com:s=fm1]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.19:from]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[aetern.org:+,messagingengine.com:+]; local_wl_from(0.00)[yuri@aetern.org]; DMARC_NA(0.00)[aetern.org] X-Rspamd-Pre-Result: action=no action; module=multimap; Matched map: local_wl_from X-ThisMailContainsUnwantedMimeParts: N Yuri wrote: > Michael Gmelin wrote: >> Hi, >> >> I didn't dig into this yet. >> >> After installing the current 14-snapshot (June 1st) in a bhyve-vm, I >> get this on boot: >> >> ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2 >> >> (booting stops at this point) >> >> Seems like the boot loader is missing this recently added feature. > > Are you sure it was June 1's? I saw this problem on: > > FreeBSD-14.0-CURRENT-amd64-20230427-60167184abd5-262599-disc1.iso > > ...but it was fixed since (for me, at least): > > FreeBSD-14.0-CURRENT-amd64-20230504-4194bbb34c60-262746-disc1.iso Trying to remember, I think I hit "send" too soon, it was 20230504 actually with a problem, and I think I had to use the previous one to install. Sorry for the noise. From nobody Thu Jun 8 17:06:23 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcVyW5R9Hz4c4sH for ; Thu, 8 Jun 2023 17:07:47 +0000 (UTC) (envelope-from grembo@freebsd.org) Received: from mail.evolve.de (mail.evolve.de [213.239.217.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.evolve.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcVyV6dkCz3PK5; Thu, 8 Jun 2023 17:07:46 +0000 (UTC) (envelope-from grembo@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: by mail.evolve.de (OpenSMTPD) with ESMTP id 98a045df; Thu, 8 Jun 2023 17:07:45 +0000 (UTC) Received: by mail.evolve.de (OpenSMTPD) with ESMTPSA id e605886f (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 8 Jun 2023 17:07:45 +0000 (UTC) Date: Thu, 8 Jun 2023 19:06:23 +0200 From: Michael Gmelin To: Glen Barber Cc: Michael Gmelin , "freebsd-current@freebsd.org" Subject: Re: CURRRENT snapshot won't boot due missing ZFS feature Message-ID: <20230608190623.14ab4ba8.grembo@freebsd.org> In-Reply-To: <20230608162012.GV1219@FreeBSD.org> References: <20230608181115.7c97ea19.grembo@freebsd.org> <20230608162012.GV1219@FreeBSD.org> X-Face: $wrgCtfdVw_H9WAY?S&9+/F"!41z'L$uo*WzT8miX?kZ~W~Lr5W7v?j0Sde\mwB&/ypo^}> +a'4xMc^^KroE~+v^&^#[B">soBo1y6(TW6#UZiC]o>C6`ej+i Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWJBwe5BQDl LASZU0/LTEWEfHbyj0Txi32+sKrp1Mv944X8/fm1rS+cAAAACXBIWXMAAAsTAAAL EwEAmpwYAAAAB3RJTUUH3wESCxwC7OBhbgAAACFpVFh0Q29tbWVudAAAAAAAQ3Jl YXRlZCB3aXRoIFRoZSBHSU1QbbCXAAAAAghJREFUOMu11DFvEzEUAGCfEhBVFzuq AKkLd0O6VrIQsLXVSZXoWE5N1K3DobBBA9fQpRWc8OkWouaIjedWKiyREOKs+3PY fvalCNjgLVHeF7/3bMtBzV8C/VsQ8tecEgCcDgrzjekwKZ7TwsJZd/ywEKwwP+ZM 8P3drTsAwWn2mpWuDDuYiK1bFs6De0KUUFw0tWxm+D4AIhuuvZqtyWYeO7jQ4Aea 7jUqI+ixhQoHex4WshEvSXdood7stlv4oSuFOC4tqGcr0NjEqXgV4mMJO38nld4+ xKNxRDon7khyKVqY7YR4d+Cg0OMrkWXZOM7YDkEfKiilCn1qYv4mighZiynuHHOA Wq9QJq+BIES7lMFUtcikMnkDGHUoncA+uHgrP0ctIEqfwLHzeSo+eUA66AqzwN6n 2ZHJhw6Qh/PoyC/QENyEyC/AyNjq74Bs+3UH0xYwzDUC4B97HgLocg1QLYgDDO1v f3UX9Y307Ew4AHh67YAFFsxEpkXwpXY3eIgMhAAE3R19L919nNnuD2wlPcDE3UeT L2ytEICQib9BXgS2fU8PrD82ToYO1OEmMSnYTjSqSv9wdC0tPYC+rQRQD9ESnldF CyqfmiYW+tlALt8gH2xrMdC/youbjzPXEun+/ReXsMCDyve3dZc09fn2Oas8oXGc Jj6/fOeK5UmSMPmf/jL+GD8BEj0k/Fn6IO4AAAAASUVORK5CYII= List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4QcVyV6dkCz3PK5 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:213.239.192.0/18, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Thu, 8 Jun 2023 16:20:12 +0000 Glen Barber wrote: > On Thu, Jun 08, 2023 at 06:11:15PM +0200, Michael Gmelin wrote: > > Hi, > > > > I didn't dig into this yet. > > > > After installing the current 14-snapshot (June 1st) in a bhyve-vm, I > > get this on boot: > > > > ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2 > > > > (booting stops at this point) > > > > Seems like the boot loader is missing this recently added feature. > > > > Can you try today's snapshot? They are propagated to most mirrors > now. > Tried today's snaphot, same problem. # reboot Waiting (max 60 seconds) for system process `vnlru' to stop... done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining... 0 0 0 0 done All buffers synced. Uptime: 4m14s Consoles: userboot FreeBSD/amd64 User boot lua, Revision 1.2 ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2 ERROR: cannot open /boot/lua/loader.lua: no such file or directory. Type '?' for a list of commands, 'help' for more detailed help. OK That's after installing CURRENT in a fresh vm managed by vm-bhyve using bsdinstall's automatic ZFS option. Best Michael -- Michael Gmelin From nobody Thu Jun 8 17:13:00 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcW4q4zmDz4c5gm; Thu, 8 Jun 2023 17:13:15 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-lj1-f171.google.com (mail-lj1-f171.google.com [209.85.208.171]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcW4n6JNVz3Qh9; Thu, 8 Jun 2023 17:13:13 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.208.171 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com; dmarc=none Received: by mail-lj1-f171.google.com with SMTP id 38308e7fff4ca-2b1aecef94fso8208241fa.3; Thu, 08 Jun 2023 10:13:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686244391; x=1688836391; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=HvGs+Rb3KUBWag4xO6E/vNKtQ9seLfR11VouXLdHFfc=; b=eagwB09hizlUVB5sLmjQFDDHrb75UzqSZYVMNBgF1tawCPzspdkjl83aQZcmKYM537 0wcExBPkgf5y5QMUQPVJTcW3cRgnLIxhgNpBw0M0WeFk6iIiOFzkXX8zTQgINLsz7mhX u87z6z5M584VgIRoeqW6aDeHhUrcz551/fmhN6yeI343Py4/kidG2QZKat4/BOUAMHW5 HqdmmwSkXD5ZQbC2LcZ8KytL4qSkNW3PzIAq4Et3EbDa4bwzjtK4xv2bv8VY/pGiKzVI kEyWjIVhXx5D5wkmzEBdY+wCXlfJAV/68dxqhHQjJBuPeHtunoDM19WMQYaIslx4w406 zsTA== X-Gm-Message-State: AC+VfDySqg/6KYpngA0e1SEEzFtKARHWJtghdR4QTapnWoOYKAa3EK75 bf/Od0PjYus0zhhmcJVBF7p81kKaO7nkgHL/x/8sowshb4U= X-Google-Smtp-Source: ACHHUZ6xa7uF+mPTy2JUUM+/kOuWDl40LsP6n8A0l+m73oSGV5y/8snRUylBGCP9VeQfpOaLdt+w586bnTpjRw+CJCs= X-Received: by 2002:a2e:9907:0:b0:2a7:6e85:e287 with SMTP id v7-20020a2e9907000000b002a76e85e287mr3963851lji.45.1686244391245; Thu, 08 Jun 2023 10:13:11 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 From: Ed Maste Date: Thu, 8 Jun 2023 13:13:00 -0400 Message-ID: Subject: OpenSSL 3.0 in the base system update To: FreeBSD Current , FreeBSD Ports Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org,freebsd-ports@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.208.171:from]; ARC_NA(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[209.85.208.171:from]; BLOCKLISTDE_FAIL(0.00)[209.85.208.171:server fail]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[carpeddiem]; RCPT_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[freebsd.org]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4QcW4n6JNVz3Qh9 X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N As previously mentioned[1] FreeBSD 14.0 will include OpenSSL 3.0. We expect to merge the update to main in the near future (within the next week or two) and are ready for wider testing. Supported by the FreeBSD Foundation, Pierre Pronchery has been working on the update in the src tree, with assistance from Enji Cooper (ngie@), and me (emaste@). Thanks to Antoine Brodin (antoine@) and Muhammad Moinur Rahman (bofh@) for ports exp-runs and fixes/workarounds and to Dag-Erling (des@) for updating ldns in the base system. ## Base system compatibility status Most of the base system is ready for a seamless switch to OpenSSL 3.0. For several components we've added `-DOPENSSL_API_COMPAT=0x10100000L` to CFLAGS to specify the API version, which avoids deprecation warnings from OpenSSL 3.0. Changes have also been made to avoid OpenSSL APIs already deprecated in OpenSSL 1.1. We can continue the process of updating to contemporary APIs after OpenSSL 3.0 is in the tree. Additional changes are still required for libarchive and seven Kerberos-related libraries or tools. Workarounds are ready to go along with the OpenSSL 3 import, and proper fixes are in progress in the upstream projects. A segfault from `openssl x509` in the i386 ports exp-run is under investigation and needs to be addressed prior to the merge. ## Ports compatibility With bofh@'s recent www/node18 and www/node20 patches the ports tree is in reasonable shape for OpenSSL 3.0 in the base system. The exp-run (link below) has a list of the failing ports, and I've emailed all of the maintainers as a heads-up. None of the remaining failures are responsible for a large number of skipped ports (i.e., the failures are either leaf ports or are responsible for only a small number of skipped ports). I expect that some or many of these will need to be addressed after the change lands in the src tree. ## Call for testing We welcome feedback from anyone willing to test the work in progress. Pierre's update can be obtained from the pull request[2] or by fetching the branch[3]. If desired I will provide a large diff against main. ## Links - Base system OpenSSL 3.0 update tracking PR: https://bugs.freebsd.org/271615 - Ports exp-run with OpenSSL 3.0 in the base system: https://bugs.freebsd.org/271656 [1] https://lists.freebsd.org/archives/freebsd-current/2023-May/003609.html [2] https://github.com/freebsd/freebsd-src/pull/760 [3] https://github.com/khorben/freebsd-src/tree/khorben/openssl-3.0.9 From nobody Thu Jun 8 17:16:37 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcWBL1Nj4z4c63M for ; Thu, 8 Jun 2023 17:18:02 +0000 (UTC) (envelope-from grembo@freebsd.org) Received: from mail.evolve.de (mail.evolve.de [213.239.217.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.evolve.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcWBK5MRVz3jWk; Thu, 8 Jun 2023 17:18:01 +0000 (UTC) (envelope-from grembo@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: by mail.evolve.de (OpenSMTPD) with ESMTP id c9e395d3; Thu, 8 Jun 2023 17:18:00 +0000 (UTC) Received: by mail.evolve.de (OpenSMTPD) with ESMTPSA id 6efabc36 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 8 Jun 2023 17:18:00 +0000 (UTC) Date: Thu, 8 Jun 2023 19:16:37 +0200 From: Michael Gmelin To: Glen Barber Cc: Michael Gmelin , "freebsd-current@freebsd.org" Subject: Re: CURRRENT snapshot won't boot due missing ZFS feature Message-ID: <20230608191637.21101c39.grembo@freebsd.org> In-Reply-To: <20230608190623.14ab4ba8.grembo@freebsd.org> References: <20230608181115.7c97ea19.grembo@freebsd.org> <20230608162012.GV1219@FreeBSD.org> <20230608190623.14ab4ba8.grembo@freebsd.org> X-Face: $wrgCtfdVw_H9WAY?S&9+/F"!41z'L$uo*WzT8miX?kZ~W~Lr5W7v?j0Sde\mwB&/ypo^}> +a'4xMc^^KroE~+v^&^#[B">soBo1y6(TW6#UZiC]o>C6`ej+i Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWJBwe5BQDl LASZU0/LTEWEfHbyj0Txi32+sKrp1Mv944X8/fm1rS+cAAAACXBIWXMAAAsTAAAL EwEAmpwYAAAAB3RJTUUH3wESCxwC7OBhbgAAACFpVFh0Q29tbWVudAAAAAAAQ3Jl YXRlZCB3aXRoIFRoZSBHSU1QbbCXAAAAAghJREFUOMu11DFvEzEUAGCfEhBVFzuq AKkLd0O6VrIQsLXVSZXoWE5N1K3DobBBA9fQpRWc8OkWouaIjedWKiyREOKs+3PY fvalCNjgLVHeF7/3bMtBzV8C/VsQ8tecEgCcDgrzjekwKZ7TwsJZd/ywEKwwP+ZM 8P3drTsAwWn2mpWuDDuYiK1bFs6De0KUUFw0tWxm+D4AIhuuvZqtyWYeO7jQ4Aea 7jUqI+ixhQoHex4WshEvSXdood7stlv4oSuFOC4tqGcr0NjEqXgV4mMJO38nld4+ xKNxRDon7khyKVqY7YR4d+Cg0OMrkWXZOM7YDkEfKiilCn1qYv4mighZiynuHHOA Wq9QJq+BIES7lMFUtcikMnkDGHUoncA+uHgrP0ctIEqfwLHzeSo+eUA66AqzwN6n 2ZHJhw6Qh/PoyC/QENyEyC/AyNjq74Bs+3UH0xYwzDUC4B97HgLocg1QLYgDDO1v f3UX9Y307Ew4AHh67YAFFsxEpkXwpXY3eIgMhAAE3R19L919nNnuD2wlPcDE3UeT L2ytEICQib9BXgS2fU8PrD82ToYO1OEmMSnYTjSqSv9wdC0tPYC+rQRQD9ESnldF CyqfmiYW+tlALt8gH2xrMdC/youbjzPXEun+/ReXsMCDyve3dZc09fn2Oas8oXGc Jj6/fOeK5UmSMPmf/jL+GD8BEj0k/Fn6IO4AAAAASUVORK5CYII= List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4QcWBK5MRVz3jWk X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:213.239.192.0/18, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Thu, 8 Jun 2023 19:06:23 +0200 Michael Gmelin wrote: > On Thu, 8 Jun 2023 16:20:12 +0000 > Glen Barber wrote: > > > On Thu, Jun 08, 2023 at 06:11:15PM +0200, Michael Gmelin wrote: > > > Hi, > > > > > > I didn't dig into this yet. > > > > > > After installing the current 14-snapshot (June 1st) in a > > > bhyve-vm, I get this on boot: > > > > > > ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2 > > > > > > (booting stops at this point) > > > > > > Seems like the boot loader is missing this recently added feature. > > > > > > > Can you try today's snapshot? They are propagated to most mirrors > > now. > > > > Tried today's snaphot, same problem. > > # reboot > Waiting (max 60 seconds) for system process `vnlru' to stop... done > Waiting (max 60 seconds) for system process `syncer' to stop... > Syncing disks, vnodes remaining... 0 0 0 0 done > All buffers synced. > Uptime: 4m14s > Consoles: userboot > > FreeBSD/amd64 User boot lua, Revision 1.2 > ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2 > ERROR: cannot open /boot/lua/loader.lua: no such file or directory. > > > Type '?' for a list of commands, 'help' for more detailed help. > OK > > > That's after installing CURRENT in a fresh vm managed by vm-bhyve > using bsdinstall's automatic ZFS option. > Thinking about this, it's possible that vm-bhyve is using the zfs boot loader from the host machine. Please consider this noise, unless you hear from me again. Best Michael -- Michael Gmelin From nobody Thu Jun 8 17:32:19 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcWW42BrVz4c7wX for ; Thu, 8 Jun 2023 17:32:32 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcWW40WRxz3p53 for ; Thu, 8 Jun 2023 17:32:32 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x62b.google.com with SMTP id a640c23a62f3a-977ed383b8aso157567566b.3 for ; Thu, 08 Jun 2023 10:32:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20221208.gappssmtp.com; s=20221208; t=1686245550; x=1688837550; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=QBtibLup0uETIsqYBKWu/L8lD8kM/QluDhAoDacViLs=; b=Qu9bfWJBZjxCZKkU50svkenCAB3L4Ei5MFkTX/CHFXzCYhrulq6Ntva+iKONETUEV9 /SX2Y9qCXtUqByjK0Ct+oBKN7lIILRrg9acuFeb2sPTuY/YI7Y9GxuW8+rfn0rZtt5tr scIDEjFmtAH8W+QWYI4c/+zDMxdSrzlO2b5su+Gi1XTPWSWKc73IQc3QqNU99RHkAtc3 M+az+GISrgn5hcsrim/6wxhpKNo562TTucDHwki0/d/LJ8ZWRJ1nVqyXIcrrDXgOI4EG 9lTejKmci5QOPmZRZU399W65Kq0QOG0ncp0itXmCMALsUrDFAkrYZqTmVErc15mOQh1p H6FA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686245550; x=1688837550; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QBtibLup0uETIsqYBKWu/L8lD8kM/QluDhAoDacViLs=; b=idQU0FJKNoL+qnu1N1o136IB+2rhyVjeVLBDRW/Bh9vrpvPL3/cbw3lYSh49UKZnS1 YnJXWqFysNYPnyUsEn+G+En7vWYKEYdz9m/3t6584Ao8g8+eSd0dQ1ov1JVNk0XFsM+t QxOXeTCJZSPPbI1XOPvJNG9laYqahP4NqigFrhakEsqJa11UR/0WVneK8E2bhgViQ2db VI/tSQgLh5fjpWMF5DBgFh2gwWeWKLdp8No7MdwY5d9w2wfO68dHfP1tB2gQiFhrM6nG PvlZ4o91mlDhftJCWrEWiKXmMMgpJQzKaSPFJ97t5sQ5R94UXB/rhK+al6PVQGmaAZub 2Jkg== X-Gm-Message-State: AC+VfDzkjmi1FjEpK16931lO/o14Uv9w9XBTT7KZsZLaBIUbFbH17Y5O 3QArG7vAAMeTDbkyCVF12XRyT1Ir1+p6m6er2Iy3NpdAu6Jy7Oga X-Google-Smtp-Source: ACHHUZ5KxH4rRZHus92qkxupcF9pfVB+M+iO1WnGrbf0C1ByIHbGQxTvBb3SYkfQG42iutZAS2IevrLZwfrecgp48cw= X-Received: by 2002:a17:906:db0e:b0:974:6026:a315 with SMTP id xj14-20020a170906db0e00b009746026a315mr296172ejb.44.1686245550404; Thu, 08 Jun 2023 10:32:30 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <20230608181115.7c97ea19.grembo@freebsd.org> <20230608162012.GV1219@FreeBSD.org> <20230608190623.14ab4ba8.grembo@freebsd.org> <20230608191637.21101c39.grembo@freebsd.org> In-Reply-To: <20230608191637.21101c39.grembo@freebsd.org> From: Warner Losh Date: Thu, 8 Jun 2023 11:32:19 -0600 Message-ID: Subject: Re: CURRRENT snapshot won't boot due missing ZFS feature To: Michael Gmelin Cc: Glen Barber , FreeBSD Current Content-Type: multipart/alternative; boundary="000000000000094fb105fda1a332" X-Rspamd-Queue-Id: 4QcWW40WRxz3p53 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000094fb105fda1a332 Content-Type: text/plain; charset="UTF-8" On Thu, Jun 8, 2023, 11:18 AM Michael Gmelin wrote: > > > On Thu, 8 Jun 2023 19:06:23 +0200 > Michael Gmelin wrote: > > > On Thu, 8 Jun 2023 16:20:12 +0000 > > Glen Barber wrote: > > > > > On Thu, Jun 08, 2023 at 06:11:15PM +0200, Michael Gmelin wrote: > > > > Hi, > > > > > > > > I didn't dig into this yet. > > > > > > > > After installing the current 14-snapshot (June 1st) in a > > > > bhyve-vm, I get this on boot: > > > > > > > > ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2 > > > > > > > > (booting stops at this point) > > > > > > > > Seems like the boot loader is missing this recently added feature. > > > > > > > > > > Can you try today's snapshot? They are propagated to most mirrors > > > now. > > > > > > > Tried today's snaphot, same problem. > > > > # reboot > > Waiting (max 60 seconds) for system process `vnlru' to stop... done > > Waiting (max 60 seconds) for system process `syncer' to stop... > > Syncing disks, vnodes remaining... 0 0 0 0 done > > All buffers synced. > > Uptime: 4m14s > > Consoles: userboot > > > > FreeBSD/amd64 User boot lua, Revision 1.2 > > ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2 > > ERROR: cannot open /boot/lua/loader.lua: no such file or directory. > > > > > > Type '?' for a list of commands, 'help' for more detailed help. > > OK > > > > > > That's after installing CURRENT in a fresh vm managed by vm-bhyve > > using bsdinstall's automatic ZFS option. > > > > Thinking about this, it's possible that vm-bhyve is using the zfs boot > loader from the host machine. > > Please consider this noise, unless you hear from me again. > Yes. It does. This can be an unfortunate design choice at times. Warner Best > Michael > > -- > Michael Gmelin > > --000000000000094fb105fda1a332 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Thu, Jun 8, 2023, 11:18 AM Michael Gmelin <grembo@freebsd.org> wrote:


On Thu, 8 Jun 2023 19:06:23 +0200
Michael Gmelin <grembo@freebsd.org> wrote:

> On Thu, 8 Jun 2023 16:20:12 +0000
> Glen Barber <gjb@freebsd.org> wrote:
>
> > On Thu, Jun 08, 2023 at 06:11:15PM +0200, Michael Gmelin wrote:= =C2=A0
> > > Hi,
> > >
> > > I didn't dig into this yet.
> > >
> > > After installing the current 14-snapshot (June 1st) in a
> > > bhyve-vm, I get this on boot:
> > >
> > >=C2=A0 =C2=A0ZFS: unsupported feature: com.klarasystems:vdev_= zaps_v2
> > >
> > > (booting stops at this point)
> > >
> > > Seems like the boot loader is missing this recently added fe= ature.
> > >=C2=A0 =C2=A0 =C2=A0
> >
> > Can you try today's snapshot?=C2=A0 They are propagated to mo= st mirrors
> > now.
> >=C2=A0 =C2=A0
>
> Tried today's snaphot, same problem.
>
>=C2=A0 =C2=A0# reboot
>=C2=A0 =C2=A0Waiting (max 60 seconds) for system process `vnlru' to= stop... done
>=C2=A0 =C2=A0Waiting (max 60 seconds) for system process `syncer' t= o stop...
>=C2=A0 =C2=A0Syncing disks, vnodes remaining... 0 0 0 0 done
>=C2=A0 =C2=A0All buffers synced.
>=C2=A0 =C2=A0Uptime: 4m14s
>=C2=A0 =C2=A0Consoles: userboot=C2=A0
>
>=C2=A0 =C2=A0FreeBSD/amd64 User boot lua, Revision 1.2
>=C2=A0 =C2=A0ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2 >=C2=A0 =C2=A0ERROR: cannot open /boot/lua/loader.lua: no such file or d= irectory.
>
>
>=C2=A0 =C2=A0Type '?' for a list of commands, 'help' fo= r more detailed help.
>=C2=A0 =C2=A0OK
>
>
> That's after installing CURRENT in a fresh vm managed by vm-bhyve<= br> > using bsdinstall's automatic ZFS option.
>

Thinking about this, it's possible that vm-bhyve is using the zfs boot<= br> loader from the host machine.

Please consider this noise, unless you hear from me again.
=

Yes. It does. Thi= s can be an unfortunate design choice at times.=C2=A0

Warner

Best
Michael

--
Michael Gmelin

--000000000000094fb105fda1a332-- From nobody Thu Jun 8 20:45:51 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QcbpH68jrz4bSqm for ; Thu, 8 Jun 2023 20:45:59 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-4018.proton.ch (mail-4018.proton.ch [185.70.40.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QcbpH2ky9z4Jkq; Thu, 8 Jun 2023 20:45:59 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; none Date: Thu, 08 Jun 2023 20:45:51 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail; t=1686257155; x=1686516355; bh=vjFFLMABMJzE5aEJ5FZOcuM7kDNd16BgYkQ1Nusq/ro=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=UBj4jRqjWcqrteZKwqNFbWAQysau+gUg+kipx1KiTw2Ngky+jsLVo3rEQbmgEOEFk RMoi/EETDhQ7SweB9jJyuK+RxgtCF0a0Igg0WY2uFbAHBZ2q+ZIKNMMReIPutS0cuy eIietaTWgQY8gmPPdHvPjOoROx16igwPjYgpqVLEQpyZbz/FRg6Ckg94HvyWcmyzdw UAz5MeoN/H9phBCPFWNOeKfFcCin7Pk4Ng3yvP3Q7qO+bmuBzmuRBlgZPBIUXoMXz9 BiQjtmpslFA2Zg2tIhGg7FPMpqTdGhBnu61AIpFV1H8yk0ZdRVwkbwbskS49Oo81L8 CUqTc2PeAqR5w== To: emaste@freebsd.org, freebsd-current@freebsd.org From: Jonathan Vasquez Subject: Re: OpenSSL 3.0 in the base system update Message-ID: In-Reply-To: References: Feedback-ID: 12351801:user:proton List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="b1_FWxjk1EGnCgmCsCYXdJ0lYCS3lWdpZD5IZSL1gMKjs" X-Rspamd-Queue-Id: 4QcbpH2ky9z4Jkq X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:62371, ipnet:185.70.40.0/24, country:CH] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --b1_FWxjk1EGnCgmCsCYXdJ0lYCS3lWdpZD5IZSL1gMKjs Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 VGhhbmtzIGZvciB0aGUgaGFyZCB3b3JrIG9uIHRoaXMgRWQgYW5kIENvLiBJJ2xsIGNvbnRpbnVl IHRvIGtlZXAgYW4gZXllIG9uIHRoaXMgYW5kIHRlc3QgQ1VSUkVOVCBhIGJpdCB3aGVuIG1vcmUg dGhpbmdzIGhpdCB0aGUgdHJlZS4KCkpvbmF0aGFuIFZhc3F1ZXoKUEdQOiAzNERBIDg1OEMgMTQ0 NyA1MDlFIEM3N0EgRDQ5RiBGQjg1IDkwQjcgQzRDQSA1Mjc5ClNlbnQgd2l0aCBQcm90b25NYWls IFNlY3VyZSBFbWFpbAoKU2VudCBmcm9tIFByb3RvbiBNYWlsIG1vYmlsZQoKLS0tLS0tLS0gT3Jp Z2luYWwgTWVzc2FnZSAtLS0tLS0tLQpPbiBKdW4gOCwgMjAyMywgMTM6MTMsIEVkIE1hc3RlIHdy b3RlOgoKPiBBcyBwcmV2aW91c2x5IG1lbnRpb25lZFsxXSBGcmVlQlNEIDE0LjAgd2lsbCBpbmNs dWRlIE9wZW5TU0wgMy4wLiBXZSBleHBlY3QgdG8gbWVyZ2UgdGhlIHVwZGF0ZSB0byBtYWluIGlu IHRoZSBuZWFyIGZ1dHVyZSAod2l0aGluIHRoZSBuZXh0IHdlZWsgb3IgdHdvKSBhbmQgYXJlIHJl YWR5IGZvciB3aWRlciB0ZXN0aW5nLiBTdXBwb3J0ZWQgYnkgdGhlIEZyZWVCU0QgRm91bmRhdGlv biwgUGllcnJlIFByb25jaGVyeSBoYXMgYmVlbiB3b3JraW5nIG9uIHRoZSB1cGRhdGUgaW4gdGhl IHNyYyB0cmVlLCB3aXRoIGFzc2lzdGFuY2UgZnJvbSBFbmppIENvb3BlciAobmdpZUApLCBhbmQg bWUgKGVtYXN0ZUApLiBUaGFua3MgdG8gQW50b2luZSBCcm9kaW4gKGFudG9pbmVAKSBhbmQgTXVo YW1tYWQgTW9pbnVyIFJhaG1hbiAoYm9maEApIGZvciBwb3J0cyBleHAtcnVucyBhbmQgZml4ZXMv d29ya2Fyb3VuZHMgYW5kIHRvIERhZy1FcmxpbmcgKGRlc0ApIGZvciB1cGRhdGluZyBsZG5zIGlu IHRoZSBiYXNlIHN5c3RlbS4gIyMgQmFzZSBzeXN0ZW0gY29tcGF0aWJpbGl0eSBzdGF0dXMgTW9z dCBvZiB0aGUgYmFzZSBzeXN0ZW0gaXMgcmVhZHkgZm9yIGEgc2VhbWxlc3Mgc3dpdGNoIHRvIE9w ZW5TU0wgMy4wLiBGb3Igc2V2ZXJhbCBjb21wb25lbnRzIHdlJ3ZlIGFkZGVkIGAtRE9QRU5TU0xf QVBJX0NPTVBBVD0weDEwMTAwMDAwTGAgdG8gQ0ZMQUdTIHRvIHNwZWNpZnkgdGhlIEFQSSB2ZXJz aW9uLCB3aGljaCBhdm9pZHMgZGVwcmVjYXRpb24gd2FybmluZ3MgZnJvbSBPcGVuU1NMIDMuMC4g Q2hhbmdlcyBoYXZlIGFsc28gYmVlbiBtYWRlIHRvIGF2b2lkIE9wZW5TU0wgQVBJcyBhbHJlYWR5 IGRlcHJlY2F0ZWQgaW4gT3BlblNTTCAxLjEuIFdlIGNhbiBjb250aW51ZSB0aGUgcHJvY2VzcyBv ZiB1cGRhdGluZyB0byBjb250ZW1wb3JhcnkgQVBJcyBhZnRlciBPcGVuU1NMIDMuMCBpcyBpbiB0 aGUgdHJlZS4gQWRkaXRpb25hbCBjaGFuZ2VzIGFyZSBzdGlsbCByZXF1aXJlZCBmb3IgbGliYXJj aGl2ZSBhbmQgc2V2ZW4gS2VyYmVyb3MtcmVsYXRlZCBsaWJyYXJpZXMgb3IgdG9vbHMuIFdvcmth cm91bmRzIGFyZSByZWFkeSB0byBnbyBhbG9uZyB3aXRoIHRoZSBPcGVuU1NMIDMgaW1wb3J0LCBh bmQgcHJvcGVyIGZpeGVzIGFyZSBpbiBwcm9ncmVzcyBpbiB0aGUgdXBzdHJlYW0gcHJvamVjdHMu IEEgc2VnZmF1bHQgZnJvbSBgb3BlbnNzbCB4NTA5YCBpbiB0aGUgaTM4NiBwb3J0cyBleHAtcnVu IGlzIHVuZGVyIGludmVzdGlnYXRpb24gYW5kIG5lZWRzIHRvIGJlIGFkZHJlc3NlZCBwcmlvciB0 byB0aGUgbWVyZ2UuICMjIFBvcnRzIGNvbXBhdGliaWxpdHkgV2l0aCBib2ZoQCdzIHJlY2VudCB3 d3cvbm9kZTE4IGFuZCB3d3cvbm9kZTIwIHBhdGNoZXMgdGhlIHBvcnRzIHRyZWUgaXMgaW4gcmVh c29uYWJsZSBzaGFwZSBmb3IgT3BlblNTTCAzLjAgaW4gdGhlIGJhc2Ugc3lzdGVtLiBUaGUgZXhw LXJ1biAobGluayBiZWxvdykgaGFzIGEgbGlzdCBvZiB0aGUgZmFpbGluZyBwb3J0cywgYW5kIEkn dmUgZW1haWxlZCBhbGwgb2YgdGhlIG1haW50YWluZXJzIGFzIGEgaGVhZHMtdXAuIE5vbmUgb2Yg dGhlIHJlbWFpbmluZyBmYWlsdXJlcyBhcmUgcmVzcG9uc2libGUgZm9yIGEgbGFyZ2UgbnVtYmVy IG9mIHNraXBwZWQgcG9ydHMgKGkuZS4sIHRoZSBmYWlsdXJlcyBhcmUgZWl0aGVyIGxlYWYgcG9y dHMgb3IgYXJlIHJlc3BvbnNpYmxlIGZvciBvbmx5IGEgc21hbGwgbnVtYmVyIG9mIHNraXBwZWQg cG9ydHMpLiBJIGV4cGVjdCB0aGF0IHNvbWUgb3IgbWFueSBvZiB0aGVzZSB3aWxsIG5lZWQgdG8g YmUgYWRkcmVzc2VkIGFmdGVyIHRoZSBjaGFuZ2UgbGFuZHMgaW4gdGhlIHNyYyB0cmVlLiAjIyBD YWxsIGZvciB0ZXN0aW5nIFdlIHdlbGNvbWUgZmVlZGJhY2sgZnJvbSBhbnlvbmUgd2lsbGluZyB0 byB0ZXN0IHRoZSB3b3JrIGluIHByb2dyZXNzLiBQaWVycmUncyB1cGRhdGUgY2FuIGJlIG9idGFp bmVkIGZyb20gdGhlIHB1bGwgcmVxdWVzdFsyXSBvciBieSBmZXRjaGluZyB0aGUgYnJhbmNoWzNd LiBJZiBkZXNpcmVkIEkgd2lsbCBwcm92aWRlIGEgbGFyZ2UgZGlmZiBhZ2FpbnN0IG1haW4uICMj IExpbmtzIC0gQmFzZSBzeXN0ZW0gT3BlblNTTCAzLjAgdXBkYXRlIHRyYWNraW5nIFBSOiBodHRw czovL2J1Z3MuZnJlZWJzZC5vcmcvMjcxNjE1IC0gUG9ydHMgZXhwLXJ1biB3aXRoIE9wZW5TU0wg My4wIGluIHRoZSBiYXNlIHN5c3RlbTogaHR0cHM6Ly9idWdzLmZyZWVic2Qub3JnLzI3MTY1NiBb MV0gaHR0cHM6Ly9saXN0cy5mcmVlYnNkLm9yZy9hcmNoaXZlcy9mcmVlYnNkLWN1cnJlbnQvMjAy My1NYXkvMDAzNjA5Lmh0bWwgWzJdIGh0dHBzOi8vZ2l0aHViLmNvbS9mcmVlYnNkL2ZyZWVic2Qt c3JjL3B1bGwvNzYwIFszXSBodHRwczovL2dpdGh1Yi5jb20va2hvcmJlbi9mcmVlYnNkLXNyYy90 cmVlL2tob3JiZW4vb3BlbnNzbC0zLjAuOQ== --b1_FWxjk1EGnCgmCsCYXdJ0lYCS3lWdpZD5IZSL1gMKjs Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 VGhhbmtzIGZvciB0aGUgaGFyZCB3b3JrIG9uIHRoaXMgRWQgYW5kIENvLiBJJ2xsIGNvbnRpbnVl IHRvIGtlZXAgYW4gZXllIG9uIHRoaXMgYW5kIHRlc3QgQ1VSUkVOVCBhIGJpdCB3aGVuIG1vcmUg dGhpbmdzIGhpdCB0aGUgdHJlZS48YnI+PGJyPjxicj48ZGl2PkpvbmF0aGFuIFZhc3F1ZXo8YnIg Lz48L2Rpdj48ZGl2PlBHUDogMzREQSA4NThDIDE0NDcgNTA5RSBDNzdBICBENDlGIEZCODUgOTBC NyBDNENBIDUyNzk8YnIgLz48L2Rpdj48ZGl2PlNlbnQgd2l0aCBQcm90b25NYWlsIFNlY3VyZSBF bWFpbDxiciAvPjwvZGl2PjxkaXY+PGJyIC8+PC9kaXY+PGJyPjxicj5TZW50IGZyb20gUHJvdG9u IE1haWwgbW9iaWxlPGJyPjxicj48YnI+PGJyPi0tLS0tLS0tIE9yaWdpbmFsIE1lc3NhZ2UgLS0t LS0tLS08YnI+T24gSnVuIDgsIDIwMjMsIDEzOjEzLCBFZCBNYXN0ZSA8IGVtYXN0ZUBmcmVlYnNk Lm9yZz4gd3JvdGU6PGJsb2NrcXVvdGUgY2xhc3M9InByb3Rvbm1haWxfcXVvdGUiPjxicj5BcyBw cmV2aW91c2x5IG1lbnRpb25lZFsxXSBGcmVlQlNEIDE0LjAgd2lsbCBpbmNsdWRlIE9wZW5TU0wg My4wLiAgV2UNCmV4cGVjdCB0byBtZXJnZSB0aGUgdXBkYXRlIHRvIG1haW4gaW4gdGhlIG5lYXIg ZnV0dXJlICh3aXRoaW4gdGhlIG5leHQNCndlZWsgb3IgdHdvKSBhbmQgYXJlIHJlYWR5IGZvciB3 aWRlciB0ZXN0aW5nLg0KDQpTdXBwb3J0ZWQgYnkgdGhlIEZyZWVCU0QgRm91bmRhdGlvbiwgUGll cnJlIFByb25jaGVyeSBoYXMgYmVlbiB3b3JraW5nDQpvbiB0aGUgdXBkYXRlIGluIHRoZSBzcmMg dHJlZSwgd2l0aCBhc3Npc3RhbmNlIGZyb20gRW5qaSBDb29wZXINCihuZ2llQCksIGFuZCBtZSAo ZW1hc3RlQCkuIFRoYW5rcyB0byBBbnRvaW5lIEJyb2RpbiAoYW50b2luZUApIGFuZA0KTXVoYW1t YWQgTW9pbnVyIFJhaG1hbiAoYm9maEApIGZvciBwb3J0cyBleHAtcnVucyBhbmQNCmZpeGVzL3dv cmthcm91bmRzIGFuZCB0byBEYWctRXJsaW5nIChkZXNAKSBmb3IgdXBkYXRpbmcgbGRucyBpbiB0 aGUNCmJhc2Ugc3lzdGVtLg0KDQojIyBCYXNlIHN5c3RlbSBjb21wYXRpYmlsaXR5IHN0YXR1cw0K DQpNb3N0IG9mIHRoZSBiYXNlIHN5c3RlbSBpcyByZWFkeSBmb3IgYSBzZWFtbGVzcyBzd2l0Y2gg dG8gT3BlblNTTCAzLjAuDQpGb3Igc2V2ZXJhbCBjb21wb25lbnRzIHdlJ3ZlIGFkZGVkIGAtRE9Q RU5TU0xfQVBJX0NPTVBBVD0weDEwMTAwMDAwTGANCnRvIENGTEFHUyB0byBzcGVjaWZ5IHRoZSBB UEkgdmVyc2lvbiwgd2hpY2ggYXZvaWRzIGRlcHJlY2F0aW9uDQp3YXJuaW5ncyBmcm9tIE9wZW5T U0wgMy4wLiBDaGFuZ2VzIGhhdmUgYWxzbyBiZWVuIG1hZGUgdG8gYXZvaWQNCk9wZW5TU0wgQVBJ cyBhbHJlYWR5IGRlcHJlY2F0ZWQgaW4gT3BlblNTTCAxLjEuIFdlIGNhbiBjb250aW51ZSB0aGUN CnByb2Nlc3Mgb2YgdXBkYXRpbmcgdG8gY29udGVtcG9yYXJ5IEFQSXMgYWZ0ZXIgT3BlblNTTCAz LjAgaXMgaW4gdGhlDQp0cmVlLg0KDQpBZGRpdGlvbmFsIGNoYW5nZXMgYXJlIHN0aWxsIHJlcXVp cmVkIGZvciBsaWJhcmNoaXZlIGFuZCBzZXZlbg0KS2VyYmVyb3MtcmVsYXRlZCBsaWJyYXJpZXMg b3IgdG9vbHMuIFdvcmthcm91bmRzIGFyZSByZWFkeSB0byBnbyBhbG9uZw0Kd2l0aCB0aGUgT3Bl blNTTCAzIGltcG9ydCwgYW5kIHByb3BlciBmaXhlcyBhcmUgaW4gcHJvZ3Jlc3MgaW4gdGhlDQp1 cHN0cmVhbSBwcm9qZWN0cy4NCg0KQSBzZWdmYXVsdCBmcm9tIGBvcGVuc3NsIHg1MDlgIGluIHRo ZSBpMzg2IHBvcnRzIGV4cC1ydW4gaXMgdW5kZXINCmludmVzdGlnYXRpb24gYW5kIG5lZWRzIHRv IGJlIGFkZHJlc3NlZCBwcmlvciB0byB0aGUgbWVyZ2UuDQoNCiMjIFBvcnRzIGNvbXBhdGliaWxp dHkNCg0KV2l0aCBib2ZoQCdzIHJlY2VudCB3d3cvbm9kZTE4IGFuZCB3d3cvbm9kZTIwIHBhdGNo ZXMgdGhlIHBvcnRzIHRyZWUNCmlzIGluIHJlYXNvbmFibGUgc2hhcGUgZm9yIE9wZW5TU0wgMy4w IGluIHRoZSBiYXNlIHN5c3RlbS4gVGhlIGV4cC1ydW4NCihsaW5rIGJlbG93KSBoYXMgYSBsaXN0 IG9mIHRoZSBmYWlsaW5nIHBvcnRzLCBhbmQgSSd2ZSBlbWFpbGVkIGFsbCBvZg0KdGhlIG1haW50 YWluZXJzIGFzIGEgaGVhZHMtdXAuIE5vbmUgb2YgdGhlIHJlbWFpbmluZyBmYWlsdXJlcyBhcmUN CnJlc3BvbnNpYmxlIGZvciBhIGxhcmdlIG51bWJlciBvZiBza2lwcGVkIHBvcnRzIChpLmUuLCB0 aGUgZmFpbHVyZXMNCmFyZSBlaXRoZXIgbGVhZiBwb3J0cyBvciBhcmUgcmVzcG9uc2libGUgZm9y IG9ubHkgYSBzbWFsbCBudW1iZXIgb2YNCnNraXBwZWQgcG9ydHMpLiBJIGV4cGVjdCB0aGF0IHNv bWUgb3IgbWFueSBvZiB0aGVzZSB3aWxsIG5lZWQgdG8gYmUNCmFkZHJlc3NlZCBhZnRlciB0aGUg Y2hhbmdlIGxhbmRzIGluIHRoZSBzcmMgdHJlZS4NCg0KIyMgQ2FsbCBmb3IgdGVzdGluZw0KDQpX ZSB3ZWxjb21lIGZlZWRiYWNrIGZyb20gYW55b25lIHdpbGxpbmcgdG8gdGVzdCB0aGUgd29yayBp biBwcm9ncmVzcy4NClBpZXJyZSdzIHVwZGF0ZSBjYW4gYmUgb2J0YWluZWQgZnJvbSB0aGUgcHVs bCByZXF1ZXN0WzJdIG9yIGJ5DQpmZXRjaGluZyB0aGUgYnJhbmNoWzNdLiBJZiBkZXNpcmVkIEkg d2lsbCBwcm92aWRlIGEgbGFyZ2UgZGlmZiBhZ2FpbnN0DQptYWluLg0KDQojIyBMaW5rcw0KDQot IEJhc2Ugc3lzdGVtIE9wZW5TU0wgMy4wIHVwZGF0ZSB0cmFja2luZyBQUjoNCiAgaHR0cHM6Ly9i dWdzLmZyZWVic2Qub3JnLzI3MTYxNQ0KDQotIFBvcnRzIGV4cC1ydW4gd2l0aCBPcGVuU1NMIDMu MCBpbiB0aGUgYmFzZSBzeXN0ZW06DQogIGh0dHBzOi8vYnVncy5mcmVlYnNkLm9yZy8yNzE2NTYN Cg0KWzFdIGh0dHBzOi8vbGlzdHMuZnJlZWJzZC5vcmcvYXJjaGl2ZXMvZnJlZWJzZC1jdXJyZW50 LzIwMjMtTWF5LzAwMzYwOS5odG1sDQpbMl0gaHR0cHM6Ly9naXRodWIuY29tL2ZyZWVic2QvZnJl ZWJzZC1zcmMvcHVsbC83NjANClszXSBodHRwczovL2dpdGh1Yi5jb20va2hvcmJlbi9mcmVlYnNk LXNyYy90cmVlL2tob3JiZW4vb3BlbnNzbC0zLjAuOQ0KDQo8L2Rpdj4= --b1_FWxjk1EGnCgmCsCYXdJ0lYCS3lWdpZD5IZSL1gMKjs-- From nobody Fri Jun 9 02:56:07 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qcm1X0slkz4bHKM for ; Fri, 9 Jun 2023 02:56:16 +0000 (UTC) (envelope-from glebius@freebsd.org) Received: from glebi.us (glebi.us [162.251.186.162]) by mx1.freebsd.org (Postfix) with ESMTP id 4Qcm1V58Jmz46RH; Fri, 9 Jun 2023 02:56:14 +0000 (UTC) (envelope-from glebius@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 162.251.186.162 is neither permitted nor denied by domain of glebius@freebsd.org) smtp.mailfrom=glebius@freebsd.org; dmarc=none Received: by glebi.us (Postfix, from userid 1000) id A64C6226B6; Thu, 8 Jun 2023 19:56:07 -0700 (PDT) Date: Thu, 8 Jun 2023 19:56:07 -0700 From: Gleb Smirnoff To: current@freebsd.org Cc: mm@freebsd.org Subject: panic(s) in ZFS on CURRENT Message-ID: List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Result: default: False [0.50 / 15.00]; VIOLATED_DIRECT_SPF(3.50)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; MLMMJ_DEST(0.00)[current@freebsd.org]; ASN(0.00)[asn:27348, ipnet:162.251.186.0/24, country:US]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[glebius]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[freebsd.org]; TO_DN_NONE(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Qcm1V58Jmz46RH X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N Hi, I got several panics on my desktop running eb2b00da564 which is after the latest OpenZFS merge. #1 (couple cores with this backtrace) --- trap 0x9, rip = 0xffffffff803ab94b, rsp = 0xfffffe022e45ed30, rbp = 0xfffffe022e45ed50 --- buf_hash_insert() at buf_hash_insert+0xab/frame 0xfffffe022e45ed50 arc_write_done() at arc_write_done+0xfa/frame 0xfffffe022e45ed90 zio_done() at zio_done+0xf0b/frame 0xfffffe022e45ee00 zio_execute() at zio_execute+0x9f/frame 0xfffffe022e45ee40 taskqueue_run_locked() at taskqueue_run_locked+0x181/frame 0xfffffe022e45eec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc3/frame 0xfffffe022e45eef0 fork_exit() at fork_exit+0x7d/frame 0xfffffe022e45ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe022e45ef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- (kgdb) frame 7 #7 buf_hash_insert (hdr=hdr@entry=0xfffff8001b21fa28, lockp=lockp@entry=0xfffffe022e45ed60) at /usr/src/FreeBSD/sys/contrib/openzfs/module/zfs/arc.c:1062 1062 if (HDR_EQUAL(hdr->b_spa, &hdr->b_dva, hdr->b_birth, fhdr)) (kgdb) p hdr $1 = (arc_buf_hdr_t *) 0xfffff8001b21fa28 (kgdb) p *hdr $2 = {b_dva = {dva_word = {16, 20406677952}}, b_birth = 38447120, b_type = ARC_BUFC_METADATA, b_complevel = 255 '\377', b_reserved1 = 0 '\000', b_reserved2 = 0, b_hash_next = 0x0, b_flags = (ARC_FLAG_L2CACHE | ARC_FLAG_IO_IN_PROGRESS | ARC_FLAG_BUFC_METADATA | ARC_FLAG_HAS_L1HDR | ARC_FLAG_COMPRESSED_ARC | ARC_FLAG_COMPRESS_0 | ARC_FLAG_COMPRESS_1 | ARC_FLAG_COMPRESS_2 | ARC_FLAG_COMPRESS_3), b_psize = 8, b_lsize = 32, b_spa = 1230587331341359116, b_l2hdr = { b_dev = 0x0, b_daddr = 0, b_hits = 0, b_arcs_state = ARC_STATE_ANON, b_l2node = {list_next = 0x0, list_prev = 0x0}}, b_l1hdr = {b_cv = { cv_description = 0xffffffff80bb5b02 "hdr->b_l1hdr.b_cv", cv_waiters = 0}, b_byteswap = 10 '\n', b_state = 0xffffffff80ef23c0 , b_arc_node = {list_next = 0x0, list_prev = 0x0}, b_arc_access = 0, b_mru_hits = 0, b_mru_ghost_hits = 0, b_mfu_hits = 0, b_mfu_ghost_hits = 0, b_bufcnt = 1, b_buf = 0xfffff80003139d80, b_refcnt = {rc_count = 2}, b_acb = 0x0, b_pabd = 0xfffff80a35dc6480}, b_crypt_hdr = {b_rabd = 0x10, b_ot = 2744191968, b_ebufcnt = 4, b_dsobj = 38340866, b_salt = "\001\000\000\000\000\000\000", b_iv = "\000\000\000\000\000\000\000\000\220\000\026\017", b_mac = "\b\000 \000\f\230\262m\250\354\023\021\000\000\000"}} #2 (single core) --- trap 0x9, rip = 0xffffffff803ab94b, rsp = 0xfffffe0256158780, rbp = 0xfffffe02561587a0 --- buf_hash_insert() at buf_hash_insert+0xab/frame 0xfffffe02561587a0 arc_hdr_realloc() at arc_hdr_realloc+0x138/frame 0xfffffe0256158800 arc_read() at arc_read+0x2dc/frame 0xfffffe02561588b0 dbuf_read() at dbuf_read+0xb3e/frame 0xfffffe02561589f0 dmu_buf_hold() at dmu_buf_hold+0x46/frame 0xfffffe0256158a30 zap_cursor_retrieve() at zap_cursor_retrieve+0x167/frame 0xfffffe0256158a90 zfs_freebsd_readdir() at zfs_freebsd_readdir+0x383/frame 0xfffffe0256158cc0 VOP_READDIR_APV() at VOP_READDIR_APV+0x1f/frame 0xfffffe0256158ce0 kern_getdirentries() at kern_getdirentries+0x186/frame 0xfffffe0256158dd0 sys_getdirentries() at sys_getdirentries+0x29/frame 0xfffffe0256158e00 amd64_syscall() at amd64_syscall+0x100/frame 0xfffffe0256158f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0256158f30 (kgdb) frame 7 #7 buf_hash_insert (hdr=0xfffff80c906b03e8, lockp=lockp@entry=0x0) at /usr/src/FreeBSD/sys/contrib/openzfs/module/zfs/arc.c:1062 1062 if (HDR_EQUAL(hdr->b_spa, &hdr->b_dva, hdr->b_birth, fhdr)) (kgdb) p *hdr $1 = {b_dva = {dva_word = {16, 19965896928}}, b_birth = 36629088, b_type = ARC_BUFC_METADATA, b_complevel = 0 '\000', b_reserved1 = 0 '\000', b_reserved2 = 0, b_hash_next = 0x0, b_flags = (ARC_FLAG_BUFC_METADATA | ARC_FLAG_HAS_L1HDR | ARC_FLAG_HAS_L2HDR | ARC_FLAG_COMPRESSED_ARC | ARC_FLAG_COMPRESS_1), b_psize = 5, b_lsize = 5, b_spa = 3583499065027950438, b_l2hdr = {b_dev = 0xfffffe02306c8000, b_daddr = 4917395456, b_hits = 0, b_arcs_state = ARC_STATE_MRU, b_l2node = {list_next = 0xfffff801313fc9b0, list_prev = 0xfffff801313fca70}}, b_l1hdr = {b_cv = { cv_description = 0xffffffff80bb5b02 "hdr->b_l1hdr.b_cv", cv_waiters = 0}, b_byteswap = 10 '\n', b_state = 0xffffffff80f02900 , b_arc_node = {list_next = 0x0, list_prev = 0x0}, b_arc_access = 0, b_mru_hits = 0, b_mru_ghost_hits = 0, b_mfu_hits = 0, b_mfu_ghost_hits = 0, b_bufcnt = 0, b_buf = 0x0, b_refcnt = {rc_count = 0}, b_acb = 0x0, b_pabd = 0x0}, b_crypt_hdr = {b_rabd = 0x10, b_ot = 2786027712, b_ebufcnt = 4, b_dsobj = 36629088, b_salt = "\001\000\000\000\000\000\000", b_iv = "\240\2769$\001\370\377\377\220\000\036\017", b_mac = "\b\000 \000fw\357\327i%\2731\000\200l0"}} #3 (not ZFS, but VFS, could be related?) --- trap 0x9, rip = 0xffffffff80801408, rsp = 0xfffffe02348cbcc0, rbp = 0xfffffe02348cbcf0 --- pwd_chdir() at pwd_chdir+0x28/frame 0xfffffe02348cbcf0 kern_chdir() at kern_chdir+0x169/frame 0xfffffe02348cbe00 amd64_syscall() at amd64_syscall+0x100/frame 0xfffffe02348cbf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe02348cbf30 (kgdb) frame 7 #7 pwd_alloc () at /usr/src/FreeBSD/sys/kern/kern_descrip.c:3920 warning: Source file is more recent than executable. 3920 bzero(pwd, sizeof(*pwd)); (kgdb) p pwd $1 = (struct pwd *) 0xfff2fff0fff1ffed (kgdb) p *pwd Cannot access memory at address 0xfff2fff0fff1ffed I'm switching to INVARIANTS kernel right now and will see if that panics earlier. -- Gleb Smirnoff From nobody Fri Jun 9 04:57:12 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qcpj72bB3z4bcSn for ; Fri, 9 Jun 2023 04:57:15 +0000 (UTC) (envelope-from glebius@freebsd.org) Received: from glebi.us (glebi.us [162.251.186.162]) by mx1.freebsd.org (Postfix) with ESMTP id 4Qcpj63nprz4QJb; Fri, 9 Jun 2023 04:57:14 +0000 (UTC) (envelope-from glebius@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 162.251.186.162 is neither permitted nor denied by domain of glebius@freebsd.org) smtp.mailfrom=glebius@freebsd.org; dmarc=none Received: by glebi.us (Postfix, from userid 1000) id 0516722D88; Thu, 8 Jun 2023 21:57:12 -0700 (PDT) Date: Thu, 8 Jun 2023 21:57:12 -0700 From: Gleb Smirnoff To: current@freebsd.org Cc: mm@freebsd.org Subject: Re: panic(s) in ZFS on CURRENT Message-ID: References: List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [0.50 / 15.00]; VIOLATED_DIRECT_SPF(3.50)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; MLMMJ_DEST(0.00)[current@freebsd.org]; ASN(0.00)[asn:27348, ipnet:162.251.186.0/24, country:US]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[glebius]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[freebsd.org]; TO_DN_NONE(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Qcpj63nprz4QJb X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N On Thu, Jun 08, 2023 at 07:56:07PM -0700, Gleb Smirnoff wrote: T> I'm switching to INVARIANTS kernel right now and will see if that panics earlier. This is what I got with INVARIANTS: panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (225142071296 <= 225142063104) cpuid = 17 time = 1686286015 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 0xfffffe0160dcea90 kdb_backtrace() at kdb_backtrace+0x46/frame 0xfffffe0160dceb40 vpanic() at vpanic+0x21f/frame 0xfffffe0160dcebe0 spl_panic() at spl_panic+0x4d/frame 0xfffffe0160dcec60 l2arc_write_buffers() at l2arc_write_buffers+0xcda/frame 0xfffffe0160dcedf0 l2arc_feed_thread() at l2arc_feed_thread+0x547/frame 0xfffffe0160dceec0 fork_exit() at fork_exit+0x122/frame 0xfffffe0160dcef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0160dcef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Uptime: 1m4s Dumping 5473 out of 65308 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% (kgdb) frame 4 #4 0xffffffff804342ea in l2arc_write_buffers (spa=0xfffffe022e942000, dev=0xfffffe023116a000, target_sz=16777216) at /usr/src/FreeBSD/sys/contrib/openzfs/module/zfs/arc.c:9445 9445 ASSERT3U(dev->l2ad_hand, <=, dev->l2ad_evict); (kgdb) p dev $1 = (l2arc_dev_t *) 0xfffffe023116a000 (kgdb) p dev->l2ad_hand $2 = 225142071296 (kgdb) p dev->l2ad_evict $3 = 225142063104 (kgdb) p *dev value of type `l2arc_dev_t' requires 66136 bytes, which is more than max-value-size Never seen kgdb not being able to print a structure that reported to be too big. -- Gleb Smirnoff From nobody Fri Jun 9 06:28:31 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qcrkp1lcCz4bvFF for ; Fri, 9 Jun 2023 06:28:50 +0000 (UTC) (envelope-from otis@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qcrkp0JHtz3MYT for ; Fri, 9 Jun 2023 06:28:50 +0000 (UTC) (envelope-from otis@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1686292130; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Fh07cMMNiO5KY7YB7FTFFZF6tyN5nKlpPeJX3EMLcYA=; b=s6etibK5yJu+W9gqPO1gqUF3VY8irYThJ/6DqiiY6iCR6Qlq4WuX4z6fUSQoLUbSXZYMrE Npphz7tuBoXMackD7t8xUqs47WjP3uwLoewhBaXjMOi3iD6HP0UDGHVw4CdMN+ciYIcXDJ rquQI9s9cIHwwj2cag7IbOW/Dbd/oOs5yEIoCHyacsNOECO3Wnqek1MVo401asH+crMQfL yX5gGAqE96Y59c7rkhTrMMq6fT/zqC6eytho/cj7x+tYtOATz/0DGf4OKvHegH5oTkV/Ka BwoK/nurAz6ae6Y//+A3mAVuSdI09NuvSMUST/4W1A4bkUIE5gO2et95S0QJig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1686292130; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Fh07cMMNiO5KY7YB7FTFFZF6tyN5nKlpPeJX3EMLcYA=; b=KMEk8meY4FPZsfvlqa2gxoN0OBowrhrGvZYaB+dD57MziCAJ9H6jiARDf/DlJcxEN+oU9i CqAkU7KoHEs//SWURs2uDtfRG2gbBP17Xo+Q/DIUfiIA1MFSAXHh+54xHVY3+6Ekgj3QlH 9S1KGM7Qz0h3/BcnR0oHkuJ8m5+pRrjLHTxd/58rV4vjArVTQRZtPzRL0y6LHg0XdqS9mF KSSwQRYpaR3+cdDxdOx4deVzxyh89YmyZ4imZ26Z/KxL8bkybo58w6wCpGci1bd3hx88M8 2rGwnNlES3r+8X3hmqXpeT4yuN9Z8cAwNn8Zldu3sK6Y39xiR/YSSlUGTPrcoQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1686292130; a=rsa-sha256; cv=none; b=rZvaBhqcY4sQAdXB1dlCqDMNXbPu5JlcBf3uksDmefoYQTU4LG0oWs9cZ5NPDY0nwo4A52 /XluR2UUEQF0AqnwcJ8kxjqKO3GTJW4TDQWyRNbxazrSd0mAYitYjuEbQlrFcbAvwe0d8x Cvunlq1DDxLJLhseSTOgiHsZhcdbIsjq3FRJY7MasYnMFB2Pd4axIbfIlckfYQ5In2WaOd tGcAoXlbWZ0/MDV0vy+qZils260EUXiAnn+sokKVA4vstPbizELMPU5d/+WEJL+raFoO/x I0pxfd/4ncN3kcThB6NVst9Rilgpe6MMn9skv1hnX/9kNTZth3MWKMvShRTjRQ== Received: from ns2.wilbury.net (ns2.wilbury.net [92.60.51.55]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "svc.wilbury.net", Issuer "R3" (verified OK)) (Authenticated sender: otis) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Qcrkn5gzhz14vT for ; Fri, 9 Jun 2023 06:28:49 +0000 (UTC) (envelope-from otis@FreeBSD.org) Received: from smtpclient.apple (gw-upc.owhome.net [188.167.168.254]) (Authenticated sender: juraj@lutter.sk) by svc.wilbury.net (Postfix) with ESMTPSA id 4C15B61F8D for ; Fri, 9 Jun 2023 08:28:46 +0200 (CEST) From: Juraj Lutter Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.7\)) Subject: CPU hog on -current (pfSense 23.05) Message-Id: <50D90C0F-4C92-4A6A-B1FD-D56609434AF4@FreeBSD.org> Date: Fri, 9 Jun 2023 08:28:31 +0200 To: FreeBSD CURRENT X-Mailer: Apple Mail (2.3731.600.7) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on ns2.wilbury.net X-ThisMailContainsUnwantedMimeParts: N Hi, since cca 7 days I=E2=80=99ve started to observe a CPU hog of one CPU = core on APU2 box running pfSense 23.05 (if that matters). mjg@ suggested to run a dtrace oneliner that showed: ~~~~~~ kernel`pmap_copy 33 kernel`amd64_syscall 33 kernel`vm_radix_insert 35 kernel`vm_map_pmap_enter 37 kernel`vm_radix_lookup_unlocked 38 kernel`memcpy_std 38 kernel`vm_object_deallocate 39 kernel`pmap_enter_quick_locked 41 kernel`em_update_stats_counters 43 kernel`copyout_nosmap_std 43 kernel`ck_epoch_poll_deferred 44 kernel`sbuf_put_bytes 46 kernel`vm_page_pqbatch_submit 48 kernel`pmap_remove_pte 51 kernel`pmap_pvh_remove 53 kernel`vm_pqbatch_process_page 54 kernel`cpu_search_highest 56 kernel`get_pv_entry 57 kernel`pmap_try_insert_pv_entry 59 kernel`vm_map_lookup_entry 65 kernel`epoch_call_task 92 kernel`pmap_enter 101 kernel`vm_fault 110 kernel`pagecopy 110 kernel`0xffffffff81 133 kernel`lock_delay 145 kernel`pmap_remove_pages 203 kernel`_thread_lock 415 kernel`pagezero_std 490 kernel`assert_rw 532 kernel`acpi_cpu_c1 600 kernel`callout_lock 641 kernel`kern_yield 1206 kernel`_callout_stop_safe 2010 kernel`spinlock_enter 2032 kernel`tcp_timer_stop 2703 0x0 5927 kernel`spinlock_exit 40964 kernel`cpu_idle 61943 kernel`sched_idletd 76722 ~~~~~~~ The symptom is that the kernel thread "kernel{if_io_tqg_1}=E2=80=9D = consumes 100% of the CPU core. Can this be debugged any further? I also suspect the hardware problem (I have one spare box where I=E2=80=99= ll put my XML config and test whether the problem will persist). I=E2=80=99d be very thankful for any pointers. Thanks! otis =E2=80=94 Juraj Lutter otis@FreeBSD.org From nobody Fri Jun 9 08:17:11 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qcv84163pz4cFXK; Fri, 9 Jun 2023 08:17:24 +0000 (UTC) (envelope-from developer@lorenzosalvadore.it) Received: from mail-4317.proton.ch (mail-4317.proton.ch [185.70.43.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qcv83593jz3qT6; Fri, 9 Jun 2023 08:17:23 +0000 (UTC) (envelope-from developer@lorenzosalvadore.it) Authentication-Results: mx1.freebsd.org; none Date: Fri, 09 Jun 2023 08:17:11 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lorenzosalvadore.it; s=protonmail2; t=1686298639; x=1686557839; bh=kigFkR4F4mxp0zj2qQXrupR/2s9ehA4IM7GD0npjlpA=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=snwMDozSHF7KVdnXiC49kVYeEcHz6w3GMrQ21QtFN/lbx64Z+cdtwEsv8dXyKil3n mVksGObqg7jaOO0re5oue/XRjOwtFKVtBFMMdwjXHnQdwef4PLPUpyu49y7ip0ToUb hSiE/WOzVQgl5eD4kupEmaYCcXkGC1mZwL/Ir+GJBm0sQyNcyRvleusB19Es5lTk+x kl8sHhGoUK15d7IbNV2zK5L+5hI2lA29VrMo+EL8mpRySBWYCQCB7cHLd/h1QsVI0x 6mbib9nMekbIgtDOXf5in57VZIfvJEJic2Z5t9qOU9xqcEv6zV+hJXY5i5xgaqC/eO xxCqFG//5L6Jg== To: emaste@freebsd.org, freebsd-current@freebsd.org, freebsd-ports@freebsd.org From: Lorenzo Salvadore Subject: Re: OpenSSL 3.0 in the base system update Message-ID: <9KoML-8eYmTtNpR9rZqCLOd7Vd05TEMWyI2J-Sau2BLFu4fLw1L-OyhaTcs2Fhv3VB8_0vHz6G0Nrm5EZPpymM3vfPWRZDog-G_EnDc6C8I=@lorenzosalvadore.it> In-Reply-To: References: Feedback-ID: 53711648:user:proton List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="b1_odkIG8Ww6iCgXyN6aMB9JxdIj4HbBzKt7FoquEaIE" X-Rspamd-Queue-Id: 4Qcv83593jz3qT6 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:62371, ipnet:185.70.43.0/24, country:CH] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --b1_odkIG8Ww6iCgXyN6aMB9JxdIj4HbBzKt7FoquEaIE Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 VGhhbmtzIGZvciB0aGUgd29yayBhbmQgdGhlIHVwZGF0ZS4KCk1heSBJIHN1Z2dlc3QgdG8gc3Vi bWl0IHRoZSB1cGRhdGUgdG8gMjAyM1EyIHN0YXR1cyByZXBvcnRzIHRvbz8KaHR0cHM6Ly9kb2Nz LmZyZWVic2Qub3JnL2VuL2FydGljbGVzL2ZyZWVic2Qtc3RhdHVzLXJlcG9ydC1wcm9jZXNzLyNf c3VibWl0X3lvdXJfcmVwb3J0CgpDaGVlcnMsCgpMb3JlbnpvIFNhbHZhZG9yZQoKSW52aWF0byBk YSBQcm90b24gTWFpbCBtb2JpbGUKCi0tLS0tLS0tIE1lc3NhZ2dpbyBvcmlnaW5hbGUgLS0tLS0t LS0KSWwgOCBHaXUgMjAyMywgMTk6MTMsIEVkIE1hc3RlIGhhIHNjcml0dG86Cgo+IEFzIHByZXZp b3VzbHkgbWVudGlvbmVkWzFdIEZyZWVCU0QgMTQuMCB3aWxsIGluY2x1ZGUgT3BlblNTTCAzLjAu IFdlIGV4cGVjdCB0byBtZXJnZSB0aGUgdXBkYXRlIHRvIG1haW4gaW4gdGhlIG5lYXIgZnV0dXJl ICh3aXRoaW4gdGhlIG5leHQgd2VlayBvciB0d28pIGFuZCBhcmUgcmVhZHkgZm9yIHdpZGVyIHRl c3RpbmcuIFN1cHBvcnRlZCBieSB0aGUgRnJlZUJTRCBGb3VuZGF0aW9uLCBQaWVycmUgUHJvbmNo ZXJ5IGhhcyBiZWVuIHdvcmtpbmcgb24gdGhlIHVwZGF0ZSBpbiB0aGUgc3JjIHRyZWUsIHdpdGgg YXNzaXN0YW5jZSBmcm9tIEVuamkgQ29vcGVyIChuZ2llQCksIGFuZCBtZSAoZW1hc3RlQCkuIFRo YW5rcyB0byBBbnRvaW5lIEJyb2RpbiAoYW50b2luZUApIGFuZCBNdWhhbW1hZCBNb2ludXIgUmFo bWFuIChib2ZoQCkgZm9yIHBvcnRzIGV4cC1ydW5zIGFuZCBmaXhlcy93b3JrYXJvdW5kcyBhbmQg dG8gRGFnLUVybGluZyAoZGVzQCkgZm9yIHVwZGF0aW5nIGxkbnMgaW4gdGhlIGJhc2Ugc3lzdGVt LiAjIyBCYXNlIHN5c3RlbSBjb21wYXRpYmlsaXR5IHN0YXR1cyBNb3N0IG9mIHRoZSBiYXNlIHN5 c3RlbSBpcyByZWFkeSBmb3IgYSBzZWFtbGVzcyBzd2l0Y2ggdG8gT3BlblNTTCAzLjAuIEZvciBz ZXZlcmFsIGNvbXBvbmVudHMgd2UndmUgYWRkZWQgYC1ET1BFTlNTTF9BUElfQ09NUEFUPTB4MTAx MDAwMDBMYCB0byBDRkxBR1MgdG8gc3BlY2lmeSB0aGUgQVBJIHZlcnNpb24sIHdoaWNoIGF2b2lk cyBkZXByZWNhdGlvbiB3YXJuaW5ncyBmcm9tIE9wZW5TU0wgMy4wLiBDaGFuZ2VzIGhhdmUgYWxz byBiZWVuIG1hZGUgdG8gYXZvaWQgT3BlblNTTCBBUElzIGFscmVhZHkgZGVwcmVjYXRlZCBpbiBP cGVuU1NMIDEuMS4gV2UgY2FuIGNvbnRpbnVlIHRoZSBwcm9jZXNzIG9mIHVwZGF0aW5nIHRvIGNv bnRlbXBvcmFyeSBBUElzIGFmdGVyIE9wZW5TU0wgMy4wIGlzIGluIHRoZSB0cmVlLiBBZGRpdGlv bmFsIGNoYW5nZXMgYXJlIHN0aWxsIHJlcXVpcmVkIGZvciBsaWJhcmNoaXZlIGFuZCBzZXZlbiBL ZXJiZXJvcy1yZWxhdGVkIGxpYnJhcmllcyBvciB0b29scy4gV29ya2Fyb3VuZHMgYXJlIHJlYWR5 IHRvIGdvIGFsb25nIHdpdGggdGhlIE9wZW5TU0wgMyBpbXBvcnQsIGFuZCBwcm9wZXIgZml4ZXMg YXJlIGluIHByb2dyZXNzIGluIHRoZSB1cHN0cmVhbSBwcm9qZWN0cy4gQSBzZWdmYXVsdCBmcm9t IGBvcGVuc3NsIHg1MDlgIGluIHRoZSBpMzg2IHBvcnRzIGV4cC1ydW4gaXMgdW5kZXIgaW52ZXN0 aWdhdGlvbiBhbmQgbmVlZHMgdG8gYmUgYWRkcmVzc2VkIHByaW9yIHRvIHRoZSBtZXJnZS4gIyMg UG9ydHMgY29tcGF0aWJpbGl0eSBXaXRoIGJvZmhAJ3MgcmVjZW50IHd3dy9ub2RlMTggYW5kIHd3 dy9ub2RlMjAgcGF0Y2hlcyB0aGUgcG9ydHMgdHJlZSBpcyBpbiByZWFzb25hYmxlIHNoYXBlIGZv ciBPcGVuU1NMIDMuMCBpbiB0aGUgYmFzZSBzeXN0ZW0uIFRoZSBleHAtcnVuIChsaW5rIGJlbG93 KSBoYXMgYSBsaXN0IG9mIHRoZSBmYWlsaW5nIHBvcnRzLCBhbmQgSSd2ZSBlbWFpbGVkIGFsbCBv ZiB0aGUgbWFpbnRhaW5lcnMgYXMgYSBoZWFkcy11cC4gTm9uZSBvZiB0aGUgcmVtYWluaW5nIGZh aWx1cmVzIGFyZSByZXNwb25zaWJsZSBmb3IgYSBsYXJnZSBudW1iZXIgb2Ygc2tpcHBlZCBwb3J0 cyAoaS5lLiwgdGhlIGZhaWx1cmVzIGFyZSBlaXRoZXIgbGVhZiBwb3J0cyBvciBhcmUgcmVzcG9u c2libGUgZm9yIG9ubHkgYSBzbWFsbCBudW1iZXIgb2Ygc2tpcHBlZCBwb3J0cykuIEkgZXhwZWN0 IHRoYXQgc29tZSBvciBtYW55IG9mIHRoZXNlIHdpbGwgbmVlZCB0byBiZSBhZGRyZXNzZWQgYWZ0 ZXIgdGhlIGNoYW5nZSBsYW5kcyBpbiB0aGUgc3JjIHRyZWUuICMjIENhbGwgZm9yIHRlc3Rpbmcg V2Ugd2VsY29tZSBmZWVkYmFjayBmcm9tIGFueW9uZSB3aWxsaW5nIHRvIHRlc3QgdGhlIHdvcmsg aW4gcHJvZ3Jlc3MuIFBpZXJyZSdzIHVwZGF0ZSBjYW4gYmUgb2J0YWluZWQgZnJvbSB0aGUgcHVs bCByZXF1ZXN0WzJdIG9yIGJ5IGZldGNoaW5nIHRoZSBicmFuY2hbM10uIElmIGRlc2lyZWQgSSB3 aWxsIHByb3ZpZGUgYSBsYXJnZSBkaWZmIGFnYWluc3QgbWFpbi4gIyMgTGlua3MgLSBCYXNlIHN5 c3RlbSBPcGVuU1NMIDMuMCB1cGRhdGUgdHJhY2tpbmcgUFI6IGh0dHBzOi8vYnVncy5mcmVlYnNk Lm9yZy8yNzE2MTUgLSBQb3J0cyBleHAtcnVuIHdpdGggT3BlblNTTCAzLjAgaW4gdGhlIGJhc2Ug c3lzdGVtOiBodHRwczovL2J1Z3MuZnJlZWJzZC5vcmcvMjcxNjU2IFsxXSBodHRwczovL2xpc3Rz LmZyZWVic2Qub3JnL2FyY2hpdmVzL2ZyZWVic2QtY3VycmVudC8yMDIzLU1heS8wMDM2MDkuaHRt bCBbMl0gaHR0cHM6Ly9naXRodWIuY29tL2ZyZWVic2QvZnJlZWJzZC1zcmMvcHVsbC83NjAgWzNd IGh0dHBzOi8vZ2l0aHViLmNvbS9raG9yYmVuL2ZyZWVic2Qtc3JjL3RyZWUva2hvcmJlbi9vcGVu c3NsLTMuMC45 --b1_odkIG8Ww6iCgXyN6aMB9JxdIj4HbBzKt7FoquEaIE Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 VGhhbmtzIGZvciB0aGUgd29yayBhbmQgdGhlIHVwZGF0ZS48YnI+PGJyPk1heSBJIHN1Z2dlc3Qg dG8gc3VibWl0IHRoZSB1cGRhdGUgdG8gMjAyM1EyIHN0YXR1cyByZXBvcnRzIHRvbz88YnI+PGEg aHJlZj0iaHR0cHM6Ly9kb2NzLmZyZWVic2Qub3JnL2VuL2FydGljbGVzL2ZyZWVic2Qtc3RhdHVz LXJlcG9ydC1wcm9jZXNzLyNfc3VibWl0X3lvdXJfcmVwb3J0Ij5odHRwczovL2RvY3MuZnJlZWJz ZC5vcmcvZW4vYXJ0aWNsZXMvZnJlZWJzZC1zdGF0dXMtcmVwb3J0LXByb2Nlc3MvI19zdWJtaXRf eW91cl9yZXBvcnQ8L2E+PGJyPjxicj5DaGVlcnMsPGJyPjxicj5Mb3JlbnpvIFNhbHZhZG9yZTxi cj48YnI+SW52aWF0byBkYSBQcm90b24gTWFpbCBtb2JpbGU8YnI+PGJyPjxicj48YnI+LS0tLS0t LS0gTWVzc2FnZ2lvIG9yaWdpbmFsZSAtLS0tLS0tLTxicj5JbCA4IEdpdSAyMDIzLCAxOToxMywg RWQgTWFzdGUgPCBlbWFzdGVAZnJlZWJzZC5vcmc+IGhhIHNjcml0dG86PGJsb2NrcXVvdGUgY2xh c3M9InByb3Rvbm1haWxfcXVvdGUiPjxicj5BcyBwcmV2aW91c2x5IG1lbnRpb25lZFsxXSBGcmVl QlNEIDE0LjAgd2lsbCBpbmNsdWRlIE9wZW5TU0wgMy4wLiAgV2UNCmV4cGVjdCB0byBtZXJnZSB0 aGUgdXBkYXRlIHRvIG1haW4gaW4gdGhlIG5lYXIgZnV0dXJlICh3aXRoaW4gdGhlIG5leHQNCndl ZWsgb3IgdHdvKSBhbmQgYXJlIHJlYWR5IGZvciB3aWRlciB0ZXN0aW5nLg0KDQpTdXBwb3J0ZWQg YnkgdGhlIEZyZWVCU0QgRm91bmRhdGlvbiwgUGllcnJlIFByb25jaGVyeSBoYXMgYmVlbiB3b3Jr aW5nDQpvbiB0aGUgdXBkYXRlIGluIHRoZSBzcmMgdHJlZSwgd2l0aCBhc3Npc3RhbmNlIGZyb20g RW5qaSBDb29wZXINCihuZ2llQCksIGFuZCBtZSAoZW1hc3RlQCkuIFRoYW5rcyB0byBBbnRvaW5l IEJyb2RpbiAoYW50b2luZUApIGFuZA0KTXVoYW1tYWQgTW9pbnVyIFJhaG1hbiAoYm9maEApIGZv ciBwb3J0cyBleHAtcnVucyBhbmQNCmZpeGVzL3dvcmthcm91bmRzIGFuZCB0byBEYWctRXJsaW5n IChkZXNAKSBmb3IgdXBkYXRpbmcgbGRucyBpbiB0aGUNCmJhc2Ugc3lzdGVtLg0KDQojIyBCYXNl IHN5c3RlbSBjb21wYXRpYmlsaXR5IHN0YXR1cw0KDQpNb3N0IG9mIHRoZSBiYXNlIHN5c3RlbSBp cyByZWFkeSBmb3IgYSBzZWFtbGVzcyBzd2l0Y2ggdG8gT3BlblNTTCAzLjAuDQpGb3Igc2V2ZXJh bCBjb21wb25lbnRzIHdlJ3ZlIGFkZGVkIGAtRE9QRU5TU0xfQVBJX0NPTVBBVD0weDEwMTAwMDAw TGANCnRvIENGTEFHUyB0byBzcGVjaWZ5IHRoZSBBUEkgdmVyc2lvbiwgd2hpY2ggYXZvaWRzIGRl cHJlY2F0aW9uDQp3YXJuaW5ncyBmcm9tIE9wZW5TU0wgMy4wLiBDaGFuZ2VzIGhhdmUgYWxzbyBi ZWVuIG1hZGUgdG8gYXZvaWQNCk9wZW5TU0wgQVBJcyBhbHJlYWR5IGRlcHJlY2F0ZWQgaW4gT3Bl blNTTCAxLjEuIFdlIGNhbiBjb250aW51ZSB0aGUNCnByb2Nlc3Mgb2YgdXBkYXRpbmcgdG8gY29u dGVtcG9yYXJ5IEFQSXMgYWZ0ZXIgT3BlblNTTCAzLjAgaXMgaW4gdGhlDQp0cmVlLg0KDQpBZGRp dGlvbmFsIGNoYW5nZXMgYXJlIHN0aWxsIHJlcXVpcmVkIGZvciBsaWJhcmNoaXZlIGFuZCBzZXZl bg0KS2VyYmVyb3MtcmVsYXRlZCBsaWJyYXJpZXMgb3IgdG9vbHMuIFdvcmthcm91bmRzIGFyZSBy ZWFkeSB0byBnbyBhbG9uZw0Kd2l0aCB0aGUgT3BlblNTTCAzIGltcG9ydCwgYW5kIHByb3BlciBm aXhlcyBhcmUgaW4gcHJvZ3Jlc3MgaW4gdGhlDQp1cHN0cmVhbSBwcm9qZWN0cy4NCg0KQSBzZWdm YXVsdCBmcm9tIGBvcGVuc3NsIHg1MDlgIGluIHRoZSBpMzg2IHBvcnRzIGV4cC1ydW4gaXMgdW5k ZXINCmludmVzdGlnYXRpb24gYW5kIG5lZWRzIHRvIGJlIGFkZHJlc3NlZCBwcmlvciB0byB0aGUg bWVyZ2UuDQoNCiMjIFBvcnRzIGNvbXBhdGliaWxpdHkNCg0KV2l0aCBib2ZoQCdzIHJlY2VudCB3 d3cvbm9kZTE4IGFuZCB3d3cvbm9kZTIwIHBhdGNoZXMgdGhlIHBvcnRzIHRyZWUNCmlzIGluIHJl YXNvbmFibGUgc2hhcGUgZm9yIE9wZW5TU0wgMy4wIGluIHRoZSBiYXNlIHN5c3RlbS4gVGhlIGV4 cC1ydW4NCihsaW5rIGJlbG93KSBoYXMgYSBsaXN0IG9mIHRoZSBmYWlsaW5nIHBvcnRzLCBhbmQg SSd2ZSBlbWFpbGVkIGFsbCBvZg0KdGhlIG1haW50YWluZXJzIGFzIGEgaGVhZHMtdXAuIE5vbmUg b2YgdGhlIHJlbWFpbmluZyBmYWlsdXJlcyBhcmUNCnJlc3BvbnNpYmxlIGZvciBhIGxhcmdlIG51 bWJlciBvZiBza2lwcGVkIHBvcnRzIChpLmUuLCB0aGUgZmFpbHVyZXMNCmFyZSBlaXRoZXIgbGVh ZiBwb3J0cyBvciBhcmUgcmVzcG9uc2libGUgZm9yIG9ubHkgYSBzbWFsbCBudW1iZXIgb2YNCnNr aXBwZWQgcG9ydHMpLiBJIGV4cGVjdCB0aGF0IHNvbWUgb3IgbWFueSBvZiB0aGVzZSB3aWxsIG5l ZWQgdG8gYmUNCmFkZHJlc3NlZCBhZnRlciB0aGUgY2hhbmdlIGxhbmRzIGluIHRoZSBzcmMgdHJl ZS4NCg0KIyMgQ2FsbCBmb3IgdGVzdGluZw0KDQpXZSB3ZWxjb21lIGZlZWRiYWNrIGZyb20gYW55 b25lIHdpbGxpbmcgdG8gdGVzdCB0aGUgd29yayBpbiBwcm9ncmVzcy4NClBpZXJyZSdzIHVwZGF0 ZSBjYW4gYmUgb2J0YWluZWQgZnJvbSB0aGUgcHVsbCByZXF1ZXN0WzJdIG9yIGJ5DQpmZXRjaGlu ZyB0aGUgYnJhbmNoWzNdLiBJZiBkZXNpcmVkIEkgd2lsbCBwcm92aWRlIGEgbGFyZ2UgZGlmZiBh Z2FpbnN0DQptYWluLg0KDQojIyBMaW5rcw0KDQotIEJhc2Ugc3lzdGVtIE9wZW5TU0wgMy4wIHVw ZGF0ZSB0cmFja2luZyBQUjoNCiAgaHR0cHM6Ly9idWdzLmZyZWVic2Qub3JnLzI3MTYxNQ0KDQot IFBvcnRzIGV4cC1ydW4gd2l0aCBPcGVuU1NMIDMuMCBpbiB0aGUgYmFzZSBzeXN0ZW06DQogIGh0 dHBzOi8vYnVncy5mcmVlYnNkLm9yZy8yNzE2NTYNCg0KWzFdIGh0dHBzOi8vbGlzdHMuZnJlZWJz ZC5vcmcvYXJjaGl2ZXMvZnJlZWJzZC1jdXJyZW50LzIwMjMtTWF5LzAwMzYwOS5odG1sDQpbMl0g aHR0cHM6Ly9naXRodWIuY29tL2ZyZWVic2QvZnJlZWJzZC1zcmMvcHVsbC83NjANClszXSBodHRw czovL2dpdGh1Yi5jb20va2hvcmJlbi9mcmVlYnNkLXNyYy90cmVlL2tob3JiZW4vb3BlbnNzbC0z LjAuOQ0KDQo8L2Rpdj4= --b1_odkIG8Ww6iCgXyN6aMB9JxdIj4HbBzKt7FoquEaIE-- From nobody Fri Jun 9 11:33:59 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QczW01r6sz4bfxX; Fri, 9 Jun 2023 11:34:04 +0000 (UTC) (envelope-from bofh@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QczW01Lyvz4GQ0; Fri, 9 Jun 2023 11:34:04 +0000 (UTC) (envelope-from bofh@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1686310444; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=tKsY5zIfHTGKaYrX9V8vpQDlt4woGkLYc9KfCorANUs=; b=FF9eoZzXc67Pd+zZPOf3oKbvcodJZ7c4XwhMe8L2YhMVpTXAYX3lYaR6kiUfpUo0dLB3gD ieTM6rqeQqFp1MNK8oBTBtGvm+RrofPImf9RtevaVfbsPP9DmNzrQm1K87LWSNEm2Rf5d3 hFy8ROuRqDD9iNyc5AKsHCRKGPGmX2/TwVZ9K93tvf1tzdb+Hpj4L/CM+khodXR/zjE92N 9aQ/ACBUHs3uAYM4S8v+zAsc0cpVYU+8zRrD+kY9QcFb576daa9NNW3ZB8D4teZ4MpToxj PSR9fnIkCnl8dD2e8zQoAklIgXxZogJVVBxA8bDJtN2Dr4aLku9vZiACcfPFxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1686310444; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=tKsY5zIfHTGKaYrX9V8vpQDlt4woGkLYc9KfCorANUs=; b=Ls+M4kFlY7saeLTSS+OQ+fX4E+6v4rofp1MZ1kFW4T2Let3O1eGDhg4LjU9CGI+Mf6cewI sc1oZNDhciqZXOvRNKLTI+4R+dYSS9U1MiCSFwMzJxjmSUJCgtF6GeT3JUK6qq4VgVqPUc UDHMb/enDLBoJ1ZYtr+22laJtnrYigeAClMze1j1TKNDuRW8NG/Ai75XkuA/YVWWBNj6xo 0HNIUvyJweUSxSAFRdw4tvcjAJVok6AmolIuOEQZ0elMIwVVEEO82nPgx6gw9nMkOl5zkI n3BtpYwSLDYOWQxIsYWJ7owsCK/xwAXp4WZlkkss1pBr3SM+JJZXD0s+Yus5eQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1686310444; a=rsa-sha256; cv=none; b=wwGRYVTW8pvMqBt3eTTPReaf1L92E8Y3Qu1aZcWc86X5XhLuyAsdwRIyjpqIElsMzcr9hR OO7MIYoypHxcjYz9M/xImpsfWATB/6ipiEMsDhJgfKrRSSplw8tmdpENgp1cwtDUeqKCii 1zZJoXhIdVbrkuJANVnBhGgsRx6lgaTcllfv7LkBD/qgFSsIGyrHHt+i01yhOPpRHbekxG JDlEAmiUjL5RwDZFR0JQ3RmD+U+X5bCtJzzEbA1WBdY4irkB0rKt6trA0dZcFvT2rzrLqL ABw8vapAQ4kYpiqwud9yRPB0eesrXYP+nKB+WZRIAeB2hQ2wD90s0LHoJw04yw== Received: from mx.bofh.network (mx.bofh.network [IPv6:2a01:4f8:261:25de::227]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mx.bofh.network", Issuer "R3" (verified OK)) (Authenticated sender: bofh/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4QczVz4FHpz19JC; Fri, 9 Jun 2023 11:34:03 +0000 (UTC) (envelope-from bofh@freebsd.org) Received: from smtpclient.apple ( [84.252.115.4]) by mx.bofh.network (OpenSMTPD) with ESMTPSA id b64c55dd (TLSv1.2:ECDHE-ECDSA-AES256-GCM-SHA384:256:NO); Fri, 9 Jun 2023 11:34:01 +0000 (UTC) Content-Type: multipart/signed; boundary="Apple-Mail=_BAB6FD6D-9471-4858-96BD-99C92F1ACD95"; protocol="application/pgp-signature"; micalg=pgp-sha512 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.3\)) Subject: Re: OpenSSL 3.0 in the base system update From: Moin Rahman In-Reply-To: <257c6981-f49a-6d31-b5f8-7cad1d3e889a@grosbein.net> Date: Fri, 9 Jun 2023 13:33:59 +0200 Cc: FreeBSD Current , FreeBSD Ports Message-Id: References: <257c6981-f49a-6d31-b5f8-7cad1d3e889a@grosbein.net> To: Eugene Grosbein X-Mailer: Apple Mail (2.3696.120.41.1.3) X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_BAB6FD6D-9471-4858-96BD-99C92F1ACD95 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Jun 9, 2023, at 5:22 AM, Eugene Grosbein = wrote: >=20 > 09.06.2023 0:13, Ed Maste wrote: >=20 > [skip] >=20 >> We welcome feedback from anyone willing to test the work in progress. >=20 > Is it possible to grab some binary snapshot of 14-CURRENT to install = into a VM and test my port that fails build? > If not, how do I build 14-CURRENT with OpenSSL 3.0 myself? >=20 > Eugene >=20 >=20 At the moment there is no binary snapshot as the codebase has not yet been merged into the Source of Truth. The best way to test your port is either with poudriere or with a local jail. In case you are using poudriere the following might be helpful for you: git clone git@github.com:khorben/freebsd-src.git openssl3-src cd openssl3-src && git checkout khorben/openssl-3.0.9 poudriere jail -c -j MAINopenssl3 -b -m src=3D/openssl3-src -J = 64 If you are the Eugen from gost-engine then I believe it should be a = simple upgrade to the latest 3.0.1 version. And if you do not have enough firepower for running HEAD jail submit a patch and assign it to me. I will look into it. Kind regards, Moin --Apple-Mail=_BAB6FD6D-9471-4858-96BD-99C92F1ACD95 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEETfdREoUGjQZKBS+fvbm1phfAvJEFAmSDDidfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDRE Rjc1MTEyODUwNjhEMDY0QTA1MkY5RkJEQjlCNUE2MTdDMEJDOTEACgkQvbm1phfA vJGmag/+N1jkZkkGftyPby4m6CJOhfvlBNlReBPvfonaFJBXgzR27KN9oxFroZnO A7UNx3lscZOLlqkXWZTOGWBHO6uVgeBRJNE//Gqxos5/7j6hzjWzzic0BML47Fb/ rT9DWSDVmzQVSqRYqcoth8sbVkEsquwxO3Fy2w6cMjqLA/ygPgG6dyvfS2AnmB+8 fzM3F/Vl7x5BmMxSB+yEJ5V7XtZDGCYseIruiM1Ut5AUw+huVw9O38P3qqSqNtZF XMI3hqa1vL4on/bSFFVCCO6YA489bTImlm2ZfJQBh/JXITmwG1oLzuxw3NOVvTFH SFtysvFMfk1yg18WNwqWEbUG9GtiB/VOg7Fcxkzj4zFoG3CP+3zPK6WhCy61joV9 so3LF5PgSXlyVOmjNiyS3Ss5QgyVBB/WGaMyYrxQPiZGbN4sKKoYKQWc9QY00I67 xXj6VEYB+YfR+doMKGclGPm2dqbgphScc2HDhoceaEfjufA3ERGjZl/qbYBtwcsl br/UVrMJh5SprzpIxw9K4BNLXKVQTe7gSWUjUTt1tR9bl23RIjx5RYfORqPdvDzJ /UQNiBy781x4cbcWSgIjJ/07h92GGLhS7fzGQdYcOVIu+eFCPV1SRVsjAN60v+9J C2PiYtKiDhNAo1b/ZBIyuw2nnMNZ/99xnGMSV3D2SPRNretGqwE= =vCwY -----END PGP SIGNATURE----- --Apple-Mail=_BAB6FD6D-9471-4858-96BD-99C92F1ACD95-- From nobody Fri Jun 9 13:59:58 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qd2lN5zYrz4c86t for ; Fri, 9 Jun 2023 14:00:00 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-yw1-x1134.google.com (mail-yw1-x1134.google.com [IPv6:2607:f8b0:4864:20::1134]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qd2lN4Dgbz4ZM4; Fri, 9 Jun 2023 14:00:00 +0000 (UTC) (envelope-from mavbsd@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x1134.google.com with SMTP id 00721157ae682-565bd368e19so15728067b3.1; Fri, 09 Jun 2023 07:00:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686319199; x=1688911199; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=+LjFrVpVUv0jij90JIoEZW3iuV2frgBjVWUKPFWxh/E=; b=FEbsHNPugL0+w0Jq6vaLuYs+F+PttJDDJBaOKc43yI8nuZI2ygsCeQ0e/5i5Dru/fB wJiJjKOE5jEmlXhI7e+FJxfrnWBeEs5uvAAOYKfXBm39MgaAO1x6xqpb19ovxKd0puT5 ViKdloCBoA1jH0Grb0cdsoLZtSwffPgRQhE/VTCEw9yA1eg1wONFcQyPhz1MWQtNY8Jp Dz5MRtGLcPGlUzt19ZqDO8bH1Qb0yO1xCUcO62Jc+EtYBfiVOT0674iCiOfpXPQE313y rApKJEA5QSeK/iY65x5UcvgEWzUKTqZbAsn2+CnEStK3oNCRfVNDcm6x8LpaSvwBJlvH Jong== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686319199; x=1688911199; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+LjFrVpVUv0jij90JIoEZW3iuV2frgBjVWUKPFWxh/E=; b=NBXzUGbqp7/bkBUcCYljIAky1CL0foXx2fbabvmk+hLa4XQfORaqQeu8jeIQXeGMGY dgadKVjr1YCVZcBEITb3TvcNGFXWMddv6jZfxyHpjQ3jiypBRAjgezOJQxMY0ell6lpm sC/lF7HNNt//fWdseZ7FXbBj46kotxlB9AKV0ovO47tPc6KSkhg096+DQIoT03Jzht52 5lbKonjs84EiL3LShtLhIFMZQX/HwMbjGDvLrgzZaECK7q2BUHvPw5Db6oQYT3LbavBL BK+jINDn1IPeM4348wBA9kk2SEgi9Mz3ncEkUr8IbNYMDX1U4nz3MW58J5oRwB+FTl7c NkvA== X-Gm-Message-State: AC+VfDzQAizmOtt5tRYK1No2QguyXubhgZCBINH0zEMT5O5RwHG4Rouw z2c+FQijUaBTcChNWUAlf8hPiaVYaeE= X-Google-Smtp-Source: ACHHUZ5TOAs3mMLLEQu0UpnyWHOvqiwqeEWNC3HgUryoE3tzMs5rMJoSQ6DE6Do1Yh6MX87KnUV1Vg== X-Received: by 2002:a81:4946:0:b0:55a:ad7d:3f38 with SMTP id w67-20020a814946000000b0055aad7d3f38mr1220317ywa.26.1686319199430; Fri, 09 Jun 2023 06:59:59 -0700 (PDT) Received: from [10.230.45.5] ([38.32.73.2]) by smtp.gmail.com with ESMTPSA id j77-20020a819250000000b00565862c5e90sm615460ywg.83.2023.06.09.06.59.58 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 09 Jun 2023 06:59:59 -0700 (PDT) Message-ID: <07c3ec8e-5089-d171-0c8d-1916249f9fd7@FreeBSD.org> Date: Fri, 9 Jun 2023 09:59:58 -0400 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: panic(s) in ZFS on CURRENT Content-Language: en-US To: Gleb Smirnoff , current@freebsd.org Cc: mm@freebsd.org References: From: Alexander Motin In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4Qd2lN4Dgbz4ZM4 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N Hi Gleb, There are two probably related PRs upstream: https://github.com/openzfs/zfs/pull/14939 https://github.com/openzfs/zfs/pull/14954 On 09.06.2023 00:57, Gleb Smirnoff wrote: > On Thu, Jun 08, 2023 at 07:56:07PM -0700, Gleb Smirnoff wrote: > T> I'm switching to INVARIANTS kernel right now and will see if that panics earlier. > > This is what I got with INVARIANTS: > > panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (225142071296 <= 225142063104) > > cpuid = 17 > time = 1686286015 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 0xfffffe0160dcea90 > kdb_backtrace() at kdb_backtrace+0x46/frame 0xfffffe0160dceb40 > vpanic() at vpanic+0x21f/frame 0xfffffe0160dcebe0 > spl_panic() at spl_panic+0x4d/frame 0xfffffe0160dcec60 > l2arc_write_buffers() at l2arc_write_buffers+0xcda/frame 0xfffffe0160dcedf0 > l2arc_feed_thread() at l2arc_feed_thread+0x547/frame 0xfffffe0160dceec0 > fork_exit() at fork_exit+0x122/frame 0xfffffe0160dcef30 > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0160dcef30 > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > Uptime: 1m4s > Dumping 5473 out of 65308 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > (kgdb) frame 4 > #4 0xffffffff804342ea in l2arc_write_buffers (spa=0xfffffe022e942000, dev=0xfffffe023116a000, target_sz=16777216) > at /usr/src/FreeBSD/sys/contrib/openzfs/module/zfs/arc.c:9445 > 9445 ASSERT3U(dev->l2ad_hand, <=, dev->l2ad_evict); > (kgdb) p dev > $1 = (l2arc_dev_t *) 0xfffffe023116a000 > (kgdb) p dev->l2ad_hand > $2 = 225142071296 > (kgdb) p dev->l2ad_evict > $3 = 225142063104 > (kgdb) p *dev > value of type `l2arc_dev_t' requires 66136 bytes, which is more than max-value-size > > Never seen kgdb not being able to print a structure that reported to be too big. > -- Alexander Motin From nobody Fri Jun 9 15:39:54 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qd4yh4mwJz4bF6N for ; Fri, 9 Jun 2023 15:39:56 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from www541.your-server.de (www541.your-server.de [213.133.107.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qd4yh4NB0z3MRs; Fri, 9 Jun 2023 15:39:56 +0000 (UTC) (envelope-from mm@FreeBSD.org) Authentication-Results: mx1.freebsd.org; none Received: from sslproxy05.your-server.de ([78.46.172.2]) by www541.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q7eDT-000PI7-KN; Fri, 09 Jun 2023 17:39:55 +0200 Received: from [188.167.171.2] (helo=[10.0.9.122]) by sslproxy05.your-server.de with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q7eDT-000ALC-DK; Fri, 09 Jun 2023 17:39:55 +0200 Message-ID: <397cda46-d02f-5aff-04b1-61fc497ba775@FreeBSD.org> Date: Fri, 9 Jun 2023 17:39:54 +0200 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: panic(s) in ZFS on CURRENT Content-Language: en-US To: Alexander Motin , Gleb Smirnoff , current@freebsd.org References: <07c3ec8e-5089-d171-0c8d-1916249f9fd7@FreeBSD.org> From: Martin Matuska In-Reply-To: <07c3ec8e-5089-d171-0c8d-1916249f9fd7@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Authenticated-Sender: martin@matuska.de X-Virus-Scanned: Clear (ClamAV 0.103.8/26934/Fri Jun 9 09:25:21 2023) X-Rspamd-Queue-Id: 4Qd4yh4NB0z3MRs X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:213.133.96.0/19, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N I will wait with my prepared merge until #14954 gets merged. On 9. 6. 2023 15:59, Alexander Motin wrote: > Hi Gleb, > > There are two probably related PRs upstream: > https://github.com/openzfs/zfs/pull/14939 > https://github.com/openzfs/zfs/pull/14954 > > On 09.06.2023 00:57, Gleb Smirnoff wrote: >> On Thu, Jun 08, 2023 at 07:56:07PM -0700, Gleb Smirnoff wrote: >> T> I'm switching to INVARIANTS kernel right now and will see if that >> panics earlier. >> >> This is what I got with INVARIANTS: >> >> panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed >> (225142071296 <= 225142063104) >> >> cpuid = 17 >> time = 1686286015 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame >> 0xfffffe0160dcea90 >> kdb_backtrace() at kdb_backtrace+0x46/frame 0xfffffe0160dceb40 >> vpanic() at vpanic+0x21f/frame 0xfffffe0160dcebe0 >> spl_panic() at spl_panic+0x4d/frame 0xfffffe0160dcec60 >> l2arc_write_buffers() at l2arc_write_buffers+0xcda/frame >> 0xfffffe0160dcedf0 >> l2arc_feed_thread() at l2arc_feed_thread+0x547/frame 0xfffffe0160dceec0 >> fork_exit() at fork_exit+0x122/frame 0xfffffe0160dcef30 >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0160dcef30 >> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >> Uptime: 1m4s >> Dumping 5473 out of 65308 >> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% >> >> (kgdb) frame 4 >> #4  0xffffffff804342ea in l2arc_write_buffers >> (spa=0xfffffe022e942000, dev=0xfffffe023116a000, target_sz=16777216) >>      at /usr/src/FreeBSD/sys/contrib/openzfs/module/zfs/arc.c:9445 >> 9445                    ASSERT3U(dev->l2ad_hand, <=, dev->l2ad_evict); >> (kgdb) p dev >> $1 = (l2arc_dev_t *) 0xfffffe023116a000 >> (kgdb) p dev->l2ad_hand >> $2 = 225142071296 >> (kgdb) p dev->l2ad_evict >> $3 = 225142063104 >> (kgdb) p *dev >> value of type `l2arc_dev_t' requires 66136 bytes, which is more than >> max-value-size >> >> Never seen kgdb not being able to print a structure that reported to >> be too big. >> > From nobody Fri Jun 9 21:53:22 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QdFFg3pM5z4cYXw for ; Fri, 9 Jun 2023 21:53:27 +0000 (UTC) (envelope-from jamie@catflap.org) Received: from donotpassgo.dyslexicfish.net (donotpassgo.dyslexicfish.net [IPv6:2001:19f0:7400:8808:123::1]) by mx1.freebsd.org (Postfix) with ESMTP id 4QdFFd2llSz3Mhp for ; Fri, 9 Jun 2023 21:53:25 +0000 (UTC) (envelope-from jamie@catflap.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of jamie@catflap.org designates 2001:19f0:7400:8808:123::1 as permitted sender) smtp.mailfrom=jamie@catflap.org; dmarc=pass (policy=none) header.from=catflap.org X-Catflap-Envelope-From: Received: from donotpassgo.dyslexicfish.net (donotpassgo.dyslexicfish.net [209.250.224.51]) by donotpassgo.dyslexicfish.net (8.14.5/8.14.5) with ESMTP id 359LrMl5041364; Fri, 9 Jun 2023 22:53:22 +0100 (BST) (envelope-from jamie@donotpassgo.dyslexicfish.net) Received: (from jamie@localhost) by donotpassgo.dyslexicfish.net (8.14.5/8.14.5/Submit) id 359LrMJA041363; Fri, 9 Jun 2023 22:53:22 +0100 (BST) (envelope-from jamie) From: Jamie Landeg-Jones Message-Id: <202306092153.359LrMJA041363@donotpassgo.dyslexicfish.net> Date: Fri, 09 Jun 2023 22:53:22 +0100 Organization: Dyslexic Fish To: freebsd-current@freebsd.org Cc: jamie@catflap.org Subject: BUG: chflags(1) fails to remove uarch flag with hardlinked files (ZFS) User-Agent: Heirloom mailx 12.4 7/29/08 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (donotpassgo.dyslexicfish.net [209.250.224.51]); Fri, 09 Jun 2023 22:53:22 +0100 (BST) X-Spamd-Result: default: False [-3.69 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.99)[-0.994]; NEURAL_HAM_SHORT(-0.99)[-0.993]; DMARC_POLICY_ALLOW(-0.50)[catflap.org,none]; R_SPF_ALLOW(-0.20)[+mx:dyslexicfish.net]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[jamie]; BLOCKLISTDE_FAIL(0.00)[2001:19f0:7400:8808:123::1:server fail,209.250.224.51:server fail]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; HAS_ORG_HEADER(0.00)[]; ASN(0.00)[asn:20473, ipnet:2001:19f0:7400::/38, country:US] X-Rspamd-Queue-Id: 4QdFFd2llSz3Mhp X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Using chflags to remove "uarch" on more than one file at a time (but when that file count is an even number) when the files are hardlinked (have the same inode) and on ZFS doesn't work. This becomes more problematic when using "find... -exec chflags nouarch {} +" or "chmod -R". Further details are in the bug report I just raised: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=271925 Cheers, Jamie From nobody Sat Jun 10 09:21:51 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QdXYv0Cxbz4cjTF for ; Sat, 10 Jun 2023 09:23:31 +0000 (UTC) (envelope-from grembo@freebsd.org) Received: from mail.evolve.de (mail.evolve.de [213.239.217.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.evolve.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QdXYs1bLrz46r5; Sat, 10 Jun 2023 09:23:29 +0000 (UTC) (envelope-from grembo@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 213.239.217.29 is neither permitted nor denied by domain of grembo@freebsd.org) smtp.mailfrom=grembo@freebsd.org; dmarc=none Received: by mail.evolve.de (OpenSMTPD) with ESMTP id 68000590; Sat, 10 Jun 2023 09:23:21 +0000 (UTC) Received: by mail.evolve.de (OpenSMTPD) with ESMTPSA id 0aa0fa8e (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Sat, 10 Jun 2023 09:23:21 +0000 (UTC) Date: Sat, 10 Jun 2023 11:21:51 +0200 From: Michael Gmelin To: Warner Losh Cc: Michael Gmelin , Glen Barber , FreeBSD Current Subject: Re: CURRRENT snapshot won't boot due missing ZFS feature Message-ID: <20230610112151.306e4c01.grembo@freebsd.org> In-Reply-To: References: <20230608181115.7c97ea19.grembo@freebsd.org> <20230608162012.GV1219@FreeBSD.org> <20230608190623.14ab4ba8.grembo@freebsd.org> <20230608191637.21101c39.grembo@freebsd.org> X-Face: $wrgCtfdVw_H9WAY?S&9+/F"!41z'L$uo*WzT8miX?kZ~W~Lr5W7v?j0Sde\mwB&/ypo^}> +a'4xMc^^KroE~+v^&^#[B">soBo1y6(TW6#UZiC]o>C6`ej+i Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWJBwe5BQDl LASZU0/LTEWEfHbyj0Txi32+sKrp1Mv944X8/fm1rS+cAAAACXBIWXMAAAsTAAAL EwEAmpwYAAAAB3RJTUUH3wESCxwC7OBhbgAAACFpVFh0Q29tbWVudAAAAAAAQ3Jl YXRlZCB3aXRoIFRoZSBHSU1QbbCXAAAAAghJREFUOMu11DFvEzEUAGCfEhBVFzuq AKkLd0O6VrIQsLXVSZXoWE5N1K3DobBBA9fQpRWc8OkWouaIjedWKiyREOKs+3PY fvalCNjgLVHeF7/3bMtBzV8C/VsQ8tecEgCcDgrzjekwKZ7TwsJZd/ywEKwwP+ZM 8P3drTsAwWn2mpWuDDuYiK1bFs6De0KUUFw0tWxm+D4AIhuuvZqtyWYeO7jQ4Aea 7jUqI+ixhQoHex4WshEvSXdood7stlv4oSuFOC4tqGcr0NjEqXgV4mMJO38nld4+ xKNxRDon7khyKVqY7YR4d+Cg0OMrkWXZOM7YDkEfKiilCn1qYv4mighZiynuHHOA Wq9QJq+BIES7lMFUtcikMnkDGHUoncA+uHgrP0ctIEqfwLHzeSo+eUA66AqzwN6n 2ZHJhw6Qh/PoyC/QENyEyC/AyNjq74Bs+3UH0xYwzDUC4B97HgLocg1QLYgDDO1v f3UX9Y307Ew4AHh67YAFFsxEpkXwpXY3eIgMhAAE3R19L919nNnuD2wlPcDE3UeT L2ytEICQib9BXgS2fU8PrD82ToYO1OEmMSnYTjSqSv9wdC0tPYC+rQRQD9ESnldF CyqfmiYW+tlALt8gH2xrMdC/youbjzPXEun+/ReXsMCDyve3dZc09fn2Oas8oXGc Jj6/fOeK5UmSMPmf/jL+GD8BEj0k/Fn6IO4AAAAASUVORK5CYII= List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-2.09 / 15.00]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.993]; MIME_GOOD(-0.10)[text/plain]; BLOCKLISTDE_FAIL(0.00)[213.239.217.29:server fail]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; ASN(0.00)[asn:24940, ipnet:213.239.192.0/18, country:DE]; FREEFALL_USER(0.00)[grembo]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_SOFTFAIL(0.00)[~all:c]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4QdXYs1bLrz46r5 X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On Thu, 8 Jun 2023 11:32:19 -0600 Warner Losh wrote: > On Thu, Jun 8, 2023, 11:18 AM Michael Gmelin > wrote: > > > > > > > Tried today's snaphot, same problem. > > > > > > # reboot > > > Waiting (max 60 seconds) for system process `vnlru' to stop... > > > done Waiting (max 60 seconds) for system process `syncer' to > > > stop... Syncing disks, vnodes remaining... 0 0 0 0 done > > > All buffers synced. > > > Uptime: 4m14s > > > Consoles: userboot > > > > > > FreeBSD/amd64 User boot lua, Revision 1.2 > > > ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2 > > > ERROR: cannot open /boot/lua/loader.lua: no such file or > > > directory. > > > > > > > > > Type '?' for a list of commands, 'help' for more detailed help. > > > OK > > > > > > > > > That's after installing CURRENT in a fresh vm managed by vm-bhyve > > > using bsdinstall's automatic ZFS option. > > > > > > > Thinking about this, it's possible that vm-bhyve is using the zfs > > boot loader from the host machine. > > > > Please consider this noise, unless you hear from me again. > > > > Yes. It does. This can be an unfortunate design choice at times. > For completeness sake, this is how I boot 14.0 on 13.2 using sysutils/vm-bhyve: ISO=FreeBSD-14.0-CURRENT-amd64-20230608-653738e895ba-263444-bootonly.iso export ISO cd /mountpoint/for/pool/vm vm iso https://download.freebsd.org/snapshots/ISO-IMAGES/14.0/$ISO mkdir .loaders tar --strip-components 1 -C .loaders -xf .iso/$ISO boot/userboot.so mv .loaders/userboot.so .loaders/userboot14.so vm create test14 sysrc -f test14/test14.conf memory=1G sysrc -f test14/test14.conf \ bhyveload_loader="$(realpath .loaders/userboot14.so)" OS installation is done the usual way (using tmux instead of cu in this example): pkg install -y tmux sysrc -f .config/system.conf console=tmux vm install test14 $ISO tmux attach -t test14 Best Michael -- Michael Gmelin From nobody Sat Jun 10 12:24:10 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QdcZP6GFtz4cJvr for ; Sat, 10 Jun 2023 12:24:13 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QdcZP5pp3z4Yhb; Sat, 10 Jun 2023 12:24:13 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1686399853; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=kzn2NEGOwn9Dxl+SL0AN+f3wp9OUEscTzZiAhscsF08=; b=sAP+LnnHK3p6MYu6Ot99U59HUM1uCP6C/NESRdrHfNfv34r7q3c8v+cRCrVLP56eA3Q8La Vl/iOskAvQtwPrlCSV8WqaoQBsTChcPqZfDPFgdqZw17xvG2p/axCqon6n+GPCQFRBybfe fTgcuS9xs019SjKV9mkoWHt7CutV20rru3Ue/AxBf/ORVKqfR2bE6TV3KGMr2zOK/luvB+ QcaC5FF8BVZVh10lGbSVyS9p6G+/XdK3+ot3lWTmGhxjJleiyOJDR5h879idK5T73EaVa6 BdOFQ3jj2mdS1af2gPia0Dr3whYk7tAvthRngo1Ht9seAgPr+/BpomNiw2zyvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1686399853; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=kzn2NEGOwn9Dxl+SL0AN+f3wp9OUEscTzZiAhscsF08=; b=Nxpg1YR0rocU5OZp1ngmr6yd/eG7YMGvhcCxs2ym2p1UoHyuwwUSLTI5ss+Fz4BqNI7Bk9 cNFv98+FUVrGYf5MJy4vqR0V4J5AWut7sCJVpcL3NZaGhqoCPG2xs7BBe77h8SKXojQtOR ivEAPlNTmShwnpaV5fhoN7WNWd5LsMhN+XIQAXsdtoBwXqlrpsQmxFE+5usaGGkrvdRTth 81FRWLlhTxLvVUElaljZVnAV9AqG6l4s9eKWAvsTZG5jkbjRFxgzrr3fOfxnWESCBx8lF+ lrbwAqki3bCu8rl+0u+LTLcfQ/XcYotStcvsOqaQuWuyVZxFGjUyoYp/XFieew== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1686399853; a=rsa-sha256; cv=none; b=viuaKN1Z0Nhh+bQU2jptGqeNhsKgKWoBw7HeSykigvyVYuqZ9Nv7l7p/+yLuhWS847R09X S6loeIUVNPQZdhIrsJLUuvbiE1Vpx80yN+Xmc7CznarO1xJOR6TN7OTgbafdnzjQ56wSPP DGCbtMnhuIDWf7KDu8QMdvhQec7ClZE4BJpVsJCvB4oeWgVmoWh0X6XyRvuLCxx/hhWCOg KMBWVM6cc9UoNrJx14642elmxzCUnAAkkzP8cb/rfCeRoz003IzEizx3xH3wXlgasn2fSe NpqKCKYlqrr0rzVg893RAljfAVONwiDsPVK4pd2sC17LX8aI3Lk64BZtmJ4N1g== Received: from [192.168.1.10] (host-2-100-161-236.as13285.net [2.100.161.236]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: grahamperrin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4QdcZP1rVXzfCC; Sat, 10 Jun 2023 12:24:13 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Message-ID: Date: Sat, 10 Jun 2023 13:24:10 +0100 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: ZFS: panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed Content-Language: en-US From: Graham Perrin To: Martin Matuska References: <84482bfa-e6a4-01a6-36c3-6b6d06e3b993@freebsd.org> Cc: FreeBSD CURRENT Organization: FreeBSD In-Reply-To: <84482bfa-e6a4-01a6-36c3-6b6d06e3b993@freebsd.org> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------CP5pRvDOCfyz7SGJ43m3Q0Wk" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------CP5pRvDOCfyz7SGJ43m3Q0Wk Content-Type: multipart/mixed; boundary="------------wsfP4U9YRWEVYFIrqLa7x1T8"; protected-headers="v1" From: Graham Perrin To: Martin Matuska Cc: FreeBSD CURRENT Message-ID: Subject: Re: ZFS: panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed References: <84482bfa-e6a4-01a6-36c3-6b6d06e3b993@freebsd.org> In-Reply-To: <84482bfa-e6a4-01a6-36c3-6b6d06e3b993@freebsd.org> --------------wsfP4U9YRWEVYFIrqLa7x1T8 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 T24gMjcvMDUvMjAyMyAxNjozMCwgR3JhaGFtIFBlcnJpbiB3cm90ZToNCj4gVGhyZWUgcGFu aWNzIHdpdGggYzJjOWFjODhjMmJiICgyMDIzLTA1LTI2LCAxNDAwMDg5KToNCj4NCj4g4oCm DQoNCk1hcnRpbiwgaWYgeW91J2QgbGlrZSB0byB0YWtlIA0KPGh0dHBzOi8vYnVncy5mcmVl YnNkLm9yZy9idWd6aWxsYS9zaG93X2J1Zy5jZ2k/aWQ9MjcxNzcyPiwgcGxlYXNlIGdvIA0K YWhlYWQuDQoNClRoYW5rcw0KDQo= --------------wsfP4U9YRWEVYFIrqLa7x1T8-- --------------CP5pRvDOCfyz7SGJ43m3Q0Wk Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsFAmSEa2oFAwAAAAAACgkQt2dIb0oY1Aty vA/9GHqeAvpe5THykY/EdgWnS4gPSXTFxpAg9RVq9vxCwV2paLHbly2WJ2qG71p+O/wrJV38NWKC g7Ydsss++8A5WpEtxHF3aGOezV3kZfqWwM8jDC64jLwzdw5X6deVYjHUBtDokpUHfRRdcyh8+aSt AqojlQG8RlwWMv9gPEeYeEuOCuytEiWJ9JdfPZvS0dH1qPhGhvTYpi02sUVcTyCnceW9/BXos+lY eqiBzUzgwq3BuzLnB6JO2OiyLS6YY9yAoTzbDAMQ/vAsQ/LbG5szgnpm7YmamCGsAGH2ryPH5X1d LULpTFH0AW8Jzk5ikinN5PaYCxuAyET1Fyh+lqtjcZf4bLDA85ZZa6O5TdcCJ6cUdtvZkcoLNvq6 0NLiR85UsNjow1z4aGa7/R2DwVFbsY9xsrPG0Ws3RAIZzwCMU2aDGVEeHZdHL+6vZ6/a46V3PeyU Ze31qdy3CiRxOYcvw8KxD2sADlfdRpkpLxq803n22egnZd1d69Cpy84D7qykFQT9GMcqcJmBY66F zdlLuenogbScYdK3tQBGFpB7yArbDEnlpuvc/o6/xEey5mPyS/lNtNzImBLeMQRgdqRD+mU/NhmQ ysbY5eAZRfk9/pMat11VDbNCW2fMkMn9hn0UKorzp72RMzJ3XMAXEiFUAmjEvxPvT267BNBOyd0N qg8= =Vwfe -----END PGP SIGNATURE----- --------------CP5pRvDOCfyz7SGJ43m3Q0Wk--