On Mon, Jul 15, 2024 at 3:57=E2=80=AF= PM Colin Percival <cperciva@free= bsd.org> wrote:

On 7/15/24 14:33, Tomoaki AOKI wrote:
> On Mon, 15 Jul 2024 11:58:16 -0700
> Colin Percival <cperciva@freebsd.org> wrote:
>> Extending the minor-release support period might be possible, but = that
>> would depend on portmgr and secteam and I can't speak for them= .=C2=A0 One issue
>> which would certainly come up is kernel module packages -- our pac= kages
>> are built for each stable branch on the oldest currently supported= release,
>> which means that e.g. new features in 14.1 can't be used until= 14.0 is EoL;
>> this is a problem particularly for graphics drivers.
>
> How do you think about flavorizing kmod ports in kmod.mk and provide
> pkgs for latest patch release (like 14.0-p8) of all supported
> point releases (like 14.0) [1]?
> Does it look possible and feasible?

Rebuilding kernel modules for each patch level shouldn't be necessary.= =C2=A0 If
we break KBI in a security or errata update, something has gone astonishing= ly
wrong.

Flavouring kernel module ports for each minor release -- possibly building = in
in an oldest-supported-release jail but with the relevant /usr/src/sys tree= --
might work well?=C2=A0 But that's a question for portmgr; I don't k= now enough about
how package building works to know how feasible this would be.

People have talked about "stacking" repos= to accomplish this. We'd build per-minor
release images for = .ko's. I'm not sure what the sticking points are for doing this,
though.

Ideally, we'd keep the same KB= I per major release, but that ideal has fallen
short.
<= br>
Warner=C2=A0

--000000000000e2a936061d506e76-- From nobody Mon Jul 15 22:20:13 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WNGq30Sg5z5QJYV for ; Mon, 15 Jul 2024 22:20:15 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from mail.tarsnap.com (mail.tarsnap.com [54.86.246.204]) by mx1.freebsd.org (Postfix) with SMTP id 4WNGq26z8Vz3yfJ for ; Mon, 15 Jul 2024 22:20:14 +0000 (UTC) (envelope-from cperciva@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: (qmail 14072 invoked from network); 15 Jul 2024 22:20:14 -0000 Received: from unknown (HELO framework.daemonology.net) (127.0.0.1) by mail.tarsnap.com with SMTP; 15 Jul 2024 22:20:14 -0000 Received: (qmail 3661 invoked from network); 15 Jul 2024 22:20:13 -0000 Received: from unknown (HELO ?127.0.0.1?) (127.0.0.1) by localhost with SMTP; 15 Jul 2024 22:20:13 -0000 Message-ID: <22612d5f-3960-4390-a484-3bd307820126@freebsd.org> Date: Mon, 15 Jul 2024 15:20:13 -0700 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Change to FreeBSD release scheduling etc. To: Warner Losh Cc: Tomoaki AOKI , Peter , freebsd-stable@freebsd.org References: <3598c6db-6d8a-4efa-b5fc-1fc697608860@freebsd.org> <20240716063356.f0993ae6f7648ae19c1ae33f@dec.sakura.ne.jp> <0fb67778-d279-420b-b482-5ce36e08a1be@freebsd.org> Content-Language: en-US From: Colin Percival Autocrypt: addr=cperciva@freebsd.org; keydata= xsFNBGWMSrYBEACdWRqDn3B3SKO7IG0/fGHYtfs26f3Q5QeAcasy1fQLniwGQWn5rlILhbCD K/jdNoDm5Zxq20eqyffoDNObCjnHgg4tGANdi+RmDy+7CDpE789H8dss9y7Pt5DlGGAXQQnt hxush3EYS/Ctprd9UUL/lzOOLOU1aNtzB84tNrJBtcJmL7OYHfyTSNFxvedqJrrasejIQOLI t/DQ89BPzz+vsKHz7FJPXh3fsVkzLA00DJYcfkgxyABfJNA7U6yMwd4DVSdx/SsvfIDMVXnu UXCXswo106WPZbYGlZPpq0wW6iibtTerJix+8AeuwXvl9O1p8yESK4ErkIxCnmghTSz+pdzj z/6xBRkdDM9VdZ0r+CzsaNXMpDOzFuKyjaiYBdgCLljbDnXIHFcqXenrZ7Xwkm09g/M4uVSh pIUG2RYa6tsHSQoGCp3f2RZv1znfViKQFbbL83QjtPA20AhseZSYbHp1FPhXyy9J0wkGL16L e99g6gdGeIRE82BZjBjKGDkoyDPq+oDRSFl8NtzmIKy+cfz00nViqcTF4bREXEawFGhlpO0X O9q8mijI9iFB6zaPBiSdJGBL5ML5qLTNCl8Zlf4m1TBvmRTqF/lzMHVXHidDoUhpSh/y3AFZ 1KrYc27ztJQywDJPJPWPbtY8YhFLFs377gfP8WldsZjzp8nvoQARAQABzSVDb2xpbiBQZXJj aXZhbCA8Y3BlcmNpdmFARnJlZUJTRC5vcmc+wsGRBBMBCAA7FiEEglY7hNBiDtwN+4ZBOJfy 4i5lrT8FAmWMSrYCGwMICwkNCAwHCwMFFQoJCAsFFgMCAQACHgUCF4AACgkQOJfy4i5lrT++ ig/9GZKdN2fHSyrANKZX38ivd7IX2wAYouqH9DrQM94W8IciaDLmarN4Pl9mY+aucMwQUSyp uNtKOJwKqhVVaalF9Zw0sRMH4CJuvT7vKCtZ3q1Okb7soRvFte4d+vXhvPxCvBFDA5JzU7Lg DR5eqqcvF1dN1OuCq16pl0zCOSH/Jr5ToE3LM3Av1KBGcZD7ZSzHRWsFjV5AOUJKySuA3GwJ e/jASQcQ0YfCnru8ntLmYg/2SKvZFlfthZiCBnAppMt4n4BUAw3TDvf10HIDtdneejawcbLS gofLCvGqumwbZYAMKWrFzT4+7KQvr0pOw8QD7EbxnB4f9hQ7UiVF8qWsyKU3iv6b5JLhbS59 ooKRccyOvdMLcVJ0ZdpqoxrNv061ZUqLL5RiWjBlc1qjBnDxeg5oyM0rT8WLftdgvyH6RQt0 KWngumBAT5AT2DUYL8Uz1490cqfO9K4yEGZAJB9XRVX1g2IWTOjae+0g9ZII+h91UngFz+Rz aKDeseKBbCGDOFXx1TqKiHl2g255ZnUxKYTlucFtguv4gDGBgEk4G9JaEWBw1IWblcKhxH7L 2vWsUhvwghjIxHdO/RkeIeHvSp4YZxCJ7a3TaJLYAlwYopfTKVzNhcDY5h5syEuoHjyJCxXK SyoJYAVu8Yl2KUhvOtOmL1VZ6xyHnpdMRWKJZ5jOwU0EZYxKtgEQANYfgbtUMVnhjxDHhWLp g5kLHK3YW0TfJKzpXqDB7NiqxHofn4OcbZnVC3MKggcbs9o1/UtsjnlsG8550PfiYkDXvPiO RJwgbGs6MGIDK797C6cnBLQ8xwBa9SL4cl5iQFnhWmt6vwnJ+an/cm5JpYves3wL7jV09qU9 57hkHXEUcl38r4FssZzVcLKPUVTa3Un+QGRTGDGe/f4ctjMaqv0ZCM+l2ixPhf/vqESrfSLv V/+T3dmtUfXjazO3SABvsHwxgGuTTYOlKoPCaebr+BRdqm0xeIShoIlhvTI8y4clchqx/Uxg UG5X2kvU13k3DS3Q8uLE4Et9x1CcZT6WGgBZSR6R0WfD0SDnzufNnRWJ0dEPA2MtJHE7+85R Vi9j/IgZV+y5Ur+bnPkjDG1s2SVciX5v9HQ0oilcBhvx0j5lGE9hhurD9F+fCvkr4KdbCknE 6Y8ce8pCNBUoB/DqibJivOzTk9K9MGB5x0De5TerIrFiaw3/mQC9nGeO9dtE7wvDJetWeoTq 4BEaCzpufNqbkpOaTQILr4V6Gp7M6v97g83TVAwZntz/q8ptwuKQPZ2JaSFLZn7oWUpYXA5s +SIODFHLn6iMoYpBQskHQjnj4lEPJadl4qj+ZKA89iDAKsniyoFXsbJe2CPbMS1yzBxKZq6K D/jpt7BOnuHr/JrXABEBAAHCwXYEGAEIACAWIQSCVjuE0GIO3A37hkE4l/LiLmWtPwUCZYxK tgIbDAAKCRA4l/LiLmWtP3jmEACQrh9gWe8F1Tkw3m6VoHKwLc5he4tX3WpQa//soPO6iGG3 S3WPruQ46NrAaAojoOcKI9UONDO5rxG0ZTX53S+lu2EO47jbcLwOCjaEpjKpDRt9ZXBQE8Xl mtBE9Bp3W9gpjB1nE3KNM1mJYgsK0QdRpwwfh4pVgGpOj8j23I6MCK+v99zEBnpgCn2GX8W/ kctRXHqWwndHysOJtRP/zrl7dDaABF1f9efUl0LL3TD3GJ9VDz+DNOin/uK2a1hiJo8QzTRk PpfUQ2ebzDsrd1i/pOWkMSkdH+rEu4AGrXWtaBwrMyrGkL6Icb6yO+P9/z0W2wlgBf3P1YRt JPgQt/Dj3yvA/UnaV/QmuVQPjl13o24UnJGsZM8XGnNdfWBKkC1Q6VXC4QT+dyBHYH9MuE9d 6oGl8pFM1+cTfEfbM62/rRoPkF1yHMsI/903VxEvuUIKfhEZAVLFyHldooNxuchntHQP9y8J 8Ou9bWYQP7MnEn+kwSwrZkjurfPkan+xQvp6dDYnj3V0GwA5pprBMaB928VIDVOv+1PNQI3t Cvk5VPv/skq+TJRMHW7bFSt8PRa91cUf1FOLIz9APDiJOzXkwxUEHGV3zPSaUhs1JYjyBeGT wDAvtLUdjOnRhEUOwlnIrztmvyciutjJoVzKEEjj5WXnHk9L9kQ1bpAjkjTONw== In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:14618, ipnet:54.86.0.0/16, country:US] X-Rspamd-Queue-Id: 4WNGq26z8Vz3yfJ On 7/15/24 15:05, Warner Losh wrote: > On Mon, Jul 15, 2024 at 3:57 PM Colin Percival > wrote: > Flavouring kernel module ports for each minor release -- possibly building in > in an oldest-supported-release jail but with the relevant /usr/src/sys tree -- > might work well? But that's a question for portmgr; I don't know enough about > how package building works to know how feasible this would be. > > People have talked about "stacking" repos to accomplish this. We'd build per-minor > release images for .ko's. I'm not sure what the sticking points are for doing > this, > though. That would be good, but drm-14.1-61-kmod-6.1.69_2 (aka have multiple versions of ports and require users to kmod packages to pick the right now) would be better than what we have now. > Ideally, we'd keep the same KBI per major release, but that ideal has fallen > short. Having a stable KBI only solves half of the problem. With DRM especially it's useful for people running the latest minor release to use kmods which make use of functionality which was added in the latest release before the previous release is EoLed. -- Colin Percival FreeBSD Release Engineering Lead & EC2 platform maintainer Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid From nobody Mon Jul 15 22:59:49 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WNHhx4crpz5QN7H for ; Mon, 15 Jul 2024 23:00:01 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WNHhx30ffz44NV for ; Mon, 15 Jul 2024 23:00:01 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pg1-x534.google.com with SMTP id 41be03b00d2f7-7163489149eso3538196a12.1 for ; Mon, 15 Jul 2024 16:00:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1721084400; x=1721689200; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=vBQ6Qew+AakdIuKgTjXTy2qol+w9TgFrWRFqV52AjOQ=; b=cxzmRUvnICJAvh7yjlEFBnr18PQyl+o8ziaAr/mvWvSVJFjFcDrTBkVpeA89QDhdzH dDNi8AwKoMUSs9UpPsuWQOn9qyX6qcNegzwNVOx6bGcYFx7q+yh1UtwJjywbGE0weEPS z773UdyPyBnIEMxQn42Z1jT/Ko6qzAJ+EMymjv+kThYd3XtGqVTcUWhW50QNc4iZGA3+ CGmNe+EUvJgYi921wILKzUN66WxlLLPDEHQgNZw/s0Z3IIQQnx8POm4J0iipJQDzH3fu ItTsvNxhIpUq/J+ZkLq3EqT3JHV38741vQtP3eLVS+lVe9wGSrcZe2tjb1f33d1opy8x qFSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721084400; x=1721689200; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=vBQ6Qew+AakdIuKgTjXTy2qol+w9TgFrWRFqV52AjOQ=; b=hEVbcuKYnHlHY9rgBi4p0r73UYrWx1GrdCKD3hiGwkmiQokbi3udC9mFu60pw+EdeV 9oIBaUkyV9my+3Dn9g14XJN1a4pDM1FVQgq5fPhucneWEgeXoOmD/jixfvd7xORZ41E7 JdAozvAu7/RepiNrXOlaxsHY7kNQYOtKzIl0VGiKRVqBLKQ8Y9CNnrfcTpEN7K8wWW9J Jx43b70HQtHEiz7Jir8mibhTKVvzHqC0R6j7u019xXUkXJ3s2KMPL5MNk7uTuKURuMF8 JRt/rCPZwVTR6j6H3km37pu8Vl0K5iJKB8lnXrW7bDjgISL9LgDU4JRg8WG6Ygfj4mtg Cj3A== X-Forwarded-Encrypted: i=1; AJvYcCU66px1K8ItvGvWMjS11k5ePzvJDeJqOwHkMohlsNb2lwI3/6h8UkKPGmBbiL4j756YhogCZBpKMv6bX+yCC77ESXSN4lvOrKQ0cw== X-Gm-Message-State: AOJu0Yx/N8inasYCF7fH4L4JOHltcO0HbL0PD0kPeKJ1tHbDl7zC1i42 jGfjBN2igC878pNTNX0gHvfvrH/312BwQp1/F9aKLnP/L9ITuXHY4xKBZMh/kEO7mvrQOIEPHMr MkFdoJ3+zZO5teXoeC9Z3n57iapNyLERX7fvQMA== X-Google-Smtp-Source: AGHT+IGZgOqmA0LvO4e1lCncCfR5bZOFDJQiXDl6e0O8EmvBljCvPf6wYE2130SiKOt5AdZSknT3VcOckEv3+2IzCkM= X-Received: by 2002:a05:6a21:398b:b0:1c3:b61c:57cb with SMTP id adf61e73a8af0-1c3f12b5ae9mr400101637.53.1721084400173; Mon, 15 Jul 2024 16:00:00 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 References: <3598c6db-6d8a-4efa-b5fc-1fc697608860@freebsd.org> <20240716063356.f0993ae6f7648ae19c1ae33f@dec.sakura.ne.jp> <0fb67778-d279-420b-b482-5ce36e08a1be@freebsd.org> <22612d5f-3960-4390-a484-3bd307820126@freebsd.org> In-Reply-To: <22612d5f-3960-4390-a484-3bd307820126@freebsd.org> From: Warner Losh Date: Mon, 15 Jul 2024 16:59:49 -0600 Message-ID: Subject: Re: Change to FreeBSD release scheduling etc. To: Colin Percival Cc: Tomoaki AOKI , Peter , freebsd-stable@freebsd.org, Baptiste Daroussin Content-Type: multipart/alternative; boundary="0000000000004d286e061d513030" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4WNHhx30ffz44NV --0000000000004d286e061d513030 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Jul 15, 2024 at 4:20=E2=80=AFPM Colin Percival wrote: > On 7/15/24 15:05, Warner Losh wrote: > > On Mon, Jul 15, 2024 at 3:57=E2=80=AFPM Colin Percival > > wrote: > > Flavouring kernel module ports for each minor release -- possibly > building in > > in an oldest-supported-release jail but with the relevant > /usr/src/sys tree -- > > might work well? But that's a question for portmgr; I don't know > enough about > > how package building works to know how feasible this would be. > > > > People have talked about "stacking" repos to accomplish this. We'd buil= d > per-minor > > release images for .ko's. I'm not sure what the sticking points are for > doing > > this, > > though. > > That would be good, but drm-14.1-61-kmod-6.1.69_2 (aka have multiple > versions > of ports and require users to kmod packages to pick the right now) would = be > better than what we have now. > Indeed, but the upgrade path sucks for that. And all the infrastructure is in place to start doing the per-minor-release repos to augment the per major release ones. I'm unsure what the holdup is, though. I've cc'd baptiste who may know. It's the least bad solution, though. It solves the problem for release points, but not for people on stable between releases (except accidentally sometimes when things haven't yet diverged), nor for current (since it's nothing but breakage of KBIs :). For that problem, only a build-from-source-for-each-new-kernel-build can work. > > Ideally, we'd keep the same KBI per major release, but that ideal has > fallen > > short. > > Having a stable KBI only solves half of the problem. With DRM especially > it's > useful for people running the latest minor release to use kmods which mak= e > use > of functionality which was added in the latest release before the previou= s > release is EoLed. > Functionality added where? I'm not following the point you're making here... But since KBI stability isn't going to happen (at least given our past track record and a lack of tools to enforce it), it may be just an academic exercise. Warner --0000000000004d286e061d513030 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Mon, Jul 15, 2024 at 4:20=E2=80=AF= PM Colin Percival <cperciva@free= bsd.org> wrote:

On 7/15/24 15:05, Warner Losh wrote:
> On Mon, Jul 15, 2024 at 3:57=E2=80=AFPM Colin Percival <cperciva@freebsd.org > <mailto:c= perciva@freebsd.org>> wrote:
>=C2=A0 =C2=A0 =C2=A0Flavouring kernel module ports for each minor relea= se -- possibly building in
>=C2=A0 =C2=A0 =C2=A0in an oldest-supported-release jail but with the re= levant /usr/src/sys tree --
>=C2=A0 =C2=A0 =C2=A0might work well?=C2=A0 But that's a question fo= r portmgr; I don't know enough about
>=C2=A0 =C2=A0 =C2=A0how package building works to know how feasible thi= s would be.
>
> People have talked about "stacking" repos to accomplish this= . We'd build per-minor
> release images for .ko's. I'm not sure what the sticking point= s are for doing
> this,
> though.

That would be good, but drm-14.1-61-kmod-6.1.69_2 (aka have multiple versio= ns
of ports and require users to kmod packages to pick the right now) would be=
better than what we have now.

Indeed, b= ut the upgrade path sucks for that. And all the infrastructure is in place<= /div>

to start doing the per-minor-release repos to augment the per maj= or release ones.

I'm unsure what the holdup is, though. I'= ;ve cc'd baptiste who may know.

It's the l= east bad solution, though. It solves the problem for release points, but

not for people on stable between releases (except accidentally some= times when

things haven't yet diverged), nor for current (sin= ce it's nothing but breakage of KBIs :).

For that problem, on= ly a build-from-source-for-each-new-kernel-build can work.

= =C2=A0

> Ideally, we'd keep the same KBI per major release, but that ideal = has fallen
> short.

Having a stable KBI only solves half of the problem.=C2=A0 With DRM especia= lly it's
useful for people running the latest minor release to use kmods which make = use
of functionality which was added in the latest release before the previous<= br> release is EoLed.

Functionality added w= here? I'm not following the point you're making here... But since

KBI stability isn't going to happen (at least given our past t= rack record and a lack of

tools to enforce it), it may be just an= academic exercise.

Warner

--0000000000004d286e061d513030-- From nobody Mon Jul 15 23:58:16 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WNK0B0wVJz5QSqp for ; Mon, 15 Jul 2024 23:58:18 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from mail.tarsnap.com (mail.tarsnap.com [54.86.246.204]) by mx1.freebsd.org (Postfix) with SMTP id 4WNK096MDrz49MZ for ; Mon, 15 Jul 2024 23:58:17 +0000 (UTC) (envelope-from cperciva@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: (qmail 16735 invoked from network); 15 Jul 2024 23:58:17 -0000 Received: from unknown (HELO framework.daemonology.net) (127.0.0.1) by mail.tarsnap.com with SMTP; 15 Jul 2024 23:58:17 -0000 Received: (qmail 3969 invoked from network); 15 Jul 2024 23:58:16 -0000 Received: from unknown (HELO ?127.0.0.1?) (127.0.0.1) by localhost with SMTP; 15 Jul 2024 23:58:16 -0000 Message-ID: <25a443cb-35d3-4d32-bb94-7aa18d4a3813@freebsd.org> Date: Mon, 15 Jul 2024 16:58:16 -0700 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Change to FreeBSD release scheduling etc. To: Warner Losh Cc: Tomoaki AOKI , Peter , freebsd-stable@freebsd.org, Baptiste Daroussin References: <3598c6db-6d8a-4efa-b5fc-1fc697608860@freebsd.org> <20240716063356.f0993ae6f7648ae19c1ae33f@dec.sakura.ne.jp> <0fb67778-d279-420b-b482-5ce36e08a1be@freebsd.org> <22612d5f-3960-4390-a484-3bd307820126@freebsd.org> Content-Language: en-US From: Colin Percival Autocrypt: addr=cperciva@freebsd.org; keydata= xsFNBGWMSrYBEACdWRqDn3B3SKO7IG0/fGHYtfs26f3Q5QeAcasy1fQLniwGQWn5rlILhbCD K/jdNoDm5Zxq20eqyffoDNObCjnHgg4tGANdi+RmDy+7CDpE789H8dss9y7Pt5DlGGAXQQnt hxush3EYS/Ctprd9UUL/lzOOLOU1aNtzB84tNrJBtcJmL7OYHfyTSNFxvedqJrrasejIQOLI t/DQ89BPzz+vsKHz7FJPXh3fsVkzLA00DJYcfkgxyABfJNA7U6yMwd4DVSdx/SsvfIDMVXnu UXCXswo106WPZbYGlZPpq0wW6iibtTerJix+8AeuwXvl9O1p8yESK4ErkIxCnmghTSz+pdzj z/6xBRkdDM9VdZ0r+CzsaNXMpDOzFuKyjaiYBdgCLljbDnXIHFcqXenrZ7Xwkm09g/M4uVSh pIUG2RYa6tsHSQoGCp3f2RZv1znfViKQFbbL83QjtPA20AhseZSYbHp1FPhXyy9J0wkGL16L e99g6gdGeIRE82BZjBjKGDkoyDPq+oDRSFl8NtzmIKy+cfz00nViqcTF4bREXEawFGhlpO0X O9q8mijI9iFB6zaPBiSdJGBL5ML5qLTNCl8Zlf4m1TBvmRTqF/lzMHVXHidDoUhpSh/y3AFZ 1KrYc27ztJQywDJPJPWPbtY8YhFLFs377gfP8WldsZjzp8nvoQARAQABzSVDb2xpbiBQZXJj aXZhbCA8Y3BlcmNpdmFARnJlZUJTRC5vcmc+wsGRBBMBCAA7FiEEglY7hNBiDtwN+4ZBOJfy 4i5lrT8FAmWMSrYCGwMICwkNCAwHCwMFFQoJCAsFFgMCAQACHgUCF4AACgkQOJfy4i5lrT++ ig/9GZKdN2fHSyrANKZX38ivd7IX2wAYouqH9DrQM94W8IciaDLmarN4Pl9mY+aucMwQUSyp uNtKOJwKqhVVaalF9Zw0sRMH4CJuvT7vKCtZ3q1Okb7soRvFte4d+vXhvPxCvBFDA5JzU7Lg DR5eqqcvF1dN1OuCq16pl0zCOSH/Jr5ToE3LM3Av1KBGcZD7ZSzHRWsFjV5AOUJKySuA3GwJ e/jASQcQ0YfCnru8ntLmYg/2SKvZFlfthZiCBnAppMt4n4BUAw3TDvf10HIDtdneejawcbLS gofLCvGqumwbZYAMKWrFzT4+7KQvr0pOw8QD7EbxnB4f9hQ7UiVF8qWsyKU3iv6b5JLhbS59 ooKRccyOvdMLcVJ0ZdpqoxrNv061ZUqLL5RiWjBlc1qjBnDxeg5oyM0rT8WLftdgvyH6RQt0 KWngumBAT5AT2DUYL8Uz1490cqfO9K4yEGZAJB9XRVX1g2IWTOjae+0g9ZII+h91UngFz+Rz aKDeseKBbCGDOFXx1TqKiHl2g255ZnUxKYTlucFtguv4gDGBgEk4G9JaEWBw1IWblcKhxH7L 2vWsUhvwghjIxHdO/RkeIeHvSp4YZxCJ7a3TaJLYAlwYopfTKVzNhcDY5h5syEuoHjyJCxXK SyoJYAVu8Yl2KUhvOtOmL1VZ6xyHnpdMRWKJZ5jOwU0EZYxKtgEQANYfgbtUMVnhjxDHhWLp g5kLHK3YW0TfJKzpXqDB7NiqxHofn4OcbZnVC3MKggcbs9o1/UtsjnlsG8550PfiYkDXvPiO RJwgbGs6MGIDK797C6cnBLQ8xwBa9SL4cl5iQFnhWmt6vwnJ+an/cm5JpYves3wL7jV09qU9 57hkHXEUcl38r4FssZzVcLKPUVTa3Un+QGRTGDGe/f4ctjMaqv0ZCM+l2ixPhf/vqESrfSLv V/+T3dmtUfXjazO3SABvsHwxgGuTTYOlKoPCaebr+BRdqm0xeIShoIlhvTI8y4clchqx/Uxg UG5X2kvU13k3DS3Q8uLE4Et9x1CcZT6WGgBZSR6R0WfD0SDnzufNnRWJ0dEPA2MtJHE7+85R Vi9j/IgZV+y5Ur+bnPkjDG1s2SVciX5v9HQ0oilcBhvx0j5lGE9hhurD9F+fCvkr4KdbCknE 6Y8ce8pCNBUoB/DqibJivOzTk9K9MGB5x0De5TerIrFiaw3/mQC9nGeO9dtE7wvDJetWeoTq 4BEaCzpufNqbkpOaTQILr4V6Gp7M6v97g83TVAwZntz/q8ptwuKQPZ2JaSFLZn7oWUpYXA5s +SIODFHLn6iMoYpBQskHQjnj4lEPJadl4qj+ZKA89iDAKsniyoFXsbJe2CPbMS1yzBxKZq6K D/jpt7BOnuHr/JrXABEBAAHCwXYEGAEIACAWIQSCVjuE0GIO3A37hkE4l/LiLmWtPwUCZYxK tgIbDAAKCRA4l/LiLmWtP3jmEACQrh9gWe8F1Tkw3m6VoHKwLc5he4tX3WpQa//soPO6iGG3 S3WPruQ46NrAaAojoOcKI9UONDO5rxG0ZTX53S+lu2EO47jbcLwOCjaEpjKpDRt9ZXBQE8Xl mtBE9Bp3W9gpjB1nE3KNM1mJYgsK0QdRpwwfh4pVgGpOj8j23I6MCK+v99zEBnpgCn2GX8W/ kctRXHqWwndHysOJtRP/zrl7dDaABF1f9efUl0LL3TD3GJ9VDz+DNOin/uK2a1hiJo8QzTRk PpfUQ2ebzDsrd1i/pOWkMSkdH+rEu4AGrXWtaBwrMyrGkL6Icb6yO+P9/z0W2wlgBf3P1YRt JPgQt/Dj3yvA/UnaV/QmuVQPjl13o24UnJGsZM8XGnNdfWBKkC1Q6VXC4QT+dyBHYH9MuE9d 6oGl8pFM1+cTfEfbM62/rRoPkF1yHMsI/903VxEvuUIKfhEZAVLFyHldooNxuchntHQP9y8J 8Ou9bWYQP7MnEn+kwSwrZkjurfPkan+xQvp6dDYnj3V0GwA5pprBMaB928VIDVOv+1PNQI3t Cvk5VPv/skq+TJRMHW7bFSt8PRa91cUf1FOLIz9APDiJOzXkwxUEHGV3zPSaUhs1JYjyBeGT wDAvtLUdjOnRhEUOwlnIrztmvyciutjJoVzKEEjj5WXnHk9L9kQ1bpAjkjTONw== In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:14618, ipnet:54.86.0.0/16, country:US] X-Rspamd-Queue-Id: 4WNK096MDrz49MZ On 7/15/24 15:59, Warner Losh wrote: > On Mon, Jul 15, 2024 at 4:20 PM Colin Percival > wrote: > > On 7/15/24 15:05, Warner Losh wrote: > > Ideally, we'd keep the same KBI per major release, but that ideal has > fallen > > short. > > Having a stable KBI only solves half of the problem. With DRM especially it's > useful for people running the latest minor release to use kmods which make use > of functionality which was added in the latest release before the previous > release is EoLed. > > Functionality added where? I'm not following the point you're making here... > But since > KBI stability isn't going to happen (at least given our past track record and > a lack of > tools to enforce it), it may be just an academic exercise. New functions in LinuxKPI is the case I was thinking of. It doesn't prevent kernel modules compiled for earlier releases working with the newer release, but because we build kernel modules on a per-stable-branch basis we have to wait until the previous release EoLs. -- Colin Percival FreeBSD Release Engineering Lead & EC2 platform maintainer Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid From nobody Tue Jul 16 04:08:44 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WNQYM3Hlgz5QvMX for ; Tue, 16 Jul 2024 04:08:55 +0000 (UTC) (envelope-from eborisch@gmail.com) Received: from mail-qv1-xf32.google.com (mail-qv1-xf32.google.com [IPv6:2607:f8b0:4864:20::f32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WNQYL04gDz4bTN; Tue, 16 Jul 2024 04:08:54 +0000 (UTC) (envelope-from eborisch@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=DZ5Hzqvg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of eborisch@gmail.com designates 2607:f8b0:4864:20::f32 as permitted sender) smtp.mailfrom=eborisch@gmail.com Received: by mail-qv1-xf32.google.com with SMTP id 6a1803df08f44-6b5e4466931so27567586d6.0; Mon, 15 Jul 2024 21:08:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721102931; x=1721707731; darn=freebsd.org; h=cc:to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=Hu/h+y+p+l9ZHejB8i0XnxcNzOWNrin2+nZZ02IJ6Sc=; b=DZ5Hzqvg6DiLxAKk3LvwM9B+TxgUH4bTInfPwPPaZZdHVKlq5QjMYCRdXiQkP0puGh n8Jog261rcbD+J44SjskdUrObCGI0L5scXpVTuzBhNu4vwM78+ck1tOIeAJM0BKLP79K fLnzDORweT3rnhfgGoSIyuOcLTP2sbQMUVFwWatyEF9Orv9iWzxlKiN4av3oaLbw5aF7 sF8wOFWPqFZuZXI5hMPt7IXUUxN7XOqAP08A8/HocLPzmvpmPOurAk/tlR0A5NLLUP/D q5v2fdQ8fGbaSxF70UeamZWenqfPTuZbLxLjGSE9fgHaZfOMbsrPxUJkO0YzIn0DlEW8 oreg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721102931; x=1721707731; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Hu/h+y+p+l9ZHejB8i0XnxcNzOWNrin2+nZZ02IJ6Sc=; b=Vy/WUg++kRdLKbZJHjavGY+ZF2e5uX+R9h5lhKBYwlZb0rvEReZmlh1RLGk7bxUb12 d6XWR53gU2GM3qFrr/i4K9b1OkU9uOD+3z2sJARAvAUBSxeuf8ZwXX9kpORVtp6GObtN kOKCH8s+jttqz7OATmG/tH5Dv4kugaM19KLlhbBxPLSfzFYD+C8hSypE6vIx4IxE5MBK RAmUDCe5n3RlzATX/p1fBRsetICYD4n9Qfu7HuADvUYFloEWAP39v0kpRZtgX+kGG+rH j4xFSTdAC30tbBkHxmjgQusz7SFhmHFegDSELZBXfH34ZC3oYzF1mikoMGHLu1s1G1Ww 985A== X-Gm-Message-State: AOJu0YzNjVYvdvH7TDD/bZRXk8Vg0n8snK2gU1AR0O6iobYJcfuTxvjU MKdSdjpZ2TZAqWr6lBW3/g7VD3Tj068fboGZB/3ZGMC2+MTLXhHCLBnpCYBb1HV36HRpGIUHFRk c/Ba4kJGWwT3NRXZIdCtulf7gCiOezHWY X-Google-Smtp-Source: AGHT+IGwxVdcxN39umpm86a/4+S3k0TdGZCQTjs529oOpcMq9fwSDYq9ydonoEry33NTWZl3ncjaAFr3taanUjM0t7I= X-Received: by 2002:a05:6214:c44:b0:6b0:774b:38c2 with SMTP id 6a1803df08f44-6b77f3844cbmr15270266d6.6.1721102931398; Mon, 15 Jul 2024 21:08:51 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 From: "Eric A. Borisch" Date: Mon, 15 Jul 2024 23:08:44 -0500 Message-ID: Subject: Re: Possible bug in zfs send or pipe implementation? To: Mark Johnston Cc: FreeBSD Stable , rick.macklem@gmail.com, Garrett Wollman Content-Type: multipart/alternative; boundary="000000000000d91af5061d558015" X-Spamd-Bar: - X-Spamd-Result: default: False [-1.99 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; FAKE_REPLY(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.992]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[FreeBSD]; RCVD_TLS_LAST(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; FREEMAIL_CC(0.00)[freebsd.org,gmail.com,bimajority.org]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_RCPT(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::f32:from]; RCVD_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com] X-Rspamd-Queue-Id: 4WNQYL04gDz4bTN --000000000000d91af5061d558015 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Jul 15, 2024 at 9:50=E2=80=AFAM Mark Johnston w= rote: > On Sun, Jul 14, 2024 at 03:14:44PM -0700, Rick Macklem wrote: > > On Sun, Jul 14, 2024 at 10:32=E2=80=AFAM Garrett Wollman > > > wrote: > > > > > < rick.macklem@gmail.com> > > > said: > > > > > > > Just to clarify it, are you saying zfs is sleeping on "pipewr"? > > > > (There is also a msleep() for "pipbww" in pipe_write().) > > > > > > It is sleeping on pipewr, yes. > > > > > > [wollman@nfs-prod-11 ~]$ sysctl kern.ipc.pipekva > > > kern.ipc.pipekva: 536576 > > > [wollman@nfs-prod-11 ~]$ sysctl kern.ipc.maxpipekva > > > kern.ipc.maxpipekva: 2144993280 > > > > > > It's not out of KVA, it's just waiting for the `pv` process to wake u= p > > > and read more data. `pv` is single-threaded and blocked on "select". > > > > > > It doesn't always get stuck in the same place, which is why I'm > > > suspecting a lost wakeup somewhere. > > > > > This snippet from sys/kern/sys_pipe.c looks a little suspicious to me..= . > > /* > > * Direct copy, bypassing a kernel buffer. > > */ > > } else if ((size =3D rpipe->pipe_pages.cnt) !=3D 0) { > > if (size > uio->uio_resid) > > size =3D (u_int) uio->uio_resid; > > PIPE_UNLOCK(rpipe); > > error =3D uiomove_fromphys(rpipe->pipe_pages.ms, > > rpipe->pipe_pages.pos, size, uio); > > PIPE_LOCK(rpipe); > > if (error) > > break; > > nread +=3D size; > > rpipe->pipe_pages.pos +=3D size; > > rpipe->pipe_pages.cnt -=3D size; > > if (rpipe->pipe_pages.cnt =3D=3D 0) { > > rpipe->pipe_state &=3D ~PIPE_WANTW; > > wakeup(rpipe); > > } > > If it reads uio_resid bytes which is less than pipe_pages.cnt, no > > wakeup() occurs. > > I'd be tempted to try getting rid of the "if (rpipe->pipe_pages.cnt =3D= =3D > 0)" > > and do the wakeup() unconditionally, to see if it helps? > > I don't think that can help. pipe_write() will block if the "direct > write" buffer is non-empty. See the comment in pipe_write(), "Pipe > buffered writes cannot be coincidental with direct writes". > > select()/poll()/etc. should return an event if pipe_pages.cnt > 0 on the > read side, so I suspect that the problem is elsewhere, or else I'm > misunderstanding something. > > > Because if the application ("pv" in this case) doesn't do another read(= ) > on > > the > > pipe before calling select(), no wakeup() is going to occur, because > here's > > what pipe_write() does... > > /* > > * We have no more space and have something to offer, > > * wake up select/poll. > > */ > > pipeselwakeup(wpipe); > > > > wpipe->pipe_state |=3D PIPE_WANTW; > > pipeunlock(wpipe); > > error =3D msleep(wpipe, PIPE_MTX(rpipe), > > PRIBIO | PCATCH, "pipewr", 0); > > pipelock(wpipe, 0); > > if (error !=3D 0) > > break; > > continue; > > Note that, once in msleep(), no call to pipeselwakeup() will occur unti= l > > it gets woken up. > > > > I think the current code assumes that the reader ("pv" in this case) wi= ll > > read all the data out of the pipe before calling select() again. > > Does it do that? > > > > rick > > ps: I've added markj@ as a cc, since he seems to have been the last guy > > involved > > in sys_pipe.c. > I was running into this (in a repeatable fashion) today with pv in a ZFS send/recv pipeline. I was able to recompile pv with debugging, and look at what is happening within pv -- it is setting its internal EOF flag for the output pipe. This is happening when it has a write(2) call return 0 (not _technically_ an error?) at https://codeberg.org/a-j-wood/pv/src/commit/42b623e9b078a6799e081587204a54f= 41e26da37/src/pv/transfer.c#L209 . Once this happens during a __transfer_write_repeated() call when nothing (within the function) has been written yet, it sets *eof_out =3D true at li= ne 711, and _never_ tries to write again. It still keeps select()-ing (I haven't run down why it's not aborting at this point if it really thinks it has an output it can't write to) like mad, however. Editing the pv source to instead handle a 0 return from write(2) similar to EINTR and EAGAIN _makes it work without a hiccup_ on the same send|pv|recv that was previously freezing up every time in a matter of seconds. (In this particular case, it's an initial receive of a tree with a number of filesystems & snapshots). So _if_ write(2) should ever return 0 is a question I'll leave to the kernel developers here, but that appears to be what is tripping up pv here, at least for me. I've submitted a pull request (with a reference back to this thread) at https://codeberg.org/a-j-wood/pv/pulls/92 I've been unable to force this to happen with synthetic workloads; I don't know if there is something in the zfs recv code that is setting / interacting with the pipe in a particular way to expose this behavior. - Eric --000000000000d91af5061d558015 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Mon, Jul 15, 2024 at 9:50=E2=80=AFAM M= ark Johnston <mar= kj@freebsd.org> wrote:

On Sun, Jul 14, 2024 = at 03:14:44PM -0700, Rick Macklem wrote:
> On Sun, Jul 14, 2024 at 10:32=E2=80=AFAM Garrett Wollman <wollman@bimajority.org= >
> wrote:
>
> > <<On Sat, 13 Jul 2024 20:50:55 -0700, Rick Macklem <rick.macklem@gmail.= com>
> > said:
> >
> > > Just to clarify it, are you saying zfs is sleeping on "= pipewr"?
> > > (There is also a msleep() for "pipbww" in pipe_wri= te().)
> >
> > It is sleeping on pipewr, yes.
> >
> > [wollman@nfs-prod-11 ~]$ sysctl kern.ipc.pipekva
> > kern.ipc.pipekva: 536576
> > [wollman@nfs-prod-11 ~]$ sysctl kern.ipc.maxpipekva
> > kern.ipc.maxpipekva: 2144993280
> >
> > It's not out of KVA, it's just waiting for the `pv` proce= ss to wake up
> > and read more data.=C2=A0 `pv` is single-threaded and blocked on = "select".
> >
> > It doesn't always get stuck in the same place, which is why I= 'm
> > suspecting a lost wakeup somewhere.
> >
> This snippet from sys/kern/sys_pipe.c looks a little suspicious to me.= ..
>=C2=A0 =C2=A0/*
> * Direct copy, bypassing a kernel buffer.
> */
> } else if ((size =3D rpipe->pipe_pages.cnt) !=3D 0) {
> if (size > uio->uio_resid)
> size =3D (u_int) uio->uio_resid;
> PIPE_UNLOCK(rpipe);
> error =3D uiomove_fromphys(rpipe->pipe_pages.ms,
>=C2=A0 =C2=A0 =C2=A0rpipe->pipe_pages.pos, size, uio);
> PIPE_LOCK(rpipe);
> if (error)
> break;
> nread +=3D size;
> rpipe->pipe_pages.pos +=3D size;
> rpipe->pipe_pages.cnt -=3D size;
> if (rpipe->pipe_pages.cnt =3D=3D 0) {
> rpipe->pipe_state &=3D ~PIPE_WANTW;
> wakeup(rpipe);
> }
> If it reads uio_resid bytes which is less than pipe_pages.cnt, no
> wakeup() occurs.
> I'd be tempted to try getting rid of the "if (rpipe->pipe_= pages.cnt =3D=3D 0)"
> and do the wakeup() unconditionally, to see if it helps?

I don't think that can help.=C2=A0 pipe_write() will block if the "= ;direct
write" buffer is non-empty.=C2=A0 See the comment in pipe_write(), &qu= ot;Pipe
buffered writes cannot be coincidental with direct writes".

select()/poll()/etc. should return an event if pipe_pages.cnt > 0 on the=
read side, so I suspect that the problem is elsewhere, or else I'm
misunderstanding something.

> Because if the application ("pv" in this case) doesn't d= o another read() on
> the
> pipe before calling select(), no wakeup() is going to occur, because h= ere's
> what pipe_write() does...
> /*
> * We have no more space and have something to offer,
> * wake up select/poll.
> */
> pipeselwakeup(wpipe);
>
> wpipe->pipe_state |=3D PIPE_WANTW;
> pipeunlock(wpipe);
> error =3D msleep(wpipe, PIPE_MTX(rpipe),
>=C2=A0 =C2=A0 =C2=A0PRIBIO | PCATCH, "pipewr", 0);
> pipelock(wpipe, 0);
> if (error !=3D 0)
> break;
> continue;
> Note that, once in msleep(), no call to pipeselwakeup() will occur unt= il
> it gets woken up.
>
> I think the current code assumes that the reader ("pv" in th= is case) will
> read all the data out of the pipe before calling select() again.
> Does it do that?
>
> rick
> ps: I've added markj@ as a cc, since he seems to have been the las= t guy
> involved
>=C2=A0 =C2=A0 =C2=A0in sys_pipe.c.

= I was running into this (in a repeatable fashion) today with pv in a ZFS se= nd/recv pipeline. I was able to recompile pv with debugging, and look at wh= at is happening within pv -- it is setting its internal EOF flag for the ou= tput pipe.

This is happening when it has a write(2= ) call return 0 (not _technically_ an error?) at https://codeberg.org/a-j-wood/pv/src/commit/42b623e9b078= a6799e081587204a54f41e26da37/src/pv/transfer.c#L209.=C2=A0

Once this happens during a=C2=A0__transfer_write_repeated() ca= ll when nothing (within the function) has been written yet, it sets *eof_ou= t =3D true at line 711, and _never_ tries to write again. It still keeps se= lect()-ing (I haven't run down why it's not aborting at this point = if it really thinks it has an output it can't write to) like mad, howev= er.

Editing the pv source to instead handle a = 0 return from write(2) similar to EINTR and EAGAIN _makes it work without a= hiccup_ on the same send|pv|recv that was previously freezing up every tim= e in a matter of seconds. (In this particular case, it's an initial rec= eive of a tree with a number of filesystems & snapshots).

So _if_ write(2) should ever return 0 is a question I'll le= ave to the kernel developers here, but that appears to be what is tripping = up pv here, at least for me.

I've submitted a = pull request (with a reference back to this thread) at https://codeberg.org/a-j-wood/pv/pulls/92=

I've been unable to force this to happen = with synthetic workloads; I don't know if there is something in the zfs= recv code that is setting / interacting with the pipe in a particular way = to expose this behavior.

=C2=A0 - Eric

=C2=A0

--000000000000d91af5061d558015-- From nobody Tue Jul 16 11:27:00 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WNcHX0wDhz5R7bP for ; Tue, 16 Jul 2024 11:27:36 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Received: from www121.sakura.ne.jp (www121.sakura.ne.jp [153.125.133.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WNcHW2Nr2z42dB; Tue, 16 Jul 2024 11:27:34 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Authentication-Results: mx1.freebsd.org; none Received: from kalamity.joker.local (123-1-21-232.area1b.commufa.jp [123.1.21.232]) (authenticated bits=0) by www121.sakura.ne.jp (8.17.1/8.17.1/[SAKURA-WEB]/20201212) with ESMTPA id 46GBR0Q8074366; Tue, 16 Jul 2024 20:27:01 +0900 (JST) (envelope-from junchoon@dec.sakura.ne.jp) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=dec.sakura.ne.jp; s=s2405; t=1721129222; bh=kX8970US2lFY7Za2SL5RACLxPTeLdvZIa/6H2f2ThvY=; h=Date:From:To:Cc:Subject:In-Reply-To:References; b=R64GYOn+S1Sf/f8F6mJGt1qsC7w0Z4G+Y1mTsO1pG6mr4xkqKoil9GlTmIW/B/K66 Q2ZAhC439jkAB1xyRqM0TN++k3tkLRFl/y6UnmRoCAXb27pFGY+VU3/CpcXURSe6xQ 6vJLH9SYuCeO8HVZiKzO9wzX0sqrS4voUW3dfz2U= Date: Tue, 16 Jul 2024 20:27:00 +0900 From: Tomoaki AOKI To: Colin Percival Cc: Warner Losh , Peter , freebsd-stable@freebsd.org, Baptiste Daroussin Subject: Re: Change to FreeBSD release scheduling etc. Message-Id: <20240716202700.548354a58e17831bee7ab449@dec.sakura.ne.jp> In-Reply-To: <25a443cb-35d3-4d32-bb94-7aa18d4a3813@freebsd.org> References: <3598c6db-6d8a-4efa-b5fc-1fc697608860@freebsd.org> <20240716063356.f0993ae6f7648ae19c1ae33f@dec.sakura.ne.jp> <0fb67778-d279-420b-b482-5ce36e08a1be@freebsd.org> <22612d5f-3960-4390-a484-3bd307820126@freebsd.org> <25a443cb-35d3-4d32-bb94-7aa18d4a3813@freebsd.org> Organization: Junchoon corps X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd14.1) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:7684, ipnet:153.125.128.0/18, country:JP] X-Rspamd-Queue-Id: 4WNcHW2Nr2z42dB On Mon, 15 Jul 2024 16:58:16 -0700 Colin Percival wrote: > On 7/15/24 15:59, Warner Losh wrote: > > On Mon, Jul 15, 2024 at 4:20 PM Colin Percival > > wrote: > > > > On 7/15/24 15:05, Warner Losh wrote: > > > Ideally, we'd keep the same KBI per major release, but that ideal has > > fallen > > > short. > > > > Having a stable KBI only solves half of the problem. With DRM especially it's > > useful for people running the latest minor release to use kmods which make use > > of functionality which was added in the latest release before the previous > > release is EoLed. > > > > Functionality added where? I'm not following the point you're making here... > > But since > > KBI stability isn't going to happen (at least given our past track record and > > a lack of > > tools to enforce it), it may be just an academic exercise. > > New functions in LinuxKPI is the case I was thinking of. It doesn't prevent > kernel modules compiled for earlier releases working with the newer release, > but because we build kernel modules on a per-stable-branch basis we have to > wait until the previous release EoLs. It would be the one most frequently happening recently. For environments where updating/upgrading via freebsd-update are not provided (like main and stable branches), admins are basically requited to update/upgrade base via src. These cases, they can put kmod ports into PORTS_MODULES variable in /etc/src.conf, thus basically no problem. Unfortunately, this WAS not enough true for drm-*-kmod ports as they often took behind and "broken time window" persisted until they catch up with base. But fortunately, at least for stable/14, it seems to be significantly stabilized thanks to the tireless efforts of graphics team. So, remaining problem would be the screams/sighs on forums.freebsd.org when any binary kernel updates (patch releases) are provided via freebsd-update, and/or when someone attempted to upgrade their base to non-*.0 releases on different stable branch (i.e., upgrade from 13.2 to 14.1) while previous point release is still not EOL'ed. As admins should be strongly encouraged to run latest patch (-p*) release of the point release they are using, kmod pkgs should be provided for latest patch release only on single point releases. (Means, when 14.0 is at 14.0-p3 and 14.1 is at 14.1-p1, and 14.0 is not yet EOL'ed, kmod pkgs for 14.0-p3 and 14.1-p1 should be provided.) > > -- > Colin Percival > FreeBSD Release Engineering Lead & EC2 platform maintainer > Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid -- Tomoaki AOKI From nobody Tue Jul 16 13:36:19 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WNg885JP7z5RKSg for ; Tue, 16 Jul 2024 13:36:24 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x733.google.com (mail-qk1-x733.google.com [IPv6:2607:f8b0:4864:20::733]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WNg8753Jrz4Flb for ; Tue, 16 Jul 2024 13:36:23 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=AC1fVOkq; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::733 as permitted sender) smtp.mailfrom=markjdb@gmail.com Received: by mail-qk1-x733.google.com with SMTP id af79cd13be357-7a05b7c8e38so341435385a.0 for ; Tue, 16 Jul 2024 06:36:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721136983; x=1721741783; darn=freebsd.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:sender :from:to:cc:subject:date:message-id:reply-to; bh=8UpnYcQFA8a4ybjgNq28DwO6WAXT8zMaPaPjBBdk4pU=; b=AC1fVOkqJQRZNgQyy96XkFJ1zaY6C/KEKOLD7KIFpYL6wvRVq0GTZL8QxXNZBOBBvz 4x0zqRvnXca2yUIC0xZV4Fj5sG8V7/i0LelD4EN9q5f9mj+LkbZAxOjZm79wE8bczB+4 AnQW1ZNaEX9TiNEcdC90ETqDYPucpsEd2LbVxKrWgZtoFpzF02cqnVz8ylNjG0QYntHG uAnS2NASBTLdNQyuZhfMvAIjjoRXLHhSoSDZ94LO3G5MhENFA+jR2wQVtjEUWLs/rJPB IVFVBEsTzDq0rYXidOIrAAoz4OPKwGbmuTIkUoxsKLyMZq0XvYfgp6Zr17i5F4AHSI9l 0ArA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721136983; x=1721741783; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8UpnYcQFA8a4ybjgNq28DwO6WAXT8zMaPaPjBBdk4pU=; b=AHa9hwgYp7VzPMXL7qoDIut0NCQ/SPZZQ86jZyH7Fy9lRk5aA6pWhY4m3S0Y/6Jw9S D8hbOSFtnOYxpT3+W0RjTQDO5t2ZCz0+w7TP5laeNv+McjGQnyDC+WOlB0+fiBhd9AfG wd0sggnsfanO0x9QS8Hs4nGFPjhXqJl3mi6rsPRgu3lp3dMUFj72WqFvvFNrg+cxdmZE DJ6HRTOQGRaNx9xp8V+NmCT+pC+DTym88aYPEo6tyqLGyZGwYeZULQGLth0yGED5EXwD LrfY0lVu3gX7ZjT5JNiFjQKX0VHz7TnWr/EVTlQ7is3rBECVKbAym6fTqMnXQV7SDbfN mlrA== X-Gm-Message-State: AOJu0YwkR3asSdtwQ/64jQuPiXto7fk08f+vaKCQRv/cCKNoqayh+4mF +QFiGXUrl8fYcslNyg4yfb4jltkiM60BNcv128n+LE4QIlvk7Voa X-Google-Smtp-Source: AGHT+IEkoTyB//qLoMHd0RhPpCPBKwZqSEa0ryAgH8S+eGEKFxkWFYkixIIOb337i+BjAXJVxx9bcQ== X-Received: by 2002:a05:620a:191f:b0:79f:515:6097 with SMTP id af79cd13be357-7a17b678709mr293528385a.26.1721136982636; Tue, 16 Jul 2024 06:36:22 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a160bbea20sm292537885a.34.2024.07.16.06.36.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 06:36:21 -0700 (PDT) Date: Tue, 16 Jul 2024 09:36:19 -0400 From: Mark Johnston To: "Eric A. Borisch" Cc: FreeBSD Stable , rick.macklem@gmail.com, Garrett Wollman Subject: Re: Possible bug in zfs send or pipe implementation? Message-ID: References: List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spamd-Bar: / X-Spamd-Result: default: False [-0.06 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.961]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), DKIM not aligned (relaxed),none]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; FREEMAIL_CC(0.00)[freebsd.org,gmail.com,bimajority.org]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::733:from]; TAGGED_RCPT(0.00)[FreeBSD]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4WNg8753Jrz4Flb On Mon, Jul 15, 2024 at 11:08:44PM -0500, Eric A. Borisch wrote: > On Mon, Jul 15, 2024 at 9:50 AM Mark Johnston wrote: > > > On Sun, Jul 14, 2024 at 03:14:44PM -0700, Rick Macklem wrote: > > > On Sun, Jul 14, 2024 at 10:32 AM Garrett Wollman > > > > > wrote: > > > > > > > < > rick.macklem@gmail.com> > > > > said: > > > > > > > > > Just to clarify it, are you saying zfs is sleeping on "pipewr"? > > > > > (There is also a msleep() for "pipbww" in pipe_write().) > > > > > > > > It is sleeping on pipewr, yes. > > > > > > > > [wollman@nfs-prod-11 ~]$ sysctl kern.ipc.pipekva > > > > kern.ipc.pipekva: 536576 > > > > [wollman@nfs-prod-11 ~]$ sysctl kern.ipc.maxpipekva > > > > kern.ipc.maxpipekva: 2144993280 > > > > > > > > It's not out of KVA, it's just waiting for the `pv` process to wake up > > > > and read more data. `pv` is single-threaded and blocked on "select". > > > > > > > > It doesn't always get stuck in the same place, which is why I'm > > > > suspecting a lost wakeup somewhere. > > > > > > > This snippet from sys/kern/sys_pipe.c looks a little suspicious to me... > > > /* > > > * Direct copy, bypassing a kernel buffer. > > > */ > > > } else if ((size = rpipe->pipe_pages.cnt) != 0) { > > > if (size > uio->uio_resid) > > > size = (u_int) uio->uio_resid; > > > PIPE_UNLOCK(rpipe); > > > error = uiomove_fromphys(rpipe->pipe_pages.ms, > > > rpipe->pipe_pages.pos, size, uio); > > > PIPE_LOCK(rpipe); > > > if (error) > > > break; > > > nread += size; > > > rpipe->pipe_pages.pos += size; > > > rpipe->pipe_pages.cnt -= size; > > > if (rpipe->pipe_pages.cnt == 0) { > > > rpipe->pipe_state &= ~PIPE_WANTW; > > > wakeup(rpipe); > > > } > > > If it reads uio_resid bytes which is less than pipe_pages.cnt, no > > > wakeup() occurs. > > > I'd be tempted to try getting rid of the "if (rpipe->pipe_pages.cnt == > > 0)" > > > and do the wakeup() unconditionally, to see if it helps? > > > > I don't think that can help. pipe_write() will block if the "direct > > write" buffer is non-empty. See the comment in pipe_write(), "Pipe > > buffered writes cannot be coincidental with direct writes". > > > > select()/poll()/etc. should return an event if pipe_pages.cnt > 0 on the > > read side, so I suspect that the problem is elsewhere, or else I'm > > misunderstanding something. > > > > > Because if the application ("pv" in this case) doesn't do another read() > > on > > > the > > > pipe before calling select(), no wakeup() is going to occur, because > > here's > > > what pipe_write() does... > > > /* > > > * We have no more space and have something to offer, > > > * wake up select/poll. > > > */ > > > pipeselwakeup(wpipe); > > > > > > wpipe->pipe_state |= PIPE_WANTW; > > > pipeunlock(wpipe); > > > error = msleep(wpipe, PIPE_MTX(rpipe), > > > PRIBIO | PCATCH, "pipewr", 0); > > > pipelock(wpipe, 0); > > > if (error != 0) > > > break; > > > continue; > > > Note that, once in msleep(), no call to pipeselwakeup() will occur until > > > it gets woken up. > > > > > > I think the current code assumes that the reader ("pv" in this case) will > > > read all the data out of the pipe before calling select() again. > > > Does it do that? > > > > > > rick > > > ps: I've added markj@ as a cc, since he seems to have been the last guy > > > involved > > > in sys_pipe.c. > > > > I was running into this (in a repeatable fashion) today with pv in a ZFS > send/recv pipeline. I was able to recompile pv with debugging, and look at > what is happening within pv -- it is setting its internal EOF flag for the > output pipe. > > This is happening when it has a write(2) call return 0 (not _technically_ > an error?) at > https://codeberg.org/a-j-wood/pv/src/commit/42b623e9b078a6799e081587204a54f41e26da37/src/pv/transfer.c#L209 > . > > Once this happens during a __transfer_write_repeated() call when nothing > (within the function) has been written yet, it sets *eof_out = true at line > 711, and _never_ tries to write again. It still keeps select()-ing (I > haven't run down why it's not aborting at this point if it really thinks it > has an output it can't write to) like mad, however. > > Editing the pv source to instead handle a 0 return from write(2) similar to > EINTR and EAGAIN _makes it work without a hiccup_ on the same send|pv|recv > that was previously freezing up every time in a matter of seconds. (In this > particular case, it's an initial receive of a tree with a number of > filesystems & snapshots). > > So _if_ write(2) should ever return 0 is a question I'll leave to the > kernel developers here, but that appears to be what is tripping up pv here, > at least for me. Looking at the sources, it seems that this is a blocking file descriptor, i.e., O_NONBLOCK is not set. Is that correct? If so, the kernel's behaviour seems wrong to me. It would be useful to see a ktrace of the process. > I've submitted a pull request (with a reference back to this thread) at > https://codeberg.org/a-j-wood/pv/pulls/92 > > I've been unable to force this to happen with synthetic workloads; I don't > know if there is something in the zfs recv code that is setting / > interacting with the pipe in a particular way to expose this behavior. > > - Eric From nobody Tue Jul 16 15:21:31 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WNjTg1tRkz5RSvv for ; Tue, 16 Jul 2024 15:21:43 +0000 (UTC) (envelope-from ml@netfence.it) Received: from soth.netfence.it (mailserver.netfence.it [78.134.96.152]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mailserver.netfence.it", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WNjTf0N4Fz4SDd for ; Tue, 16 Jul 2024 15:21:41 +0000 (UTC) (envelope-from ml@netfence.it) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=netfence.it header.s=202404 header.b=laC2itAJ; dmarc=pass (policy=none) header.from=netfence.it; spf=pass (mx1.freebsd.org: domain of ml@netfence.it designates 78.134.96.152 as permitted sender) smtp.mailfrom=ml@netfence.it Received: from [10.1.2.18] (mailserver.netfence.it [78.134.96.152]) (authenticated bits=0) by soth.netfence.it (8.18.1/8.17.2) with ESMTPSA id 46GFLVK7057111 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Tue, 16 Jul 2024 17:21:32 +0200 (CEST) (envelope-from ml@netfence.it) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netfence.it; s=202404; t=1721143294; bh=6L0pq75LOXBhfEpSKRaM0sdbrVOJvPr8i4yR8DpL7Pc=; h=Date:Subject:To:References:From:In-Reply-To; b=laC2itAJwgIHUcusBbuKlVEo4wXavKcUq09GcySkhO69WfI/LuDYOZAyRYaaYl5zP Aw/8gcAub0CIzKlBzZFsaNy7mryat4jZd+0Ar98TbKPWybRJ58C7j3gsQBXx2SW+XD GTk3mx3lNyYfyC87VbD6OAq3lNpGlpHyark1vGTI= X-Authentication-Warning: soth.netfence.it: Host mailserver.netfence.it [78.134.96.152] claimed to be [10.1.2.18] Message-ID: Date: Tue, 16 Jul 2024 17:21:31 +0200 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Change to FreeBSD release scheduling etc. Content-Language: en-US To: stable@freebsd.org References: <3598c6db-6d8a-4efa-b5fc-1fc697608860@freebsd.org> From: Andrea Venturoli In-Reply-To: <3598c6db-6d8a-4efa-b5fc-1fc697608860@freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.86 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[netfence.it,none]; R_SPF_ALLOW(-0.20)[+ip4:78.134.96.152]; R_DKIM_ALLOW(-0.20)[netfence.it:s=202404]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:35612, ipnet:78.134.0.0/17, country:IT]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ONE(0.00)[1]; RCVD_TLS_ALL(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; HAS_XAW(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; DKIM_TRACE(0.00)[netfence.it:+] X-Rspamd-Queue-Id: 4WNjTf0N4Fz4SDd On 7/15/24 20:58, Colin Percival wrote: Hello. Again, as a foreword, please don't get me wrong, I appreciate your work and that of all the team. >> In short, it takes me about 3 months to catch the surprizes[*] and fix > > While I see your point, we're hoping that having roughly 2x as many minor > releases will result in at least a 2x reduction in the number of surprises > per minor release Unfortunately, that's not all. Upgrading means rebooting and, while that's something that should happen without troubles, it also means that if something goes wrong someone might need to take a 200km trip in a hurry. We usually do these "deep" upgrades when we are at the customer's site, but, for some customers, this might happen once/twice a year. > Extending the minor-release support period might be possible, but that > would depend on portmgr and secteam and I can't speak for them. One issue > which would certainly come up is kernel module packages -- our packages > are built for each stable branch on the oldest currently supported release, > which means that e.g. new features in 14.1 can't be used until 14.0 is EoL; > this is a problem particularly for graphics drivers. Since we've got our Poudriere system, I'm in no place to speak about this issue (although I understand it). bye & Thanks av. From nobody Tue Jul 16 18:49:50 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WNpCV0Pp0z5RmCH for ; Tue, 16 Jul 2024 18:54:46 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WNpCT4LSHz3xbr; Tue, 16 Jul 2024 18:54:45 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.18.1/8.18.1) with ESMTPS id 46GIsAm5038408 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 16 Jul 2024 20:54:10 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) ARC-Seal: i=1; a=rsa-sha256; d=uucp.dinoex.org; s=M20221114; t=1721156053; cv=none; b=A7NV3hSDTiPzXAo+FAkKXxwqSPte663S7fQ8mwq04iQnZT7riHElLBxxB8QJCXyuc33Fu4NlVeyOE4UBm1+n3uGXbT+6dF708IDxJjz3nNtiro069ozYeXTj5HahIxiAxw0iJZrsXn3BkOaf8zHaIT5Q2K3pIyZYzEVeAJODTeo= ARC-Message-Signature: i=1; a=rsa-sha256; d=uucp.dinoex.org; s=M20221114; t=1721156053; c=relaxed/simple; bh=eBkS9gawxJdhBZyuWZGk/D58ikQ5VC4Xn24n2VWAgG0=; h=Received:Received:Received:Received:X-Authentication-Warning:Date: From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type: Content-Disposition:In-Reply-To:X-Milter:X-Greylist; b=jcJ7ihTnSETVIvZz1Qt89nfGQ9GQ6uNuKtXOMBWqURaGyuQc5nwggKOWL7c/iM3h+kJviTBRSJN/vAgVrLgcTRRtIBEbFL8vtnU4xI6BU0PhT0GXIrCqLxQDUbI7EWv4R1LoO69gXe0vFrOqILaaSHDTO0We9KJsU0kj+rPLvdc= ARC-Authentication-Results: i=1; uucp.dinoex.org Received: (from uucp@localhost) by uucp.dinoex.org (8.18.1/8.18.1/Submit) with UUCP id 46GIsA8F038407; Tue, 16 Jul 2024 20:54:10 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.18.1/8.18.1) with ESMTPS id 46GInx2b054316 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Tue, 16 Jul 2024 20:49:59 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.18.1/8.18.1) with ESMTPS id 46GInoWm073634 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 16 Jul 2024 20:49:51 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.18.1/8.18.1/Submit) id 46GInojH073633; Tue, 16 Jul 2024 20:49:50 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Tue, 16 Jul 2024 20:49:50 +0200 From: Peter To: Colin Percival Cc: freebsd-stable@freebsd.org Subject: Re: Change to FreeBSD release scheduling etc. Message-ID: References: <3598c6db-6d8a-4efa-b5fc-1fc697608860@freebsd.org> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3598c6db-6d8a-4efa-b5fc-1fc697608860@freebsd.org> X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Tue, 16 Jul 2024 20:54:13 +0200 (CEST) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE] X-Rspamd-Queue-Id: 4WNpCT4LSHz3xbr Folks, thank You all for the feedback. As it seems. I'm not the only one concerned. On Mon, Jul 15, 2024 at 11:58:16AM -0700, Colin Percival wrote: > While I see your point, we're hoping that having roughly 2x as many minor > releases will result in at least a 2x reduction in the number of surprises > per minor release -- because not only is less code changing between minor > releases, but also committers feeling less pressure to hit a deadline may > result in code being better tested and less surprise-prone when it lands > in a minor release. That sounds nice, I just do not believe it: the surprizes which I happen to encounter, do not appear as having been created in a hurry of pressing release date. Also, the Agile/Devops/etc theorists teach to release often and with small increments. So what will most likely happen is just smaller increments, again in a hurry, to meet the release date. > Extending the minor-release support period might be possible, but that > would depend on portmgr and secteam and I can't speak for them. One issue > which would certainly come up is kernel module packages -- our packages > are built for each stable branch on the oldest currently supported release, > which means that e.g. new features in 14.1 can't be used until 14.0 is EoL; > this is a problem particularly for graphics drivers. It concerns secteam, certainly. Maybe you can speak /to/ them... ;) The ports issue seems rather a technical shortcoming insofar as kernel modules may need recompile for minor releases, while the pkg infrastructure has no notion of a minor release. This is a pain-point already: I remember frequent discussions in the forums whenever a new minor release appears and something with the graphics driver doesn't work as expected (I don't know the details, as I for my part do deploy from source.) An improvement with this might be desireable anyway and independent from the release schedule. regards, PMc From nobody Tue Jul 16 20:48:47 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WNrlF2fnXz5QCyv for ; Tue, 16 Jul 2024 20:48:57 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-40136.proton.ch (mail-40136.proton.ch [185.70.40.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WNrlD0L8Yz4GMj; Tue, 16 Jul 2024 20:48:54 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail; t=1721162932; x=1721422132; bh=nMOTmLoVTVCsca0F4+YkBnPL2FvkwQtPauJNhR3fBsA=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=Iy59MnABXLhPNnGapRN2SYofq53dBhGbcUYkO7QSZuidcmgTai/PORlbab/oHZOaO TWuN2OxLaTitdz/uDMRY+0HsHOOFr3k1DYERqq4oc7NdLFqdy2vZH8kTgRhw/PTZd7 aTJTl/EHhFSpC5nVbTirFopJMC9d9DVhd/OVMhcLaYTL+AgbVzBIDuzIVY6Wra5nWf ZcqSMscpqm23BifMqi7aQkmLVwIu27TUZG1RfgjLJkZlaCnnnTY2Rmuyzah+ezfuxe 82g4EGoTUd6EpueL9mlf2WguM+GChdgs9RFEyYy3wT5QRjEWOvtOKgeYuCejES3wjm LHzDEfmFIa+Fg== Date: Tue, 16 Jul 2024 20:48:47 +0000 To: "pmc@citylink.dinoex.sub.org" , "cperciva@freebsd.org" From: Jonathan Vasquez Cc: "freebsd-stable@freebsd.org" Subject: Re: Change to FreeBSD release scheduling etc. Message-ID: <_HuyEsRX_mI0SJbTeeDWkH1y8eteJAHXJlUg6w09I7J77Ln8sJJnE0e57FLH9d5_MkN080FVlfMiLyeZvV08R7bHuebiiScYsnOr8XywOmA=@xyinn.org> In-Reply-To: References: <3598c6db-6d8a-4efa-b5fc-1fc697608860@freebsd.org> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: 6f60817a4a6b5e57942690b36a8d6969a85d6aff List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:62371, ipnet:185.70.40.0/24, country:CH] X-Rspamd-Queue-Id: 4WNrlD0L8Yz4GMj I think the "rush to merge stuff into FreeBSD" concern that Colin was speak= ing about is valid, and it's actually the situation that the Linux kernel f= inds itself in all the time. They tend to have a relatively fast developmen= t cycle (2-3 months) because of this, and even with that people will still = want to merge stuff in because they don't want to wait until another 2-3 mo= nths. The Linux kernel does have a lot more activity than FreeBSD/kernel, s= o it's possible we may not need to put too much emphasis on a strategy to s= olve that particular point. Which also means developers can/should still wa= it for the next release. Below is a quote of the current release policy on kernel.org for thought: When is the next mainline kernel version going to be released? Linux kernel follows a simple release cadence: after each mainline release, there is a 2-week "merge window" period du= ring which new major features are introduced into the kernel after the merge window closes, there is a 7-week bugfix and stabilizati= on period with weekly "release candidate" snapshots rc7 is usually the last release candidate, though occasionally there ma= y be additional rc8+ releases if that is deemed necessary So, to find the approximate date of the next mainline kernel release, take = the date of the previous mainline release and add 9-10 weeks. https://www.kernel.org/category/releases.html Jonathan Vasquez PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 Sent with ProtonMail Secure Email -------- Original Message -------- On 7/16/24 14:49, Peter wrote: > Folks, > =20 > thank You all for the feedback. As it seems. I'm not the only one > concerned. > =20 > On Mon, Jul 15, 2024 at 11:58:16AM -0700, Colin Percival wrote: > =20 > > While I see your point, we're hoping that having roughly 2x as many mi= nor > > releases will result in at least a 2x reduction in the number of surpr= ises > > per minor release -- because not only is less code changing between mi= nor > > releases, but also committers feeling less pressure to hit a deadline = may > > result in code being better tested and less surprise-prone when it lan= ds > > in a minor release. > =20 > That sounds nice, I just do not believe it: the surprizes which I > happen to encounter, do not appear as having been created in a hurry of > pressing release date. > Also, the Agile/Devops/etc theorists teach to release often and with > small increments. So what will most likely happen is just smaller > increments, again in a hurry, to meet the release date. > =20 > > Extending the minor-release support period might be possible, but that > > would depend on portmgr and secteam and I can't speak for them. One i= ssue > > which would certainly come up is kernel module packages -- our package= s > > are built for each stable branch on the oldest currently supported rel= ease, > > which means that e.g. new features in 14.1 can't be used until 14.0 is= EoL; > > this is a problem particularly for graphics drivers. > =20 > It concerns secteam, certainly. Maybe you can speak /to/ them... ;) > =20 > The ports issue seems rather a technical shortcoming insofar as kernel > modules may need recompile for minor releases, while the pkg > infrastructure has no notion of a minor release. > This is a pain-point already: I remember frequent discussions in the > forums whenever a new minor release appears and something with the > graphics driver doesn't work as expected (I don't know the details, > as I for my part do deploy from source.) > An improvement with this might be desireable anyway and independent > from the release schedule. > =20 > =20 > regards, > PMc > =20 > From nobody Tue Jul 16 21:40:26 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WNsv84LbMz5QJ43 for ; Tue, 16 Jul 2024 21:40:52 +0000 (UTC) (envelope-from jason@tubnor.net) Received: from mail.tubnor.net (mail.tubnor.net [103.236.162.16]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WNsv654D7z4KyH for ; Tue, 16 Jul 2024 21:40:50 +0000 (UTC) (envelope-from jason@tubnor.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tubnor.net header.s=20220915 header.b=lKjqGhBT; dmarc=pass (policy=none) header.from=tubnor.net; spf=pass (mx1.freebsd.org: domain of jason@tubnor.net designates 103.236.162.16 as permitted sender) smtp.mailfrom=jason@tubnor.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tubnor.net; s=20220915; t=1721166038; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Adjx/0eIfYXGRqG/bVa0M/MmugNfP+CAP84CgiP+Ra8=; b=lKjqGhBTUhsiB0PBxABxH19sasfh7vrlNeCGbkSXwXxLIkcfvszj8ybLXQTZTlotqLV6e+ 9ELX9wFuLNj3AvGgi3lrnTSVIVamGWDesbgacRgkgu7+xMxNdYW7/CdyyaJyhTK8TkoV5I ef1ODNeI/YeR+MKLKFVdHN+hF9qxwFI= Received: from smtpclient.apple ( [2001:8004:44b0:1b09:c528:56e8:c3ea:9263]) by mel01.ar18.net (OpenSMTPD) with ESMTPSA id 2af21a72 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 17 Jul 2024 07:40:37 +1000 (AEST) Content-Type: multipart/alternative; boundary=Apple-Mail-E7688D7A-DC31-4F7B-96F1-10E612C87F8B Content-Transfer-Encoding: 7bit From: Jason Tubnor List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org Mime-Version: 1.0 (1.0) Subject: Re: Possible bug in zfs send or pipe implementation? Date: Wed, 17 Jul 2024 07:40:26 +1000 Message-Id: References: Cc: freebsd-stable@freebsd.org In-Reply-To: To: Garrett Wollman , Rick Macklem X-Mailer: iPhone Mail (21F90) X-Spamd-Bar: -- X-Spamd-Result: default: False [-3.00 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[tubnor.net,none]; R_DKIM_ALLOW(-0.20)[tubnor.net:s=20220915]; R_SPF_ALLOW(-0.20)[+ip4:103.236.162.16]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_ALL(0.00)[]; DKIM_TRACE(0.00)[tubnor.net:+]; FROM_HAS_DN(0.00)[]; FREEMAIL_TO(0.00)[bimajority.org,gmail.com]; RCPT_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; APPLE_IOS_MAILER_COMMON(0.00)[]; ASN(0.00)[asn:133159, ipnet:103.236.162.0/23, country:AU]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEFALL_USER(0.00)[jason]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_RCPT(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[] X-Rspamd-Queue-Id: 4WNsv654D7z4KyH --Apple-Mail-E7688D7A-DC31-4F7B-96F1-10E612C87F8B Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Confirmed. Removing PV from the pipeline has solved my issue. I was able to m= ove 20TB around in the last 18 hours without issue. Sent from my iPhone > On 15 Jul 2024, at 6:42=E2=80=AFPM, Jason Tubnor wrote:= >=20 > =EF=BB=BF >=20 >=20 > On 15/07/2024 9:35 am, Garrett Wollman wrote: >> I'm currently running syncoid with `--quiet`, which removes `pv` from >> the pipeline. It seems to be moving along, but of course I can't >> easily tell what it's doing. >>=20 >> Without `pv` in the way, the process reading the pipe is `lzop` >> instead, which doesn't try to do any fancy select() stuff, it's just a >> normal filter reading data from stdin and writing compressed data to >> stdout, in sequence. > I'm having the same issue here. I thought it was just my hardware and/or s= yncoid that was having this issue on 3.5TB dataset replications. >=20 > I'll try the quiet mode to see if removing PV solves the problem. I'm also= using compression=3Dnone so I won't see lzop. >=20 > Cheers. >=20 >>=20 --Apple-Mail-E7688D7A-DC31-4F7B-96F1-10E612C87F8B Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable Confirmed. Removing PV from the pipeline ha= s solved my issue. I was able to move 20TB around in the last 18 hours witho= ut issue.

Se= nt from my iPhone

On 15 J= ul 2024, at 6:42=E2=80=AFPM, Jason Tubnor <jason@tubnor.net> wrote:

=EF=BB=BF= =20 =20 =20

On 15/07/2024 9:35 am, Garrett Wollman wrote:
I'm currently running syncoid w=
ith `--quiet`, which removes `pv` from
the pipeline.  It seems to be moving along, but of course I can't
easily tell what it's doing.

Without `pv` in the way, the process reading the pipe is `lzop`
instead, which doesn't try to do any fancy select() stuff, it's just a
normal filter reading data from stdin and writing compressed data to
stdout, in sequence.
I'm having the same issue here. I thought it was just my hardware and/or syncoid that was having this issue on 3.5TB dataset replications.

I'll try the quiet mode to see if removing PV solves the problem. I'm also using compression=3Dnone so I won't see lzop.

Cheers.
=20

= --Apple-Mail-E7688D7A-DC31-4F7B-96F1-10E612C87F8B-- From nobody Tue Jul 16 23:41:33 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WNwZf6KB3z5QV8x for ; Tue, 16 Jul 2024 23:41:46 +0000 (UTC) (envelope-from eborisch@gmail.com) Received: from mail-qv1-xf33.google.com (mail-qv1-xf33.google.com [IPv6:2607:f8b0:4864:20::f33]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WNwZd5fLBz4WTF for ; Tue, 16 Jul 2024 23:41:45 +0000 (UTC) (envelope-from eborisch@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ad2GttrM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of eborisch@gmail.com designates 2607:f8b0:4864:20::f33 as permitted sender) smtp.mailfrom=eborisch@gmail.com Received: by mail-qv1-xf33.google.com with SMTP id 6a1803df08f44-6b5dd7cd945so31358566d6.1 for ; Tue, 16 Jul 2024 16:41:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721173304; x=1721778104; darn=freebsd.org; h=cc:to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=LibgtovcoT3o120H6yF9UuY19Ry4jvn9pqZ28BWvHZE=; b=Ad2GttrMQAzeDxMPs7wyrEnDYiF5WgLaii2WUcnRh+3JetbWRQ0qjZOhz+PhLVO4gq 9ETWFcRePiQU15ujJgkSvpIIOWvS1pwgcismQLsrFhUmoFUDA40sPA2SJTZ64P9AJzSY YGtXZ8YZzjNU9g5JvmlfH4TeHrrvc4RylzvtZMcMN0swdG87xA790wWYxuTXRyY7b9Ys UvgZ8ge6BPKHVPjj0mPqnYqZJ/JN5p5JhU/N0FnPAjn5rBCM+qi8dd2clUPk2u2D4aoU rRztkpvxxX7HewhJs/4RsIhbTqyYgPOB5xBWuiPUTzaHky5OU5pm1J23hnfytBURbKsw 2W9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721173304; x=1721778104; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=LibgtovcoT3o120H6yF9UuY19Ry4jvn9pqZ28BWvHZE=; b=JiXiUsWEsZQDxV/ROj3MjtvhuMHKlcD4XzYp1MbWiMD0MK5iMZVIXyhThtPl31IM5Q /FyJ3dL5E8Gt9YpgUD1U9ZxEolcy7zL2VOPIiOrRv7RIc0gpRbMcPslRls5bgn+KDh4A 0YAomcXbttqHYvJ7g37WRPaCM7L2x3Tw42wMa59yUq+KGSKdVx7l0v2FWMj6KIIZ2T9I Sxt9UraFZNb4ODSl8MeEmSzxF9RaPHkp4dVkv5yTkrUk2gN21YXVXI+OB1SferyoGTF+ XZ9rwMGZp6eLFsZ+tVjkw1Wqrkinb+ZuiyTzyWAyK0Y7Caal/NHXnzAUOIGYxqXIa5VQ cuQA== X-Forwarded-Encrypted: i=1; AJvYcCXsKiNLx1rVn31wTSKsWbYJBstN7MH23+3l6yFzZFHG0nI3TOGjOUr+9eEfYMB6AnfxvUurTia4nFoQlQtMJYnFPJGZVna4zbx4gw== X-Gm-Message-State: AOJu0Yxuv2vJFglIw41Z7lsK1d80AcGlP+sLoIo+PnDWlPYv9W1n9yKA lmq76g0igtmFSd8Z/4CMazZUpyfVVe/KItOkIy/3eBYikWfOeJjxfX2zyqQFw6VNqUaKRPL/VlO HRUM4vnki2vCd7Bsty4FOVVzNiEQ= X-Google-Smtp-Source: AGHT+IEbQKDXghQQMxgnLXf8LkF6KdyW9AyXPC6Liz/ocACaSM0/5cM1+G7OM6v/DYpUmHMSTjvKSVjDIbnVe2lg1MI= X-Received: by 2002:a05:6214:1cce:b0:6b5:23eb:3a49 with SMTP id 6a1803df08f44-6b78cab3f6dmr1688116d6.25.1721173304511; Tue, 16 Jul 2024 16:41:44 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 From: "Eric A. Borisch" Date: Tue, 16 Jul 2024 18:41:33 -0500 Message-ID: Subject: Re: Possible bug in zfs send or pipe implementation? To: jason@tubnor.net Cc: Garrett Wollman , rick.macklem@gmail.com, FreeBSD Stable Content-Type: multipart/alternative; boundary="0000000000006996ec061d65e301" X-Spamd-Bar: / X-Spamd-Result: default: False [-0.21 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; SUBJECT_ENDS_QUESTION(1.00)[]; FAKE_REPLY(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.71)[-0.710]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[FreeBSD]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_CC(0.00)[bimajority.org,gmail.com,freebsd.org]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_RCPT(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::f33:from] X-Rspamd-Queue-Id: 4WNwZd5fLBz4WTF --0000000000006996ec061d65e301 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Jul 16, 2024 at 4:41=E2=80=AFPM Jason Tubnor wro= te: > Confirmed. Removing PV from the pipeline has solved my issue. I was able > to move 20TB around in the last 18 hours without issue. > > Sent from my iPhone > > On 15 Jul 2024, at 6:42=E2=80=AFPM, Jason Tubnor wrote= : > > =EF=BB=BF > > > On 15/07/2024 9:35 am, Garrett Wollman wrote: > > I'm currently running syncoid with `--quiet`, which removes `pv` from > the pipeline. It seems to be moving along, but of course I can't > easily tell what it's doing. > > Without `pv` in the way, the process reading the pipe is `lzop` > instead, which doesn't try to do any fancy select() stuff, it's just a > normal filter reading data from stdin and writing compressed data to > stdout, in sequence. > > I'm having the same issue here. I thought it was just my hardware and/or > syncoid that was having this issue on 3.5TB dataset replications. > > I'll try the quiet mode to see if removing PV solves the problem. I'm als= o > using compression=3Dnone so I won't see lzop. > > Cheers. > > So I thought I had things pinned down in the debugger, but to clarify: writ= e() is _not_ returning 0; there=E2=80=99s an alarm going off that is breaking o= ut of the write loop while total_written is 0. They (pv) are returning (within pv=E2=80=99s write loop) 0 =E2=80=94 and while that=E2=80=99s completely va= lid if they=E2=80=99ve set an alarm and it goes off, they are interpreting it as =E2=80=9Cwe can=E2=80=99= t write to the file anymore.=E2=80=9D I=E2=80=99m pretty sure the other users on here experiencing errors with pv= in the zfs send|recv path are hitting this. What is interesting is that select() is indicating that the stdout is writable, but when pv tries to write on it, it blocks and makes no progress. You can see this with this reproducer: $ pv /dev/zero | { while sleep 1.1; do dd bs=3D63k count=3D1 of=3D/dev/null= ; done ; } 1+0 records in1 [62.4KiB/s] [ <=3D> ] 1+0 records out 64512 bytes transferred in 0.001132 secs (57005867 bytes/sec) 0+1 records in2 [0.00 B/s] [ <=3D> ] 0+1 records out 1024 bytes transferred in 0.000218 secs (4689890 bytes/sec) 64.0KiB 0:00:11 [0.00 B/s] [ <=3D> Looking at the ktrace, and focusing on the selects and writes on stdout, we have: 9684 pv CALL select(0x2,0x820c27060,0x820c26fe0,0x820c26f60,0x820c26f50) 9684 pv RET select 1 #### This is "writable on stdout" based on what happens next 9684 pv CALL setitimer(ITIMER_REAL,0x820c27060,0) 9684 pv STRU itimerval { .interval =3D {1, 0}, .value =3D {1, 0} } 9684 pv STRU itimerval { .interval =3D {0, 0}, .value =3D {0, 0} } 9684 pv RET setitimer 0 9684 pv CALL write(0x1,0xe084aa3a000,0x20000) #### Tries to write 64k 9684 pv GIO fd 1 wrote 4096 bytes "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\= 0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0= \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\= 0\0\0\0\0\0\0\0\0\ .... 9684 pv RET write 65536/0x10000 #### Wrote 64k to stdout. This is all that ever makes it out [skip] (read; display update, etc.) 9684 pv CALL select(0x2,0x820c27060,0x820c26fe0,0x820c26f60,0x820c26f50) 9684 pv RET select 1 #### This is again "writable on stdout" based on what happens next 9684 pv CALL setitimer(ITIMER_REAL,0x820c27060,0) 9684 pv STRU itimerval { .interval =3D {1, 0}, .value =3D {1, 0} } 9684 pv STRU itimerval { .interval =3D {0, 0}, .value =3D {0, 0} } 9684 pv RET setitimer 0 9684 pv CALL write(0x1,0xe084aa3a000,0x20000) #### Tries to write 64k (since select() indicated it was writable) 9684 pv GIO fd 1 wrote 4096 bytes "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\= 0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0= \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\= 0\0\0\0\0\0\0\0\0\ .... 9684 pv RET write -1 errno 4 Interrupted system call #### Gets -1 / EINTR (pv sets up a 1s. timer), so no progress write()-ing at all after the successful select(). So it is a bit odd to return the FD as writable, but then block on a write to that FD / not make any progress. It appears Linux (if I compile pv without splice()) does a partial write to fill up the buffer, (total_written is then > 0) and then blocks on the subsequent call and ends up with -1/EINTR in the write loop, likewise on MacOS. Do we intend to return a FD as writable from select() only to have it block and not progress on the next attempted write()? I imagine there is a smaller write that would succeed at this point. Thanks, - Eric --0000000000006996ec061d65e301 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Tue, Jul 16, 20= 24 at 4:41=E2=80=AFPM Jason Tubnor <= jason@tubnor.net> wrote:

Confirmed. Removing PV from the pipeline has solved my issue. I was able t= o move 20TB around in the last 18 hours without issue.
Sent from = my iPhone
On 15 Jul 202= 4, at 6:42=E2=80=AFPM, Jason Tubnor <jason@tubnor.net> wrote:
=EF=BB=BF

On 15= /07/2024 9:35 am, Garrett Wollman wrote:
I'm currently running syncoid with `--quiet`, which removes `pv` =
from
the pipeline.  It seems to be moving along, but of course I can't
easily tell what it's doing.

Without `pv` in the way, the process reading the pipe is `lzop`
instead, which doesn't try to do any fancy select() stuff, it's jus=
t a
normal filter reading data from stdin and writing compressed data to
stdout, in sequence.
I'm having the same issue here. I thought it was = just my hardware and/or syncoid that was having this issue on 3.5TB dataset= replications.
I'll try the quiet mode to see if removing PV solv= es the problem. I'm also using compression=3Dnone so I won't see lz= op.
Cheers.

So I thought I had t= hings pinned down in the debugger, but to clarify:=C2=A0write() is _not_ returning 0; there=E2= =80=99s an alarm going off that is breaking out of the write loop while tot= al_written is 0. They (pv) are returning (within pv=E2=80=99s write loop) 0= =E2=80=94 and while that=E2=80=99s completely valid if they=E2=80=99ve set= an alarm and it goes off, they are interpreting it as =E2=80=9Cwe can=E2= =80=99t write to the file anymore.=E2=80=9D

<= span style=3D"font-family:-apple-system,helveticaneue">

I=E2= =80=99m pretty sure the other users on here experiencing errors with pv in = the zfs send|recv path are hitting this.

<= span style=3D"font-family:-apple-system,helveticaneue">What is interesting = is that select() is indicating that the stdout is writable, but when pv tri= es to write on it, it blocks and makes no progress. You can see this with t= his reproducer:

$ pv /dev/zero | { while sleep 1.1; do = dd bs=3D63k count=3D1 of=3D/dev/null; done ; }

1+0 records in1 [62.4KiB/s= ] [ =C2=A0 =C2=A0<=3D> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0]
= 1+0 records out
64512 bytes transferred in 0.001132 secs (57005867 bytes= /sec)
0+1 records in2 [0.00 =C2=A0B/s] [ =C2=A0 =C2=A0<=3D> =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ]
0+1 records out
1024 bytes tran= sferred in 0.000218 secs (4689890 bytes/sec)
64.0KiB 0:00:11 [0.00 =C2= =A0B/s] [ =C2=A0 =C2=A0<=3D>

<Never gets past 64KB&g= t;

Looking at the ktrace, and focusing on the sele= cts and writes on stdout, we have:

=C2=A0 9684 pv = =C2=A0 =C2=A0 =C2=A0 CALL =C2=A0select(0x2,0x820c27060,0x820c26fe0,0x820c26= f60,0x820c26f50)
=C2=A0 9684 pv =C2=A0 =C2=A0 =C2=A0 RET =C2=A0 select 1=

#### This is "writable on stdout" based= on what happens next

=C2=A0 9684 pv =C2=A0 =C2=A0= =C2=A0 CALL =C2=A0setitimer(ITIMER_REAL,0x820c27060,0)
=C2=A0 9684 pv = =C2=A0 =C2=A0 =C2=A0 STRU =C2=A0itimerval { .interval =3D {1, 0}, .value = =3D {1, 0} }
=C2=A0 9684 pv =C2=A0 =C2=A0 =C2=A0 STRU =C2=A0itimerval { = .interval =3D {0, 0}, .value =3D {0, 0} }
=C2=A0 9684 pv =C2=A0 =C2=A0 = =C2=A0 RET =C2=A0 setitimer 0
=C2=A0 9684 pv =C2=A0 =C2=A0 =C2=A0 CALL = =C2=A0write(0x1,0xe084aa3a000,0x20000)

#### T= ries to write 64k

=C2=A0 9684 pv =C2=A0 =C2=A0 =C2=A0 G= IO =C2=A0 fd 1 wrote 4096 bytes
=C2=A0 =C2=A0 =C2=A0 =C2=A0"\0\0\0\= 0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0= \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\= 0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0= \0\0\0\0\

....

=C2=A0 9684 pv =C2=A0 =C2=A0 =C2=A0 = RET =C2=A0 write 65536/0x10000

#### Wrote= 64k to stdout. This is all that ever makes it out

[skip] (read; display update, etc.)

=C2=A0 9684 pv =C2=A0 =C2=A0 =C2=A0 CALL =C2=A0sele= ct(0x2,0x820c27060,0x820c26fe0,0x820c26f60,0x820c26f50)
=C2=A0 9684 pv = =C2=A0 =C2=A0 =C2=A0 RET =C2=A0 select 1

####= This is again "writable on stdout" based on what happens next

=C2=A0 9684 pv =C2=A0 =C2=A0 =C2=A0 CALL =C2= =A0setitimer(ITIMER_REAL,0x820c27060,0)
=C2=A0 9684 pv =C2=A0 =C2=A0 =C2= =A0 STRU =C2=A0itimerval { .interval =3D {1, 0}, .value =3D {1, 0} }
=C2= =A0 9684 pv =C2=A0 =C2=A0 =C2=A0 STRU =C2=A0itimerval { .interval =3D {0, 0= }, .value =3D {0, 0} }
=C2=A0 9684 pv =C2=A0 =C2=A0 =C2=A0 RET =C2=A0 se= titimer 0
=C2=A0 9684 pv =C2=A0 =C2=A0 =C2=A0 CALL =C2=A0write(0x1,0xe08= 4aa3a000,0x20000)

#### Tries to write 64k (si= nce select() indicated it was writable)

=C2=A0 9684 pv = =C2=A0 =C2=A0 =C2=A0 GIO =C2=A0 fd 1 wrote 4096 bytes
=C2=A0 =C2=A0 =C2= =A0 =C2=A0"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\= 0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0= \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\= 0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\

....

=C2=A0 9684 p= v =C2=A0 =C2=A0 =C2=A0 RET =C2=A0 write -1 errno 4 Interrupted system call<= br>

#### Gets -1 / EINTR (pv sets up a 1s. timer),= so no progress write()-ing at all after the successful select().

So it is a bit odd to return the FD as writable, but then b= lock on a write to that FD / not make any progress. It appears Linux (if I = compile pv without splice()) does a partial write to fill up the buffer, (t= otal_written is then > 0) and then blocks on the subsequent call and end= s up with -1/EINTR in the write loop,=C2=A0likewise on MacOS.

Do we intend to return a FD as writable from select() only to h= ave it block and not progress on the next attempted write()? I imagine ther= e is a smaller write that would succeed at this point.

Thanks,

=C2=A0 =C2=A0- Eric

--0000000000006996ec061d65e301-- From nobody Thu Jul 18 20:45:54 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQ4bD0GfFz5QM30 for ; Thu, 18 Jul 2024 20:46:16 +0000 (UTC) (envelope-from stable@lordcow.org) Received: from mail.lordcow.org (lordcow.org [IPv6:2c0f:fb18:402:5::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "devaux.za.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQ4bB2XVyz43HR for ; Thu, 18 Jul 2024 20:46:14 +0000 (UTC) (envelope-from stable@lordcow.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of stable@lordcow.org designates 2c0f:fb18:402:5::2 as permitted sender) smtp.mailfrom=stable@lordcow.org Received: from lordcow.org (localhost [127.0.0.1]) by mail.lordcow.org (8.17.2/8.17.2) with ESMTPS id 46IKk0Gm082733 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT) for ; Thu, 18 Jul 2024 22:46:00 +0200 (SAST) (envelope-from lordcow@lordcow.org) X-Authentication-Warning: lordcow.org: Host localhost [127.0.0.1] claimed to be lordcow.org Received: (from lordcow@localhost) by lordcow.org (8.17.2/8.17.2/Submit) id 46IKjshf082527 for freebsd-stable@freebsd.org; Thu, 18 Jul 2024 22:45:54 +0200 (SAST) (envelope-from lordcow) Date: Thu, 18 Jul 2024 22:45:54 +0200 From: Gareth de Vaux To: freebsd-stable@freebsd.org Subject: nextboot warns it won't reset Message-ID: List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on lordcow.org X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; R_SPF_ALLOW(-0.20)[+ip6:2c0f:fb18:402:5::2]; MIME_GOOD(-0.10)[text/plain]; MID_RHS_MATCH_FROM(0.00)[]; HAS_XAW(0.00)[]; ASN(0.00)[asn:37199, ipnet:2c0f:fb18::/32, country:ZA]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; MISSING_XM_UA(0.00)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[lordcow.org]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4WQ4bB2XVyz43HR Hi all, nextboot warns as follows: # nextboot -k testkernel WARNING: loader(8) has only R/O support for ZFS nextboot.conf will NOT be reset in case of kernel boot failure This's on a ZFS zroot 12.4-STABLE system. Does this mean that nextboot will not do its job? I see a blog mentioning that nextboot will write metadata to the ZFS pool label, in which case maybe the warning is no longer applicable? Also I suspect I should rather run "nextboot -D" to revert the situation. Is this equivalent to removing /boot/nextboot.conf and/or this metadata in the ZFS pool label? From nobody Fri Jul 19 01:10:05 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQBRm6Ypxz5Qm8s for ; Fri, 19 Jul 2024 01:10:12 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQBRm62srz4SDM; Fri, 19 Jul 2024 01:10:12 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721351412; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=2oAD/HWKFDfA1fM+OmQ4OEJQQxlFZMNSgl82ux2TUAE=; b=akhRZrVC5mv9QIk9F/3E2rsCkC9KWky44YN91iAmfwcikNylejVK3Py60CKqh8ETwl1soH 884ov4cS5Tdnx3L6bPrAO9pi3soQdtrmwhwEesIxdBXIPJ9v3aGEPDUUS0N2mplPpZp7AJ YJHHk1sKXCK7YdWt8hVd3UsiNUnjETW7029+bILfsTwUAehf+iF9+hkQDvYpPPiNF4I2+c oC8qFyuqAyODBsUMUHjkpIcVjWQUYJMY42A5mYPNw+ahCfCSh337rh1KBkcas3hAMlsPhE VppgqBg6/aMRQE5vWDsnLxv9EDZsPiPGjdUGEXg034fDSiqT0ScJ4lNWQif6zA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721351412; a=rsa-sha256; cv=none; b=iYE3W8av3nPxuq8Utz5CCcR7oNnSNKKPQRjJKDNF/RqNMkhszDUjBqdKS2EjUSOp+/ZPgo 4s/d9+hStRGdYXm2TLnd+9OtkL3aYUDHbdFdC2fInGMclzfc7VgesR2Rvd3+TRNS/1f0f0 f3rL9GHskSurb8GmSKyXJsz8YDsC7HfO0wl4+Hfuf+sqaOgUKK7CXcQ5Hv0bbO5+Zi+wWc cMDu2EPZT4Hd7oOW25LDR2wJQ0f3ITaugqyXv2sSN9NoSISWun3AR9EK4M3PQFDNHsh2TB wlfZp8L0KsGN133ge1rgHh8ALjy/ycJT1U7/bNq2+zUxMptmRDM4i/cKj1VJ8Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721351412; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=2oAD/HWKFDfA1fM+OmQ4OEJQQxlFZMNSgl82ux2TUAE=; b=DZlrLWAEnCV1v/dHSG7hqBThtlg9udhNPVICTyq3Cudr83ZbN06KLS1MVWZGC5NEknpwqs Dsfaie/Y18QhLXVeZih/gURssHhYQ/Nn5gGKaRrRRJZH+GmKEbJMfTvN84XZfKHWxWHlYO GreZVLoTEcQZlyc4ecvnO8PTrN0ibJPv3DvtHs4oLllBBBlpr6O5pIxJk6/RqnvIErveyL xapD6D8fUp/5rujJ69150eRBplOiqeFGYgdtsNVNznT8pZV3+1nD4/8CuIVwbNGHcJlqih 9jfvAsM6ZYIlXnrhrkv48Qw6bE6eau/m/Z5A/wj855DpS5JwdPfE3pmdErG1LQ== Received: from smtpclient.apple (unknown [IPv6:2001:19f0:6001:9db:98f0:9fe0:3545:10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4WQBRl6jBZz12xh; Fri, 19 Jul 2024 01:10:11 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Message-Id: <01D917DB-7E47-46D7-AD22-AD09C4F89A96@FreeBSD.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_071E00C1-5AEE-4D1D-80DC-867748BDB8A2" List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 $3696.120.41.1.8$) Subject: Re: nextboot warns it won't reset Date: Fri, 19 Jul 2024 09:10:05 +0800 In-Reply-To: Cc: freebsd-stable@freebsd.org To: Gareth de Vaux References: X-Mailer: Apple Mail (2.3696.120.41.1.8) --Apple-Mail=_071E00C1-5AEE-4D1D-80DC-867748BDB8A2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Jul 19, 2024, at 4:45 AM, Gareth de Vaux = wrote: >=20 > Hi all, nextboot warns as follows: >=20 > # nextboot -k testkernel > WARNING: loader(8) has only R/O support for ZFS > nextboot.conf will NOT be reset in case of kernel boot failure >=20 >=20 > This's on a ZFS zroot 12.4-STABLE system. You're encouraged to upgrade to supported releases ;) >=20 > Does this mean that nextboot will not do its job? I think the WARNING is a good hint and explain why it does not work = incase boot failure. The implementation of load desired kernel is write loader parameters to = /boot/nexboot.conf , to indicate the loader(8) do its job, that is loading the desired kernel on next = boot. Well it is a oneshot boot so the loader will try to disable these parameters by writing = `nextboot_enable=3DNO` to /boot/nexboot.conf. That is the magic. Apparently if loader(8) does not support write to = filesystem, in this case the ZFS,=20 it will fail to write /boot/nexboot.conf in case kernel boot failure. = Thus subsequent boot the loader(8) would still try to boot from the oneshot `testkernel`. >=20 > I see a blog mentioning that nextboot will write metadata to the ZFS = pool label, > in which case maybe the warning is no longer applicable? That is implementation specific. Normally you can ignore the warning, = unless you have trouble booting the kernel. >=20 >=20 > Also I suspect I should rather run "nextboot -D" to revert the = situation. > Is this equivalent to removing /boot/nextboot.conf and/or this = metadata in the > ZFS pool label? I think normally you do not need that. Best regards, Zhenlei --Apple-Mail=_071E00C1-5AEE-4D1D-80DC-867748BDB8A2 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

On Jul 19, 2024, at 4:45 AM, Gareth de Vaux <stable@lordcow.org> = wrote:

Hi all, nextboot warns as follows:

# nextboot -k testkernel
WARNING: loader(8) has = only R/O support for ZFS
nextboot.conf will NOT be reset = in case of kernel boot failure

This's on a ZFS zroot 12.4-STABLE system.

You're = encouraged to upgrade to supported releases ;)

Does this mean that nextboot will not do its = job?

I think the WARNING is a good hint and explain why = it does not work incase boot failure.

The implementation of load desired kernel is write = loader parameters to /boot/nexboot.conf , to indicate

the = loader(8) do its job, that is loading the desired kernel on next boot. = Well it is a oneshot boot

so the loader will try to disable = these parameters by writing `nextboot_enable=3DNO` to /boot/nexboot.conf.

That is the magic. Apparently if = loader(8) does not support write to filesystem, in this case the = ZFS,

it will fail to write /boot/nexboot.conf in case kernel boot failure. Thus = subsequent boot the loader(8) would

still= try to boot from the oneshot `testkernel`.

I see a blog mentioning that nextboot will = write metadata to the ZFS pool label,
in which case maybe = the warning is no longer applicable?

That = is implementation specific. Normally you can ignore the warning, unless = you have trouble booting

the kernel.

Also I suspect I should rather = run "nextboot -D" to revert the situation.
Is this = equivalent to removing /boot/nextboot.conf and/or this metadata in = the
ZFS pool label?

I = think normally you do not need that.

Best regards,

Zhenlei

= --Apple-Mail=_071E00C1-5AEE-4D1D-80DC-867748BDB8A2-- From nobody Fri Jul 19 02:19:12 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQCzf4gBHz5QtC9 for ; Fri, 19 Jul 2024 02:19:26 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-oo1-xc29.google.com (mail-oo1-xc29.google.com [IPv6:2607:f8b0:4864:20::c29]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQCzd4ThNz4YHg for ; Fri, 19 Jul 2024 02:19:25 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oo1-xc29.google.com with SMTP id 006d021491bc7-5c669a0b5d1so791767eaf.3 for ; Thu, 18 Jul 2024 19:19:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1721355564; x=1721960364; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=HnXk6+VUbVJDgzVoMDRxwT3i0Gf54fkEdx6LY55RDAs=; b=P8Tk30dYljSM6fplQoFQ0t3DdGrml5nchcobsJ9GnNcluqZd8l/NwRwc9hV6lfIlPE rHpQYN1LyNZcAjqjmmuK4Y2EbdfUT7aHcBW3Bi93zB5tjaCC13PNFlP2C+ZmOiUzws0D EyjVOZBcm07vMwh1C05rVfpaP9Ato+cF3mPWM1IZzDA1Qyw7jP0ua/B7Mnn7WT4udq3r gnOtJ+UdwpPmwjo/RlOM7SA9I8ELhDsIAHR5zteg0TbkPzgb9SYkA3b6ccCbNiP3Qymf Y9gDigKdNA7+a35aTeZQk5TYkk4ldaZofux6TMK9FCdlZLGrA5BJfxMvS9H26/PIx20I APPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721355564; x=1721960364; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=HnXk6+VUbVJDgzVoMDRxwT3i0Gf54fkEdx6LY55RDAs=; b=TzyGWQByCJ5ncscxKcCa5J9CqZHxg58wDuZ88OiVarEpln1OUthmpJ2igE+zDnxGN6 b/52e9Ib7ZTE9iqYID2vDP7gtREpaVz7qb7Svm2FVsZeyhHzxJc2CTm0NMi9T82rQYlM CehBVAMgwHbdIvG5dRbbOmAfsAOVPP+17mIPYLk5HD7DHZg0N/NHkZKM8ld7nO8IJEGP 7LmP/NyVBbzMtebXTNgk91pUdvHi1RlscRlAu+4YVsO4LHp23zKy1cSAkRxuyBABz4aK Xy/gKddrBuwFriSJuT680FaxlLNrVCvf7u7AQKz1RZJyzBQOKYPOZktv5J6PjFgKsbCP zgyg== X-Forwarded-Encrypted: i=1; AJvYcCXap5XxdYS7TR4jd+T1Pn+A1dNM6l5Je3cbM20ydva78TtZemOFsEtSiQEnqKstVr7uV2wrj5MxPsz7hCKX7o2kQ2R7aZ42ucKDvw== X-Gm-Message-State: AOJu0YwKouhOM05rw/RFW4eMKaop7sGAqp7vSMHlKvknbDsMSBaGw/uv LG+axkxD1W6ut1jzEKcNACHmRlccdYIaJev8t8KEHVIvvlWw0zfenBh7uSs1mZCkXM5n7OPqdKy DCWpsC57H04ib7sTSB7sBcrrMjBlvC5UYtdnf1V5Tziy1+4we X-Google-Smtp-Source: AGHT+IGsx89iV4gfdUq5S6u1sH1SZ1B1D/2pzWniX3pbfkB++GDmOK0ourd1qO8i7xoJIEAAd3Ddo0ylotAlxOTUWak= X-Received: by 2002:a05:6359:5a8f:b0:1aa:a01a:23da with SMTP id e5c5f4694b2df-1aca9f9c8aemr330161255d.27.1721355564236; Thu, 18 Jul 2024 19:19:24 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 References: <01D917DB-7E47-46D7-AD22-AD09C4F89A96@FreeBSD.org> In-Reply-To: <01D917DB-7E47-46D7-AD22-AD09C4F89A96@FreeBSD.org> From: Warner Losh Date: Thu, 18 Jul 2024 20:19:12 -0600 Message-ID: Subject: Re: nextboot warns it won't reset To: Zhenlei Huang Cc: Gareth de Vaux , FreeBSD-STABLE Mailing List Content-Type: multipart/alternative; boundary="000000000000f06acc061d905292" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4WQCzd4ThNz4YHg --000000000000f06acc061d905292 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Jul 18, 2024, 7:10=E2=80=AFPM Zhenlei Huang wrot= e: > > > On Jul 19, 2024, at 4:45 AM, Gareth de Vaux wrote: > > Hi all, nextboot warns as follows: > > # nextboot -k testkernel > WARNING: loader(8) has only R/O support for ZFS > nextboot.conf will NOT be reset in case of kernel boot failure > > > This's on a ZFS zroot 12.4-STABLE system. > > > You're encouraged to upgrade to supported releases ;) > > > Does this mean that nextboot will not do its job? > > > I think the WARNING is a good hint and explain why it does not work incas= e > boot failure. > > The implementation of load desired kernel is write loader parameters to > /boot/nexboot.conf , to indicate > the loader(8) do its job, that is loading the desired kernel on next boot= . > Well it is a oneshot boot > so the loader will try to disable these parameters by writing `nextboot_e= nable=3DNO` > to /boot/nexboot.conf. > > That is the magic. Apparently if loader(8) does not support write to > filesystem, in this case the ZFS, > it will fail to write /boot/nexboot.conf in case kernel boot failure. > Thus subsequent boot the loader(8) would > still try to boot from the oneshot `testkernel`. > However, ZFS has special support via properties in the BE that lets you do boot once and nextboot stuff. But even better are boot environments. That's a more comple solution that supports bootnext features and more. See bectl... Warner I see a blog mentioning that nextboot will write metadata to the ZFS pool > label, > in which case maybe the warning is no longer applicable? > > > That is implementation specific. Normally you can ignore the warning, > unless you have trouble booting > the kernel. > > > > Also I suspect I should rather run "nextboot -D" to revert the situation. > Is this equivalent to removing /boot/nextboot.conf and/or this metadata i= n > the > ZFS pool label? > > > I think normally you do not need that. > > Best regards, > Zhenlei > > > --000000000000f06acc061d905292 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Thu, Jul 18, 2024, 7:10=E2=80=AFPM Zhenlei Huang &l= t;zlei@freebsd.org> wrote:

On Jul= 19, 2024, at 4:45 AM, Gareth de Vaux <stable@lordcow.org> wrote:=

Hi all, nextboot warns as follows:

# nextboot -k= testkernel
WARNING: loader(8) has only R/O support for ZFS
nextboot.= conf will NOT be reset in case of kernel boot failure

This's= on a ZFS zroot 12.4-STABLE system.

You're encouraged to upgrade to supported releases ;)

Does this mean that nextboot will = not do its job?

I think the= WARNING is a good hint and explain why it does not work incase boot failur= e.

The implementation of load desired kernel is wr= ite loader parameters to /boot/nexboot.conf , to indicate
the loa= der(8) do its job, that is loading the desired kernel on next boot. Well it= is a oneshot boot
so the loader will try to disable these=C2=A0<= span style=3D"color:rgb(0,0,0)">parameters by writing `nextboot_enable=3DNO` to =C2=A0/boot/nexboot.conf.

Tha= t is the magic. Apparently if loader(8) does not support write to filesyste= m, in this case the ZFS,=C2=A0
it will fail to write=C2=A0/b= oot/nexboot.conf in case kernel boot failure. Thus subsequent boot the load= er(8) would
still try to = boot from the oneshot `testkernel`.

However, ZFS has sp= ecial support via properties in the BE that lets you do boot once and nextb= oot stuff.

But even bett= er are boot environments. That's a more comple solution that supports b= ootnext features and more. See bectl...

Warner=C2=A0

I see a blog mentioning that nextboot will write metad= ata to the ZFS pool label,
in which case maybe the warning is no longer = applicable?

That is impleme= ntation specific. Normally you can ignore the warning, unless you have trou= ble booting
the kernel.

<= div>

Also I suspect I should rather run "nextboot -D" to r= evert the situation.
Is this equivalent to removing /boot/nextboot.conf = and/or this metadata in the
ZFS pool label?

I think normally you do not need that.

Best regards,

Zhenlei

--000000000000f06acc061d905292-- From nobody Fri Jul 19 04:49:13 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQHJd2LTjz5R7Gw for ; Fri, 19 Jul 2024 04:49:21 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQHJd1n80z4mYC; Fri, 19 Jul 2024 04:49:21 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721364561; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=BR+HNjNkR1aoEP/NF8JdqBB+8YqnPwbyC8EmGGBfeqY=; b=HkcWB1pFOcooN8GZ94wOCn/coIWrFze4EbkkOJhOyoPRYG/whqyPF93mjv2/khMxwkw15z XE6yrjNvA4UQH+GZdpIB9UUmbodkyGJLagEwPa2fcc6yP9SwvA+UMZH9nI9MbS+qEGTuk9 X0izvGsjPy6vW0uu+6URn8yA/i+Hq2lNRZPrJBcl7eorf6mGsAAw0FJR8TakM2BjyD2kMb UtKtWvYB5gcT/DBb6OXlhutIfRMEY/XINsQglvqXIDgvHIB7vBk3AiDK4Osgk2mk4qcnZe dybnqY+sd/RPNmpuRjqKekzLhvAQp2ueLRZgv7JVky5WW44glRVKnl6/xQ+2Vg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721364561; a=rsa-sha256; cv=none; b=Z0cW0JLrNhd9KnA+z10m4hj+qzqWSiSbPAwjzTSW5vAM309PQEXxXI0/SERuqDUnDs8Hmn kdFOIlcqNJNGj4fZ0ggm1QWkHEa4qGlRGFxhDjrhZjIDCkjMSg/T48wDwiiR6c0bkA4uLz kPBePF7zxLlxGXLBYHa6oKUcDF0xowu85PUDHa9kfdOmd3MwGwI+MDNhUx1cAPowlZyGtm YPAC7hi4HiyjWDH0skuzOVmn029EOPlXQSM50n0Sv2gLzlSUr+/IugAw9wwZIJ0+sTZNSZ FqxKNP8omgdaV/gBpvsoqETd/EumBZ9hd0z9RfQAFJKjSNrKKh2tjrRfRl+ExA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721364561; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=BR+HNjNkR1aoEP/NF8JdqBB+8YqnPwbyC8EmGGBfeqY=; b=x0/Wb/NOKJT0CjHPuVL91V2tF28rxEsYQskMzluy+nfqKW3xt+RJqyf7Cz+7nD2mF0N2m1 Oc99prLigtVJk0lnxTo51hT5XyQ4ERNcewc4mqkPxujGdk1O00l/VYDsviy/vXAwSQ2CtY KK+E7cMIHd/k/X6PPC2NbUlzLP61RL5M/XppxIhltFQlP3PuSo6/ekwSXVXAX4CFiwdt0f ZtigqCukNioNJ3FR8mBeVm8fL2NYkMisrlZ5ZOqrH8zQMRw7fq8FLKMTWvEc4R4hZCBkJh JiRRDPqDqZ/k+PzZkmopa0Uy/DvG443INlaMM1YJaOCKnRDEPCe5nIp2D8NkIg== Received: from smtpclient.apple (ns1.oxydns.net [45.32.91.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4WQHJc1Th6z172k; Fri, 19 Jul 2024 04:49:19 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Message-Id: <9CCA0BFB-70D3-4E21-8BA7-3E9F35110E8D@FreeBSD.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_84D08D30-9414-451B-ACA2-C530FE1DB7F8" List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 $3696.120.41.1.8$) Subject: Re: nextboot warns it won't reset Date: Fri, 19 Jul 2024 12:49:13 +0800 In-Reply-To: Cc: Gareth de Vaux , FreeBSD-STABLE Mailing List To: Warner Losh References: <01D917DB-7E47-46D7-AD22-AD09C4F89A96@FreeBSD.org> X-Mailer: Apple Mail (2.3696.120.41.1.8) --Apple-Mail=_84D08D30-9414-451B-ACA2-C530FE1DB7F8 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jul 19, 2024, at 10:19 AM, Warner Losh wrote: >=20 >=20 >=20 > On Thu, Jul 18, 2024, 7:10=E2=80=AFPM Zhenlei Huang > wrote: >=20 >=20 >> On Jul 19, 2024, at 4:45 AM, Gareth de Vaux > wrote: >>=20 >> Hi all, nextboot warns as follows: >>=20 >> # nextboot -k testkernel >> WARNING: loader(8) has only R/O support for ZFS >> nextboot.conf will NOT be reset in case of kernel boot failure >>=20 >>=20 >> This's on a ZFS zroot 12.4-STABLE system. >=20 > You're encouraged to upgrade to supported releases ;) >=20 >>=20 >> Does this mean that nextboot will not do its job? >=20 > I think the WARNING is a good hint and explain why it does not work = incase boot failure. >=20 > The implementation of load desired kernel is write loader parameters = to /boot/nexboot.conf , to indicate > the loader(8) do its job, that is loading the desired kernel on next = boot. Well it is a oneshot boot > so the loader will try to disable these parameters by writing = `nextboot_enable=3DNO` to /boot/nexboot.conf. >=20 > That is the magic. Apparently if loader(8) does not support write to = filesystem, in this case the ZFS,=20 > it will fail to write /boot/nexboot.conf in case kernel boot failure. = Thus subsequent boot the loader(8) would > still try to boot from the oneshot `testkernel`. >=20 > However, ZFS has special support via properties in the BE that lets = you do boot once and nextboot stuff. Yes and NO. Yes for 13.x, 14.x and CURRENT. nextboot(8) will consults zfsbootcfg(8) = to do the nextboot stuff if file system is ZFS. No for 12.x. neither nextboot(8) nor nextboot.sh will do that. And it = seems that loader(8) also do not get / set ZFS properties to enable / disable nextboot stuff. >=20 > But even better are boot environments. That's a more comple solution = that supports bootnext features and more. See bectl... >=20 > Warner=20 >=20 >> I see a blog mentioning that nextboot will write metadata to the ZFS = pool label, >> in which case maybe the warning is no longer applicable? >=20 > That is implementation specific. Normally you can ignore the warning, = unless you have trouble booting > the kernel. >=20 >>=20 >>=20 >> Also I suspect I should rather run "nextboot -D" to revert the = situation. >> Is this equivalent to removing /boot/nextboot.conf and/or this = metadata in the >> ZFS pool label? >=20 > I think normally you do not need that. >=20 > Best regards, > Zhenlei --Apple-Mail=_84D08D30-9414-451B-ACA2-C530FE1DB7F8 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Jul 19, 2024, at 10:19 AM, Warner Losh <imp@bsdimp.com> = wrote:

On Thu, Jul 18, 2024, 7:10=E2=80=AFPM Zhenlei Huang = <zlei@freebsd.org> wrote:

On Jul 19, 2024, at 4:45 AM, Gareth de Vaux <stable@lordcow.org> wrote:

Hi all, nextboot warns as follows:

# nextboot -k testkernel
WARNING: = loader(8) has only R/O support for ZFS
nextboot.conf will = NOT be reset in case of kernel boot failure

This's on a ZFS zroot 12.4-STABLE system.

You're encouraged to upgrade to = supported releases ;)

Does this mean = that nextboot will not do its job?

I think the WARNING is a good hint and = explain why it does not work incase boot failure.
The implementation of load desired = kernel is write loader parameters to /boot/nexboot.conf , to = indicate
the loader(8) do its job, that is loading = the desired kernel on next boot. Well it is a oneshot boot
so the loader will try to disable these parameters by writing `nextboot_enable=3DNO` to /boot/nexboot.conf.

That is the magic. Apparently if loader(8) does = not support write to filesystem, in this case the = ZFS,
it = will fail to write /boot/nexboot.conf in case kernel boot failure. Thus = subsequent boot the loader(8) would
still try to boot from the oneshot = `testkernel`.

However, ZFS has special support via properties in the BE = that lets you do boot once and nextboot = stuff.

Yes = and NO.

Yes for 13.x, 14.x and = CURRENT. nextboot(8) will consults = zfsbootcfg(8) to do the nextboot stuff if file = system

is ZFS.

No for 12.x. neither nextboot(8) nor = nextboot.sh will do that. And it seems that loader(8) also do not get / = set

ZFS properties to enable / disable nextboot = stuff.

But even better are boot = environments. That's a more comple solution that supports bootnext = features and more. See bectl...

Warner

I see a blog mentioning that = nextboot will write metadata to the ZFS pool label,
in = which case maybe the warning is no longer applicable?

That is implementation specific. = Normally you can ignore the warning, unless you have trouble = booting
the kernel.

Also I suspect I should rather run "nextboot = -D" to revert the situation.
Is this equivalent to = removing /boot/nextboot.conf and/or this metadata in the
ZFS= pool label?

I think normally you do not need = that.

Best regards,
Zhenlei
<= /div>

= --Apple-Mail=_84D08D30-9414-451B-ACA2-C530FE1DB7F8-- From nobody Fri Jul 19 04:58:58 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQHX02k7qz5QtW3 for ; Fri, 19 Jul 2024 04:59:12 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQHWz69cCz4pjr for ; Fri, 19 Jul 2024 04:59:11 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pg1-x52b.google.com with SMTP id 41be03b00d2f7-78c84837564so1062570a12.3 for ; Thu, 18 Jul 2024 21:59:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1721365151; x=1721969951; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=D0SjkXrpQHZgQROzpQwXqfXPLpjdRTwbllzvBbECvMs=; b=Bu+QXNMPBBbwhoBdSqzWfY9EM/4jcQ9+vJutKr+UjtBx60zCA3P7KywJylDFW8p5di qZ6V+GQObuELnRYg5A0bNxkNwrNNWea3qWbDH0o9/avFxmWsWtJd+2lcFhRTWzz5Xeu6 COsvwxpSY/maQdtukOhRPrcSoz7k4cVIiXe+7QBUOlfGaMSXXGsH5c6XISSc98mM2OR5 dU/cgeGHu/+ZmMTKwa77VhGf1t6KyYD3lzoCxi7Pc0UIGqYt2FN7uaUvfVrJykC/e32l u4fgPaXggSp0gfvNDbcajEQPGhIPJK65JHWr2VTpUjeLdosqfSmmOKbMLS6L/xC1z7w2 lt2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721365151; x=1721969951; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=D0SjkXrpQHZgQROzpQwXqfXPLpjdRTwbllzvBbECvMs=; b=NrJSQckVnJtMSreT1QxyqY5qOqeAOe/YWUePVB+Bvzpm3VzqCjf0mbAjJPRzk/FJEW 5bLzyjc487woC5PQDg5zQ6FkUEqVCHI6zXPLB9l3ju+Ht4ZfYw825b3u1sMVmAQwhBHq GJv6uaPmwhi+E8FUMjxMsp9qhFittCED2lT60xMC2BN8F3IW2zhsxResLaf32XXX2QLK NN10Jc0b/mz8vKzysFaJSzHgJHt87VkCZohbO1qAA98fBM8M9tjfMqhrKdK+DlTAKgLm 9P6oKNTpWp84u0YU5ppxAirMC+hr9QgoDfg2CDzR5mhrBNbqUnm48+Bj5oCRIw5YuQb9 TUxA== X-Forwarded-Encrypted: i=1; AJvYcCVbNmLwyHFjlcbXrGxLrsUvu7F0Sh6I+9OlFRnGfMKDEZugD6oS9dbiETBjbKOhYsXWci9k6nV39sHyxK7N0YQeOezbMalr1chZHg== X-Gm-Message-State: AOJu0Yy3Yyqk6R6+XgHlBCkOBezVAw/ge2hAeq666yu0GDVPy+Ro0eCQ 6MeNt4cbBNDoP2p3N1zXvGv3EELL1lVmEPrIWK5H1503/lVHMrLbyOB0aYwvDg1VPA9uX00TyeV teBD1aji/cKpoNktEqqYg9RqlIJP6J84b0g6FYR9b0OmrPDJlttU= X-Google-Smtp-Source: AGHT+IHVQGB+D9woyxUPCya6D4kfrLpY2NKm00xAMlIKn+H0Tn9+5J/ZyW1U1NCTt2pEBqpWv2ofgUZBafq7xfP9gW4= X-Received: by 2002:a05:6a20:43a9:b0:1c3:fcbb:5670 with SMTP id adf61e73a8af0-1c3fddcd3c1mr8395552637.48.1721365150523; Thu, 18 Jul 2024 21:59:10 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 References: <01D917DB-7E47-46D7-AD22-AD09C4F89A96@FreeBSD.org> <9CCA0BFB-70D3-4E21-8BA7-3E9F35110E8D@FreeBSD.org> In-Reply-To: <9CCA0BFB-70D3-4E21-8BA7-3E9F35110E8D@FreeBSD.org> From: Warner Losh Date: Thu, 18 Jul 2024 22:58:58 -0600 Message-ID: Subject: Re: nextboot warns it won't reset To: Zhenlei Huang Cc: Gareth de Vaux , FreeBSD-STABLE Mailing List Content-Type: multipart/alternative; boundary="000000000000537e8f061d928e19" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4WQHWz69cCz4pjr --000000000000537e8f061d928e19 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Jul 18, 2024 at 10:49=E2=80=AFPM Zhenlei Huang w= rote: > > > On Jul 19, 2024, at 10:19 AM, Warner Losh wrote: > > > > On Thu, Jul 18, 2024, 7:10=E2=80=AFPM Zhenlei Huang wr= ote: > >> >> >> On Jul 19, 2024, at 4:45 AM, Gareth de Vaux wrote: >> >> Hi all, nextboot warns as follows: >> >> # nextboot -k testkernel >> WARNING: loader(8) has only R/O support for ZFS >> nextboot.conf will NOT be reset in case of kernel boot failure >> >> >> This's on a ZFS zroot 12.4-STABLE system. >> >> >> You're encouraged to upgrade to supported releases ;) >> >> >> Does this mean that nextboot will not do its job? >> >> >> I think the WARNING is a good hint and explain why it does not work >> incase boot failure. >> >> The implementation of load desired kernel is write loader parameters to >> /boot/nexboot.conf , to indicate >> the loader(8) do its job, that is loading the desired kernel on next >> boot. Well it is a oneshot boot >> so the loader will try to disable these parameters by writing `nextboot_= enable=3DNO` >> to /boot/nexboot.conf. >> >> That is the magic. Apparently if loader(8) does not support write to >> filesystem, in this case the ZFS, >> it will fail to write /boot/nexboot.conf in case kernel boot failure. >> Thus subsequent boot the loader(8) would >> still try to boot from the oneshot `testkernel`. >> > > However, ZFS has special support via properties in the BE that lets you d= o > boot once and nextboot stuff. > > > Yes and NO. > > Yes for 13.x, 14.x and CURRENT. nextboot(8) will consults zfsbootcfg(8) t= o > do the nextboot stuff if file system > is ZFS. > > No for 12.x. neither nextboot(8) nor nextboot.sh will do that. And it > seems that loader(8) also do not get / set > ZFS properties to enable / disable nextboot stuff. > I missed that detail and understand your comments about running a supported release. It does indeed not work on 12. Warner > But even better are boot environments. That's a more comple solution that > supports bootnext features and more. See bectl... > > Warner > > I see a blog mentioning that nextboot will write metadata to the ZFS pool >> label, >> in which case maybe the warning is no longer applicable? >> >> >> That is implementation specific. Normally you can ignore the warning, >> unless you have trouble booting >> the kernel. >> >> >> >> Also I suspect I should rather run "nextboot -D" to revert the situation= . >> Is this equivalent to removing /boot/nextboot.conf and/or this metadata >> in the >> ZFS pool label? >> >> >> I think normally you do not need that. >> >> Best regards, >> Zhenlei >> > > > > --000000000000537e8f061d928e19 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Thu, Jul 18, 2024 at 10:49=E2=80= =AFPM Zhenlei Huang <zlei@freebsd.or= g> wrote:

On Jul 19, 2024, at 10:19 AM, Warner Losh <imp@bsdimp.com> wrote:

<= div>

=
On Thu, Jul 18, 2024, 7:10=E2=80=AFPM= Zhenlei Huang <zl= ei@freebsd.org> wrote:

On Jul 19, 2024, at 4:45 AM, Gareth de Vaux <st= able@lordcow.org> wrote:

Hi all, nextboot warns a= s follows:

# nextboot -k testkernel
WARNING: loader(8) has only R= /O support for ZFS
nextboot.conf will NOT be reset in case of kernel boo= t failure

This's on a ZFS zroot 12.4-STABLE system.

You're encouraged to upgrade to= supported releases ;)

Doe= s this mean that nextboot will not do its job?
=

I think the WARNING is a good hint and explain why it d= oes not work incase boot failure.

The implementati= on of load desired kernel is write loader parameters to /boot/nexboot.conf = , to indicate
the loader(8) do its job, that is loading the desir= ed kernel on next boot. Well it is a oneshot boot
so the loader w= ill try to disable these=C2=A0parameters by writing `nextboot_enable=3DNO` to =C2=A0/boot/nexboot.conf.

That is the magic. Appare= ntly if loader(8) does not support write to filesystem, in this case the ZF= S,=C2=A0
it will fail to write=C2=A0/bo= ot/nexboot.conf in case kernel boot failure. Thus subsequent boot the loade= r(8) would
still try to boot from the oneshot `testk= ernel`.
=
However, ZFS has special support via properties= in the BE that lets you do boot once and nextboot stuff.

Yes and NO.

Yes f= or 13.x, 14.x and CURRENT.=C2=A0nextboot(8= ) will=C2=A0consults zfsbootcfg(8)<= /span>=C2=A0to do the nextboot stuff if fi= le system

is ZFS.<= /div>

No for 12.x. =C2=A0neither nextboot(8) nor nextboo= t.sh will do that. And it seems that loader(8) also do not get / set

<= div>ZFS properties to enable / disable nextboot stuff.