From nobody Sat Nov 18 23:23:29 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SXqb14ZCJz50pgt for ; Sat, 18 Nov 2023 23:23:41 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-oo1-xc33.google.com (mail-oo1-xc33.google.com [IPv6:2607:f8b0:4864:20::c33]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SXqb05Pnvz3HPd; Sat, 18 Nov 2023 23:23:40 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=UjpGtsEZ; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::c33 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oo1-xc33.google.com with SMTP id 006d021491bc7-581de3e691dso1726981eaf.3; Sat, 18 Nov 2023 15:23:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700349819; x=1700954619; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=41SuuBuihEFShPon4NZZIddX/lb02DSI37qybqY0N4I=; b=UjpGtsEZrisFwd+swlgYY0Ca+7oDi/OusnA5pllfQb+eY5h8lJwhwRDVTXSoiFFGRu DO4n9YZlM74aY7jxdRUYrxao+VkrWrUhJEllso/3amNbnr8UwPaDQhh4TNOxe/3NImdr 3TaVMqrE8Zr2mWCmuqJKk9nACAVYXy1ukIcqHZH2Z/ecYjmpnNxHUZb4gDD2DdfQZ+PG cuES+M0zSmj4AG+wU0rJCe0uMXnu0rX4CwBKGzlG2ztIr2CgJU0K8BB4xwmntOoSripY bFP5iuBTyPYjpNP+wpzCvBCktuQzfYXGLOgXcTuU0KLjZNf9HekxpjnpvFGKPisCWycH gH0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700349819; x=1700954619; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=41SuuBuihEFShPon4NZZIddX/lb02DSI37qybqY0N4I=; b=f2KyLVN1S9SQuORRr/uozafcixfPxbM6WfFCUkqnAkP8SBHfz42EE8lA2Arb9oRQIQ SnDLY6mlt8XPoErkg6Y+v84RhYQIEGzbkjsuRdu5vFnvdVH40Lg3bdtXdMOMtqL53Lax EnbxucLsDgKFH6BDG9xKHR4iNEsiKEx3HB46moIA22IC+fCQflMmTonihHBIt5hT4boZ Za5j0UYP/5CaflwLPBVju7NlLoJLJSbuLxbs4LAA+vqRvvm+pUmCGiShx72qTU2EUWZy xqoQrFxv9uFhBpcm7GaYAgz7QJjdDZTl0b02yfVw4AMHi5EsM5mHWIgp0OhYjmNamBP0 YU7w== X-Gm-Message-State: AOJu0YxrFwoDY8tV8lDZsLWniqY5lH9IsZ5G0xCCPBCX9MxGJQrrZ8Ct S5RiG4I1Jd3vwJI9qOnk66JJf3HqXUNpBB/e3w== X-Google-Smtp-Source: AGHT+IFJz5cn/Vqc6k1xFF+XfUytGAHgjwj2nfz9sEQWf2zZPJqfftbqJ0CS9HScmU9FexRbE+HBocTxWI4n9v0fwvY= X-Received: by 2002:a05:6871:a011:b0:1f0:811a:324d with SMTP id vp17-20020a056871a01100b001f0811a324dmr4019147oab.51.1700349819305; Sat, 18 Nov 2023 15:23:39 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <25943.60056.880614.452966@hergotha.csail.mit.edu> <91988E23-ED50-4379-AA5F-4B069E08D80F@karels.net> In-Reply-To: <91988E23-ED50-4379-AA5F-4B069E08D80F@karels.net> From: Rick Macklem Date: Sat, 18 Nov 2023 15:23:29 -0800 Message-ID: Subject: Re: NFS exports of ZFS snapshots broken To: Mike Karels , FreeBSD CURRENT Cc: Alexander Motin , Martin Matuska , Garrett Wollman Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.97 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.971]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::c33:from]; DKIM_TRACE(0.00)[gmail.com:+]; TAGGED_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; TO_DN_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4SXqb05Pnvz3HPd X-Spamd-Bar: --- On Sat, Nov 18, 2023 at 2:27=E2=80=AFPM Mike Karels wrote= : > > On 18 Nov 2023, at 15:58, Rick Macklem wrote: > > > On Sat, Nov 18, 2023 at 8:09=E2=80=AFAM Rick Macklem wrote: > >> > >> On Fri, Nov 17, 2023 at 8:19=E2=80=AFPM Mike Karels = wrote: > >>> > >>> On 17 Nov 2023, at 22:14, Mike Karels wrote: > >>> > >>>> On 17 Nov 2023, at 21:24, Rick Macklem wrote: > >>>> > >>>>> Most of the changes in stable/13 that are not in releng/13.2 > >>>>> are the "make it work in a jail" stuff. Unfortunately, they are > >>>>> a large # of changes (mostly trivial edits adding vnet macros), > >>>>> but it also includes export check changes. > >>>>> > >>>>> I have attached a trivial patch that I think disables the export > >>>>> checks for jails. If either of you can try it and see if it fixes > >>>>> the problem, that would be great. > >>>>> (Note that this is only for testing, although it probably does not > >>>>> matter unless you are running nfsd(8) in vnet jails.) > >>>> > >>>> Yes, I can see snapshots with the patch. This system is just a test > >>>> system that doesn't normally run ZFS or NFS, so no problem messing > >>>> with permissions. It's a bhyve VM, so I just added a small disk and > >>>> enabled ZFS for testing. > >>> > >>> btw, you might try to get mm@ or maybe mav@ to help out from the ZFS > >>> side. It must be doing something differently inside a snapshot than > >>> outside, maybe with file handles or something like that. > >> Yes. I've added freebsd-current@ (although Garrett is not on it, he is > >> cc'd) and these guys specifically... > >> > >> So, here's what appears to be the problem... > >> Commit 88175af (in main and stable/13, but not 13.2) added checks for > >> nfsd(8) running in jails by filling in mnt_exjail with a reference to = the cred > >> used when the file system is exported. > >> When mnt_exjail is found NULL, the current nfsd code assumes that ther= e > >> is no access allowed for the mount. > >> > >> My vague understanding is that when a ZFS snapshot is accessed, it is > >> "pseudo-mounted" by zfsctl_snapdir_lookup() and I am guessing that > >> mnt_exjail is NULL as a result. > >> Since I do not know the ZFS code and don't even have an easy way to > >> test this (thankfully Mike can test easily), I do not know what to do = from > >> here? > >> > >> Is there a "struct mount" constructed for this pseudo mount > >> (or it actually appears to be the lookup of ".." that fails, so it > >> might be the parent of the snapshot subdir?)? > >> > >> One thought is that I can check to see if the mount pointer is in the > >> mountlist (I don't think the snapshot's mount is in the mountlist) and > >> avoid the jail test for this case. This would assume that snapshots a= re > >> always within the file system(s) exported via that jail (which include= s > >> the case of prison0, of course), so that they do not need a separate > >> jail check. > >> > >> If this doesn't work, there will need to be some sort of messing about > >> in ZFS to set mnt_exjail for these. > > Ok, so now onto the hard part... > > Thanks to Mike and others, I did create a snapshot under .zfs and I can > > see the problem. It is that mnt_exjail =3D=3D NULL. > > Now, is there a way that this "struct mount" can be recognized as "spec= ial" > > for snapshots, so I can avoid the mnt_exjail =3D=3D NULL test? > > (I had hoped that "mp->mnt_list.tqe_prev" would be NULL, but that is no= t > > the case.) > > Dumb question, is the mount point (mp presumably) different between the > snapshot and the main file system? Not a dump question and the answer is rather interesting... It is "sometimes" or "usually" according to my printf(). It seems that when you first "cd . * * This is where we lie about our v_vfsp in order to * make .zfs/snapshot/ accessible over NFS * without requiring manual mounts of . */ ASSERT3P(VTOZ(*vpp)->z_zfsvfs, !=3D, zfsvfs); VTOZ(*vpp)->z_zfsvfs->z_parent =3D zfsvfs; /* Clear the root flag (set via VFS_ROOT) as well. */ (*vpp)->v_vflag &=3D ~VV_ROOT; which seems to set the mp to that of the parent, but it seems this does not happen for the initial lookup of the ? I'll note that there is code before this in zfsctl_snapdir_lookup() for handling cases like "." and ".." that return without doing this. Now, why does this work without the mnt_exjail check (as in 13.2)? I am not quite sure, but there is this "cheat" in the NFS server (it has been there for years, maybe decades): /* * Allow a Lookup, Getattr, GetFH, Secinfo on an * non-exported directory if * nfs_rootfhset. Do I need to allow any other Ops? * (You can only have a non-exported vpnes if * nfs_rootfhset is true. See nfsd_fhtovp()) * Allow AUTH_SYS to be used for file systems * exported GSS only for certain Ops, to allow * clients to do mounts more easily. */ if (nfsv4_opflag[op].needscfh && vp) { if (!NFSVNO_EXPORTED(&vpnes) && op !=3D NFSV4OP_LOOKUP && op !=3D NFSV4OP_GETATTR && op !=3D NFSV4OP_GETFH && op !=3D NFSV4OP_ACCESS && op !=3D NFSV4OP_READLINK && op !=3D NFSV4OP_SECINFO && op !=3D NFSV4OP_SECINFONONAME) nd->nd_repstat =3D NFSERR_NOFILEHANDLE; This allows certain operations to be done on non-exported file systems and I think that is enough to allow this to work when mnt_exjail is not checked. (Note that NFSV4OP_LOOKUPP is not in the list, which might explain why it is the one that fails for Garrett. I don't think it can be added to this list safely, since that would allow a client to move above the exported file system into "uncharted territory".) > Just curious. Also, what is mnt_exjail > normally set to for file systems not in a jail? mnt_exjail is set to the credentials of the thread/process that exported the file system (usually mountd(8)). When not in a jail, cr_prison for these credentials points to prison0. Btw, I checked and the "other mp that has mnt_exjail =3D=3D NULL is in the mountlist, so the idea of checking "not in mountlist" is a dead end. I am looking for something "unique" about this other mp, but haven't found anything yet. Alternately, it might be necessary to add code to zfsctl_snapdir_lookup() to "cheat and change the mp" in more cases, such as "." and ".." lookups? rick ps: I added all the cc's back in because I want the ZFS folk to hopefully chime in. > > Mike > > > Do I need to search mountlist for it? > > > > rick > > ps: The hack patch attached should fix the problem, but can only be > > safely used if mountd/nfsd are not run in any jails. > > > >> > >> I will try and get a test setup going here, which leads me to.. > >> how do I create a ZFS snapshot? (I do have a simple ZFS pool running > >> on a test machine, but I've never done a snapshot.) > >> > >> Although this problem is not in 13.2, it will have shipped in 14.0. > >> > >> Any help with be appreciated, rick > >> > >>> > >>> Mike > >>>> > >>>>> rick > >>>>> > >>>>> On Fri, Nov 17, 2023 at 6:14=E2=80=AFPM Mike Karels wrote: > >>>>>> > >>>>>> CAUTION: This email originated from outside of the University of G= uelph. Do not click links or open attachments unless you recognize the send= er and know the content is safe. If in doubt, forward suspicious emails to = IThelp@uoguelph.ca. > >>>>>> > >>>>>> > >>>>>> Rick, have you been following this thread on freebsd-stable? I ha= ve been able > >>>>>> to reproduce this using a 13-stable server from Oct 7 and a 15-cur= rent system > >>>>>> that is up to date using NFSv3. I did not reproduce with a 13.2 s= erver. The > >>>>>> client was running 13.2. Any ideas? A full bisect seems fairly p= ainful, but > >>>>>> maybe you have an idea of points to try. Fortunately, these are a= ll test > >>>>>> systems that I can reboot at will. > >>>>>> > >>>>>> Mike > >>>>>> > >>>>>> Forwarded message: > >>>>>> > >>>>>>> From: Garrett Wollman > >>>>>>> To: Mike Karels > >>>>>>> Cc: freebsd-stable@freebsd.org > >>>>>>> Subject: Re: NFS exports of ZFS snapshots broken > >>>>>>> Date: Fri, 17 Nov 2023 17:35:04 -0500 > >>>>>>> > >>>>>>> < said: > >>>>>>> > >>>>>>>> I have not run into this, so I tried it just now. I had no prob= lem. > >>>>>>>> The server is 13.2, fully patched, the client is up-to-date -cur= rent, > >>>>>>>> and the mount is v4. > >>>>>>> > >>>>>>> On my 13.2 client and 13-stable server, I see: > >>>>>>> > >>>>>>> 25034 ls CALL open(0x237d32f9a000,0x120004) > >>>>>>> 25034 ls NAMI "/mnt/tools/.zfs/snapshot/weekly-2023-45" > >>>>>>> 25034 ls RET open 4 > >>>>>>> 25034 ls CALL fcntl(0x4,F_ISUNIONSTACK,0x0) > >>>>>>> 25034 ls RET fcntl 0 > >>>>>>> 25034 ls CALL getdirentries(0x4,0x237d32faa000,0x1000,0x2= 37d32fa7028) > >>>>>>> 25034 ls RET getdirentries -1 errno 5 Input/output error > >>>>>>> 25034 ls CALL close(0x4) > >>>>>>> 25034 ls RET close 0 > >>>>>>> 25034 ls CALL exit(0) > >>>>>>> > >>>>>>> Certainly a libc bug here that getdirentries(2) returning [EIO] > >>>>>>> results in ls(1) returning EXIT_SUCCESS, but the [EIO] error is > >>>>>>> consistent across both FreeBSD and Linux clients. > >>>>>>> > >>>>>>> Looking at this from the RPC side: > >>>>>>> > >>>>>>> (PUTFH, GETATTR, LOOKUP(snapshotname), GETFH, GETATTR) > >>>>>>> [NFS4_OK for all ops] > >>>>>>> (PUTFH, GETATTR) > >>>>>>> [NFS4_OK, NFS4_OK] > >>>>>>> (PUTFH, ACCESS(0x3f), GETATTR) > >>>>>>> [NFS4_OK, NFS4_OK, rights =3D 0x03, NFS4_OK] > >>>>>>> (PUTFH, GETATTR, LOOKUPP, GETFH, GETATTR) > >>>>>>> [NFS4_OK, NFS4_OK, NFS4ERR_NOFILEHANDLE] > >>>>>>> > >>>>>>> and at this point the [EIO] is returned. > >>>>>>> > >>>>>>> It seems that clients always do a LOOKUPP before calling READDIR,= and > >>>>>>> this is failing when the subject file handle is the snapshot. Th= e > >>>>>>> client is perfectly able to *traverse into* the snapshot: if I tr= y to > >>>>>>> list a subdirectory I know exists in the snapshot, the client is = able to > >>>>>>> LOOKUP(dirname) just fine, but LOOKUPP still fails with > >>>>>>> NFS4ERR_NOFILEHANDLE *on the subndirectory*. > >>>>>>> > >>>>>>> -GAWollman > >>>>>> From nobody Sun Nov 19 00:42:57 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SXsLv23PTz50vjf for ; Sun, 19 Nov 2023 00:43:19 +0000 (UTC) (envelope-from mike@karels.net) Received: from mail2.karels.net (mail2.karels.net [3.19.118.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "freebsd", Issuer "freebsd" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SXsLt2K7Kz3NF5; Sun, 19 Nov 2023 00:43:18 +0000 (UTC) (envelope-from mike@karels.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=karels.net header.s=mail2 header.b=MKgphzQw; spf=pass (mx1.freebsd.org: domain of mike@karels.net designates 3.19.118.201 as permitted sender) smtp.mailfrom=mike@karels.net; dmarc=none Received: from mail2.karels.net (localhost [IPv6:0:0:0:0:0:0:0:1]) by mail2.karels.net (8.17.1/8.17.1) with ESMTP id 3AJ0gvpm061541; Sat, 18 Nov 2023 18:42:58 -0600 (CST) (envelope-from mike@karels.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=karels.net; s=mail2; t=1700354578; bh=axqWBZR23vPddu93kuypa8NtJH9uTiHJlQdxOqlepqk=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=MKgphzQwzdllWx71qvkrBQ8IwIgUrhJ6TDO7VYMGjAzxApolQNR7eC9eQD/bVAHco RDAs2O23/wl9fHgwXx2Ibn6RRNvGTeOCCMAxJjgzEQuItuO31G2TJWoiaXnwvbzlIu +qJZPB3TD4mO7NfpwapNj+NFbKcTVD9sh2JjoqbGk/NyiCWBLB0EdLSuCPE9Q50K/9 jCLAEfcFtaWI9Dsz/fikg++9R/k1ggjWrzXanH8MuWq3xFaSD3k9NOy6HE95TVJp4w ZQ9sl8uL8eeLYWCM3NrJtARNCyG+xxkmLtB+izvpcYCR0E2pPrYJZPhSiKWGmWGu0d 6K+y4CpO38NzA== Received: from [10.0.2.130] ([73.62.165.147]) by mail2.karels.net with ESMTPSA id COcbNBFaWWVj8AAAs/W3XQ (envelope-from ); Sat, 18 Nov 2023 18:42:57 -0600 From: Mike Karels To: Rick Macklem Cc: FreeBSD CURRENT , Alexander Motin , Martin Matuska , Garrett Wollman Subject: Re: NFS exports of ZFS snapshots broken Date: Sat, 18 Nov 2023 18:42:57 -0600 X-Mailer: MailMate (1.14r5964) Message-ID: In-Reply-To: References: <25943.60056.880614.452966@hergotha.csail.mit.edu> <91988E23-ED50-4379-AA5F-4B069E08D80F@karels.net> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-2.00 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[karels.net:s=mail2]; R_SPF_ALLOW(-0.20)[+ip4:3.19.118.201]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_TO(0.00)[gmail.com]; ASN(0.00)[asn:16509, ipnet:3.16.0.0/14, country:US]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[karels.net:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[mike]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DMARC_NA(0.00)[karels.net]; TAGGED_RCPT(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4SXsLt2K7Kz3NF5 X-Spamd-Bar: - On 18 Nov 2023, at 17:23, Rick Macklem wrote: > On Sat, Nov 18, 2023 at 2:27=E2=80=AFPM Mike Karels w= rote: >> >> On 18 Nov 2023, at 15:58, Rick Macklem wrote: >> >>> On Sat, Nov 18, 2023 at 8:09=E2=80=AFAM Rick Macklem wrote: >>>> >>>> On Fri, Nov 17, 2023 at 8:19=E2=80=AFPM Mike Karels wrote: >>>>> >>>>> On 17 Nov 2023, at 22:14, Mike Karels wrote: >>>>> >>>>>> On 17 Nov 2023, at 21:24, Rick Macklem wrote: >>>>>> >>>>>>> Most of the changes in stable/13 that are not in releng/13.2 >>>>>>> are the "make it work in a jail" stuff. Unfortunately, they are >>>>>>> a large # of changes (mostly trivial edits adding vnet macros), >>>>>>> but it also includes export check changes. >>>>>>> >>>>>>> I have attached a trivial patch that I think disables the export >>>>>>> checks for jails. If either of you can try it and see if it fixes= >>>>>>> the problem, that would be great. >>>>>>> (Note that this is only for testing, although it probably does no= t >>>>>>> matter unless you are running nfsd(8) in vnet jails.) >>>>>> >>>>>> Yes, I can see snapshots with the patch. This system is just a te= st >>>>>> system that doesn't normally run ZFS or NFS, so no problem messing= >>>>>> with permissions. It's a bhyve VM, so I just added a small disk a= nd >>>>>> enabled ZFS for testing. >>>>> >>>>> btw, you might try to get mm@ or maybe mav@ to help out from the ZF= S >>>>> side. It must be doing something differently inside a snapshot tha= n >>>>> outside, maybe with file handles or something like that. >>>> Yes. I've added freebsd-current@ (although Garrett is not on it, he = is >>>> cc'd) and these guys specifically... >>>> >>>> So, here's what appears to be the problem... >>>> Commit 88175af (in main and stable/13, but not 13.2) added checks fo= r >>>> nfsd(8) running in jails by filling in mnt_exjail with a reference t= o the cred >>>> used when the file system is exported. >>>> When mnt_exjail is found NULL, the current nfsd code assumes that th= ere >>>> is no access allowed for the mount. >>>> >>>> My vague understanding is that when a ZFS snapshot is accessed, it i= s >>>> "pseudo-mounted" by zfsctl_snapdir_lookup() and I am guessing that >>>> mnt_exjail is NULL as a result. >>>> Since I do not know the ZFS code and don't even have an easy way to >>>> test this (thankfully Mike can test easily), I do not know what to d= o from >>>> here? >>>> >>>> Is there a "struct mount" constructed for this pseudo mount >>>> (or it actually appears to be the lookup of ".." that fails, so it >>>> might be the parent of the snapshot subdir?)? >>>> >>>> One thought is that I can check to see if the mount pointer is in th= e >>>> mountlist (I don't think the snapshot's mount is in the mountlist) a= nd >>>> avoid the jail test for this case. This would assume that snapshots= are >>>> always within the file system(s) exported via that jail (which inclu= des >>>> the case of prison0, of course), so that they do not need a separate= >>>> jail check. >>>> >>>> If this doesn't work, there will need to be some sort of messing abo= ut >>>> in ZFS to set mnt_exjail for these. >>> Ok, so now onto the hard part... >>> Thanks to Mike and others, I did create a snapshot under .zfs and I c= an >>> see the problem. It is that mnt_exjail =3D=3D NULL. >>> Now, is there a way that this "struct mount" can be recognized as "sp= ecial" >>> for snapshots, so I can avoid the mnt_exjail =3D=3D NULL test? >>> (I had hoped that "mp->mnt_list.tqe_prev" would be NULL, but that is = not >>> the case.) >> >> Dumb question, is the mount point (mp presumably) different between th= e >> snapshot and the main file system? > Not a dump question and the answer is rather interesting... > It is "sometimes" or "usually" according to my printf(). > It seems that when you first "cd where mnt_exjail =3D=3D NULL.. Then when you look at directories within= the > snapshot, you get the mp of the file system that .zfs exists in, which = does > have mnt_exjail set non-NULL. > > There is this snippet of code in zfsctl_snapdir_lookup(): > /* > * Fix up the root vnode mounted on .zfs/snapshot/. > * > * This is where we lie about our v_vfsp in order to > * make .zfs/snapshot/ accessible over NFS > * without requiring manual mounts of . > */ > ASSERT3P(VTOZ(*vpp)->z_zfsvfs, !=3D, zfsvfs); > VTOZ(*vpp)->z_zfsvfs->z_parent =3D zfsvfs; > > /* Clear the root flag (set via VFS_ROOT) as well. */ > (*vpp)->v_vflag &=3D ~VV_ROOT; > which seems to set the mp to that of the parent, but it > seems this does not happen for the initial lookup of > the ? > > I'll note that there is code before this in > zfsctl_snapdir_lookup() for handling cases > like "." and ".." that return without doing this. > > Now, why does this work without the mnt_exjail > check (as in 13.2)? > I am not quite sure, but there is this "cheat" in the > NFS server (it has been there for years, maybe decades): > /* > * Allow a Lookup, Getattr, GetFH, Secinfo on an > * non-exported directory if > * nfs_rootfhset. Do I need to allow any other Ops? > * (You can only have a non-exported vpnes if > * nfs_rootfhset is true. See nfsd_fhtovp()) > * Allow AUTH_SYS to be used for file systems > * exported GSS only for certain Ops, to allow > * clients to do mounts more easily. > */ > if (nfsv4_opflag[op].needscfh && vp) { > if (!NFSVNO_EXPORTED(&vpnes) && > op !=3D NFSV4OP_LOOKUP && > op !=3D NFSV4OP_GETATTR && > op !=3D NFSV4OP_GETFH && > op !=3D NFSV4OP_ACCESS && > op !=3D NFSV4OP_READLINK && > op !=3D NFSV4OP_SECINFO && > op !=3D NFSV4OP_SECINFONONAME) > nd->nd_repstat =3D NFSERR_NOFILEHANDLE; > This allows certain operations to be done on > non-exported file systems and I think that is enough > to allow this to work when mnt_exjail is not checked. > (Note that NFSV4OP_LOOKUPP is not in the list, > which might explain why it is the one that fails for > Garrett. I don't think it can be added to this list > safely, since that would allow a client to move above > the exported file system into "uncharted territory".) > >> Just curious. Also, what is mnt_exjail >> normally set to for file systems not in a jail? > mnt_exjail is set to the credentials of the thread/process > that exported the file system (usually mountd(8)). > When not in a jail, cr_prison for these credentials > points to prison0. > > Btw, I checked and the "other mp that has mnt_exjail =3D=3D NULL > is in the mountlist, so the idea of checking "not in mountlist" > is a dead end. > > I am looking for something "unique" about this other mp, > but haven't found anything yet. > Alternately, it might be necessary to add code to > zfsctl_snapdir_lookup() to "cheat and change the mp" > in more cases, such as "." and ".." lookups? It seems to me that if ZFS is creating an additional mount structure, it should be responsible for setting it up correctly. That could involve a vfs-level routine to do some of the cloning. In any case, it seems to me that mnt_exjail should be set properly, e.g. by duping the one in the original mount structure. Probably ZFS is the only file system type that would need this added. Mike > rick > ps: I added all the cc's back in because I want the > ZFS folk to hopefully chime in. > >> >> Mike >> >>> Do I need to search mountlist for it? >>> >>> rick >>> ps: The hack patch attached should fix the problem, but can only be >>> safely used if mountd/nfsd are not run in any jails. >>> >>>> >>>> I will try and get a test setup going here, which leads me to.. >>>> how do I create a ZFS snapshot? (I do have a simple ZFS pool running= >>>> on a test machine, but I've never done a snapshot.) >>>> >>>> Although this problem is not in 13.2, it will have shipped in 14.0. >>>> >>>> Any help with be appreciated, rick >>>> >>>>> >>>>> Mike >>>>>> >>>>>>> rick >>>>>>> >>>>>>> On Fri, Nov 17, 2023 at 6:14=E2=80=AFPM Mike Karels wrote: >>>>>>>> >>>>>>>> CAUTION: This email originated from outside of the University of= Guelph. Do not click links or open attachments unless you recognize the = sender and know the content is safe. If in doubt, forward suspicious emai= ls to IThelp@uoguelph.ca. >>>>>>>> >>>>>>>> >>>>>>>> Rick, have you been following this thread on freebsd-stable? I = have been able >>>>>>>> to reproduce this using a 13-stable server from Oct 7 and a 15-c= urrent system >>>>>>>> that is up to date using NFSv3. I did not reproduce with a 13.2= server. The >>>>>>>> client was running 13.2. Any ideas? A full bisect seems fairly= painful, but >>>>>>>> maybe you have an idea of points to try. Fortunately, these are= all test >>>>>>>> systems that I can reboot at will. >>>>>>>> >>>>>>>> Mike >>>>>>>> >>>>>>>> Forwarded message: >>>>>>>> >>>>>>>>> From: Garrett Wollman >>>>>>>>> To: Mike Karels >>>>>>>>> Cc: freebsd-stable@freebsd.org >>>>>>>>> Subject: Re: NFS exports of ZFS snapshots broken >>>>>>>>> Date: Fri, 17 Nov 2023 17:35:04 -0500 >>>>>>>>> >>>>>>>>> < said: >>>>>>>>> >>>>>>>>>> I have not run into this, so I tried it just now. I had no pr= oblem. >>>>>>>>>> The server is 13.2, fully patched, the client is up-to-date -c= urrent, >>>>>>>>>> and the mount is v4. >>>>>>>>> >>>>>>>>> On my 13.2 client and 13-stable server, I see: >>>>>>>>> >>>>>>>>> 25034 ls CALL open(0x237d32f9a000,0x120004) >>>>>>>>> 25034 ls NAMI "/mnt/tools/.zfs/snapshot/weekly-2023-45"= >>>>>>>>> 25034 ls RET open 4 >>>>>>>>> 25034 ls CALL fcntl(0x4,F_ISUNIONSTACK,0x0) >>>>>>>>> 25034 ls RET fcntl 0 >>>>>>>>> 25034 ls CALL getdirentries(0x4,0x237d32faa000,0x1000,0= x237d32fa7028) >>>>>>>>> 25034 ls RET getdirentries -1 errno 5 Input/output err= or >>>>>>>>> 25034 ls CALL close(0x4) >>>>>>>>> 25034 ls RET close 0 >>>>>>>>> 25034 ls CALL exit(0) >>>>>>>>> >>>>>>>>> Certainly a libc bug here that getdirentries(2) returning [EIO]= >>>>>>>>> results in ls(1) returning EXIT_SUCCESS, but the [EIO] error is= >>>>>>>>> consistent across both FreeBSD and Linux clients. >>>>>>>>> >>>>>>>>> Looking at this from the RPC side: >>>>>>>>> >>>>>>>>> (PUTFH, GETATTR, LOOKUP(snapshotname), GETFH, GETATTR) >>>>>>>>> [NFS4_OK for all ops] >>>>>>>>> (PUTFH, GETATTR) >>>>>>>>> [NFS4_OK, NFS4_OK] >>>>>>>>> (PUTFH, ACCESS(0x3f), GETATTR) >>>>>>>>> [NFS4_OK, NFS4_OK, rights =3D 0x03, NFS4_OK] >>>>>>>>> (PUTFH, GETATTR, LOOKUPP, GETFH, GETATTR) >>>>>>>>> [NFS4_OK, NFS4_OK, NFS4ERR_NOFILEHANDLE] >>>>>>>>> >>>>>>>>> and at this point the [EIO] is returned. >>>>>>>>> >>>>>>>>> It seems that clients always do a LOOKUPP before calling READDI= R, and >>>>>>>>> this is failing when the subject file handle is the snapshot. = The >>>>>>>>> client is perfectly able to *traverse into* the snapshot: if I = try to >>>>>>>>> list a subdirectory I know exists in the snapshot, the client i= s able to >>>>>>>>> LOOKUP(dirname) just fine, but LOOKUPP still fails with >>>>>>>>> NFS4ERR_NOFILEHANDLE *on the subndirectory*. >>>>>>>>> >>>>>>>>> -GAWollman >>>>>>>> From nobody Sun Nov 19 00:47:06 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SXsRZ5YGxz50vqH for ; Sun, 19 Nov 2023 00:47:22 +0000 (UTC) (envelope-from mattik@gwsit.com.au) Received: from m4.out4.mxs.au (m4.out4.mxs.au [110.232.143.188]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SXsRW6MYwz3Ptv for ; Sun, 19 Nov 2023 00:47:19 +0000 (UTC) (envelope-from mattik@gwsit.com.au) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gwsit.com.au header.s=default header.b=XxNvN7ou; spf=pass (mx1.freebsd.org: domain of mattik@gwsit.com.au designates 110.232.143.188 as permitted sender) smtp.mailfrom=mattik@gwsit.com.au; dmarc=pass (policy=none) header.from=gwsit.com.au Received: from s02ad.syd2.hostingplatform.net.au (s02ad.syd2.hostingplatform.net.au [103.27.32.38]) by out4.mxs.au (Halon) with ESMTPS (TLSv1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 id 29f4fb15-8675-11ee-9cb4-00163c87da3f; Sun, 19 Nov 2023 11:47:06 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gwsit.com.au; s=default; h=Content-Transfer-Encoding:Content-Type: MIME-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date:Sender :Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=grCF1HPwtiqcnIkhj8lmUasJ3gWidjvhNlegTiaHK68=; b=XxNvN7ouUFAs3WHJEPhxJJ0bDM 5uHEEKquyB6NZnqwxBayrRnsHo/qiOZ3cagrHu+p9xMxYdNFy/Rk71qaY2FCSe2SOfsG/Pn37wikR m7VtByDLS2miN/NGPkt4yolyFKxFuQmg1JszR4I1ESfvxJW6TtxjHx27nrJmznhzgMijChJgc9Ext MSK5cRz0rhoImlXR4kIbt40cfdBwbCCaZvkr0NB6mZ2CdawqeS71aTsCr0xmSQYRNWC9OwV6bngFt UzishjzYSN+3k7JJd3/VQbzskc5gNEkcSDIfoSeD7SvOVuv5YGUc3lmATwVLnEY6bG9/yBgTOs99j iA4pzMpg==; Received: from 180-150-31-87.b4961f.syd.static.aussiebb.net ([180.150.31.87]:52134 helo=ws1.wobblyboot.net) by s02ad.syd2.hostingplatform.net.au with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96.2) (envelope-from ) id 1r4Vxq-001DZ7-2S; Sun, 19 Nov 2023 11:47:06 +1100 Date: Sun, 19 Nov 2023 11:47:06 +1100 From: matti k To: Glen Barber Cc: freebsd-current@freebsd.org, FreeBSD Release Engineering Team Subject: Re: Update to the Release Engineering Team Roster Message-ID: <20231119114706.6afa02ea@ws1.wobblyboot.net> In-Reply-To: <20231117235240.GU1307@FreeBSD.org> References: <20231117235240.GU1307@FreeBSD.org> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd13.2) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - s02ad.syd2.hostingplatform.net.au X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - gwsit.com.au X-Get-Message-Sender-Via: s02ad.syd2.hostingplatform.net.au: authenticated_id: mattik@gwsit.com.au X-Authenticated-Sender: s02ad.syd2.hostingplatform.net.au: mattik@gwsit.com.au X-Source: X-Source-Args: X-Source-Dir: X-Spamd-Result: default: False [-3.97 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.97)[-0.969]; DMARC_POLICY_ALLOW(-0.50)[gwsit.com.au,none]; R_SPF_ALLOW(-0.20)[+ip4:110.232.143.0/24]; R_DKIM_ALLOW(-0.20)[gwsit.com.au:s=default]; MIME_GOOD(-0.10)[text/plain]; HAS_X_GMSV(0.00)[mattik@gwsit.com.au]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; HAS_X_ANTIABUSE(0.00)[]; ASN(0.00)[asn:45638, ipnet:110.232.143.0/24, country:AU]; MIME_TRACE(0.00)[0:+]; DKIM_TRACE(0.00)[gwsit.com.au:+]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; HAS_X_SOURCE(0.00)[]; HAS_X_AS(0.00)[mattik@gwsit.com.au] X-Rspamd-Queue-Id: 4SXsRW6MYwz3Ptv X-Spamd-Bar: --- On Fri, 17 Nov 2023 23:52:40 +0000 Glen Barber wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > FreeBSD community, > > It is with a heavy heart that I have decided to resign my role as the > Release Engineering Team Lead. > > Colin Percival will take over this role, and I will act as his Deputy > Lead unless and until he finds a more suitable person to fill this > role. > > Regardless, I do not have any immediate plans to resign from the RE > Team as a whole, and though Colin has proved during the 13.2-RELEASE > cycle that he is fully capable of overtaking this role, I will remain > active on the team and will be available to him to provide any > guidance or insights as needed. > > I personally want to thank Colin for accepting this role change. I > have worked with Colin for several years directly, and indirectly, > much longer than I can remember. > > As I am sure many of you all are aware, Colin served the Project as > the Security Officer for several years, and authored important > utilities used by many within the community, such as portsnap and > freebsd-update. > > I have sent and received acknowledgement from the Core Team of this > decision to step aside, which was well received and no objection to me > selecting Colin to be my successor had been posed. > > Colin, thank you, again, for overtaking this role, and as we > discussed, I will be here to provide any needed guidance, insight, or > information as you need. As I had mentioned to you, I do not have > any immediate plans to leave the Team, nor do I have any question in > my mind that you are the obvious successor to this role. > > I am proud to be able to hand over the proverbial torch to you, and > look very much forward to your successes as the Release Engineering > Team Lead. > > It has been an absolute pleasure to be able to serve you all to the > best of my abilities over the past several years. > > With my very best regards to all involved in the Project, and nothing > but the best intentions of the Project in mind. > > Glen > > -----BEGIN PGP SIGNATURE----- > > iQIzBAEBCAAdFiEEjRJAPC5sqwhs9k2jAxRYpUeP4pMFAmVX/MgACgkQAxRYpUeP > 4pPLVg/9HW6nhw6EXTGqkrdibCkThurdasCqftkLSGrsABDj1AsErHbBnbO3W04G > KSVZ6Jw28JQNGUsprJwcAsJre8PXgL2iKOkIZTL31CYSBvQbfGEzIWTC+18oTQ6Q > d1uxqGcqoMcpPfFxSry8wnfxO7hYGabv38lPy2KipskCqq/3zVqsdVb6UxGdo+5Q > 0sis6OVh8KhNdcgOu+RTZr194KJogsdqqGll4pOQNMv/Qpy5e9JOqHgVXueseUC8 > 6GDHa990VLF22+DLgq3qzVS0xGLIOM5MuC0EjWJEjaq5bANEzQX0SvMEKQEbGLrI > MjBXGoJGTreBMr0VbCewhuDIiYy7DU5dtohvXvCx+UoBryT9qU1RKmRyRxxp3hQ4 > dcZUyZ3RarwmZfJBTs7ib2s9LztUlpyIRAU7gJ2vRKcegVTH/Fe1FuRZHOFHZf46 > kzHeIygER3nitPtJj9ePQgq97F2WBjRDZ5MSzwNWsOMknsbUKekIB9xshbhxlUw8 > 5ZN61k3frKM/dOcnLulsNHFaMGJaodDsxMuTKhJCEHvHWLXOin66WrfuHKbeqzVf > LDgMCWiHcM0GwrtzXR/aITVZmwyg096gZw0NLXtlNadY+PyQrg8R83DmOKTk0fWo > MYaMnWBcZM78DviTaBPhcblMtB5/IqrbJzqnJqsnCBEW2Z29qgs= > =sYrP > -----END PGP SIGNATURE----- > Hi Glen I find myself at a time where I could possibly lend a hand, would someone like me be able to help with re@freebsd.org? Teach and I can learn I guess ... I tend to excel but totally unsure if I can fit the bill! Regards Matti From nobody Sun Nov 19 03:19:04 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SXwpr1c5jz516Pk for ; Sun, 19 Nov 2023 03:19:16 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-oa1-x2f.google.com (mail-oa1-x2f.google.com [IPv6:2001:4860:4864:20::2f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SXwpq6kz2z3d9P; Sun, 19 Nov 2023 03:19:15 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oa1-x2f.google.com with SMTP id 586e51a60fabf-1f4e17c1edfso1862309fac.0; Sat, 18 Nov 2023 19:19:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700363954; x=1700968754; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=gMHpTv6+zG90s5Z8AXWgF41S0GXE6LLnxdRW2wViCvQ=; b=QeGt0iynIh71vu2gvLdDBeA/PWrbHsukio8uHPj3qzOffae2kruxOHew2F0sU3Dz1z VZYatL2eajcyLwh8Qx5Cu1PMoVQbdWGGx12DoRHLjpyohDmjeLmHoEzGTWhbTLBE53qE Z4VDw0QCYFUdF0S9pQjEwNFJoOIQTjfSFXBACe9hh+gVC4I9Pz8NMGJDY4rC0+8ga1UU 0PASUT8Ea2+0wkkpqvBgDWM8kPGPUC/91L7x0xEciP1ygD2cTc5C6ONS0LG+A6/k2/GV osoN9YyD0vqijJkH0Djck59ONCOsfTdWBYLFUlGdI8k2TLWsL2HjMhZKAuxmjhFqnEic yaLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700363954; x=1700968754; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=gMHpTv6+zG90s5Z8AXWgF41S0GXE6LLnxdRW2wViCvQ=; b=OgoWQXpmpzyir0EpA4OBZiKeXgOP4Orr5cAV1sR8XDRqAxA8WOjZKEvWsLzHNqkxPp f3mEEbAwggClYzdAhlnQz54iJAmoAFXG9x2yqv7AnmOUiBTSWpwXLxEp+7NhcLB8Eezg Iu1WUjQq37icirtiOomf/Ecf7wLf/bWBemHTaTVeMDoiuHaHc5QHjgD0TQopTxUjmf/t /M1XvsBjvpgCHEsJNoFPOlRRGpPb35c9SksC9i+57zCic4iX0HN7fESJQtl26A6Cv6Yc 3ETjjt15Ugt/NiWDru29hXgKIHURWGpRD0F4duc56NDgINm4xfiXSZBgh6ClDtaN2B+9 fY/g== X-Gm-Message-State: AOJu0Yy66dY/m7CJ8818GQKMAsD3aurmTVEKwC9xBhFL9j+LaE8kG0bC c/UTbiQyHejvYjIbB1rfkQ1mj6gBslKbePHM6Q== X-Google-Smtp-Source: AGHT+IG3pmzdtaYvbEguRdDsBWq6cre1PZVar9UQ+uMNYWE8zkGTSOYFBmE1JnPu3FnmdPaXaW0cJGHWLROSay4PPgY= X-Received: by 2002:a05:6871:878a:b0:1e9:c28f:45b9 with SMTP id td10-20020a056871878a00b001e9c28f45b9mr4421795oab.19.1700363953576; Sat, 18 Nov 2023 19:19:13 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <25943.60056.880614.452966@hergotha.csail.mit.edu> <91988E23-ED50-4379-AA5F-4B069E08D80F@karels.net> In-Reply-To: From: Rick Macklem Date: Sat, 18 Nov 2023 19:19:04 -0800 Message-ID: Subject: Re: NFS exports of ZFS snapshots broken To: Mike Karels Cc: FreeBSD CURRENT , Alexander Motin , Martin Matuska , Garrett Wollman Content-Type: multipart/mixed; boundary="00000000000071e6ce060a78d585" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Queue-Id: 4SXwpq6kz2z3d9P --00000000000071e6ce060a78d585 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Nov 18, 2023 at 4:43=E2=80=AFPM Mike Karels wrote= : > > On 18 Nov 2023, at 17:23, Rick Macklem wrote: > > > On Sat, Nov 18, 2023 at 2:27=E2=80=AFPM Mike Karels w= rote: > >> > >> On 18 Nov 2023, at 15:58, Rick Macklem wrote: > >> > >>> On Sat, Nov 18, 2023 at 8:09=E2=80=AFAM Rick Macklem wrote: > >>>> > >>>> On Fri, Nov 17, 2023 at 8:19=E2=80=AFPM Mike Karels wrote: > >>>>> > >>>>> On 17 Nov 2023, at 22:14, Mike Karels wrote: > >>>>> > >>>>>> On 17 Nov 2023, at 21:24, Rick Macklem wrote: > >>>>>> > >>>>>>> Most of the changes in stable/13 that are not in releng/13.2 > >>>>>>> are the "make it work in a jail" stuff. Unfortunately, they are > >>>>>>> a large # of changes (mostly trivial edits adding vnet macros), > >>>>>>> but it also includes export check changes. > >>>>>>> > >>>>>>> I have attached a trivial patch that I think disables the export > >>>>>>> checks for jails. If either of you can try it and see if it fixes > >>>>>>> the problem, that would be great. > >>>>>>> (Note that this is only for testing, although it probably does no= t > >>>>>>> matter unless you are running nfsd(8) in vnet jails.) > >>>>>> > >>>>>> Yes, I can see snapshots with the patch. This system is just a te= st > >>>>>> system that doesn't normally run ZFS or NFS, so no problem messing > >>>>>> with permissions. It's a bhyve VM, so I just added a small disk a= nd > >>>>>> enabled ZFS for testing. > >>>>> > >>>>> btw, you might try to get mm@ or maybe mav@ to help out from the ZF= S > >>>>> side. It must be doing something differently inside a snapshot tha= n > >>>>> outside, maybe with file handles or something like that. > >>>> Yes. I've added freebsd-current@ (although Garrett is not on it, he = is > >>>> cc'd) and these guys specifically... > >>>> > >>>> So, here's what appears to be the problem... > >>>> Commit 88175af (in main and stable/13, but not 13.2) added checks fo= r > >>>> nfsd(8) running in jails by filling in mnt_exjail with a reference t= o the cred > >>>> used when the file system is exported. > >>>> When mnt_exjail is found NULL, the current nfsd code assumes that th= ere > >>>> is no access allowed for the mount. > >>>> > >>>> My vague understanding is that when a ZFS snapshot is accessed, it i= s > >>>> "pseudo-mounted" by zfsctl_snapdir_lookup() and I am guessing that > >>>> mnt_exjail is NULL as a result. > >>>> Since I do not know the ZFS code and don't even have an easy way to > >>>> test this (thankfully Mike can test easily), I do not know what to d= o from > >>>> here? > >>>> > >>>> Is there a "struct mount" constructed for this pseudo mount > >>>> (or it actually appears to be the lookup of ".." that fails, so it > >>>> might be the parent of the snapshot subdir?)? > >>>> > >>>> One thought is that I can check to see if the mount pointer is in th= e > >>>> mountlist (I don't think the snapshot's mount is in the mountlist) a= nd > >>>> avoid the jail test for this case. This would assume that snapshots= are > >>>> always within the file system(s) exported via that jail (which inclu= des > >>>> the case of prison0, of course), so that they do not need a separate > >>>> jail check. > >>>> > >>>> If this doesn't work, there will need to be some sort of messing abo= ut > >>>> in ZFS to set mnt_exjail for these. > >>> Ok, so now onto the hard part... > >>> Thanks to Mike and others, I did create a snapshot under .zfs and I c= an > >>> see the problem. It is that mnt_exjail =3D=3D NULL. > >>> Now, is there a way that this "struct mount" can be recognized as "sp= ecial" > >>> for snapshots, so I can avoid the mnt_exjail =3D=3D NULL test? > >>> (I had hoped that "mp->mnt_list.tqe_prev" would be NULL, but that is = not > >>> the case.) > >> > >> Dumb question, is the mount point (mp presumably) different between th= e > >> snapshot and the main file system? > > Not a dump question and the answer is rather interesting... > > It is "sometimes" or "usually" according to my printf(). > > It seems that when you first "cd > where mnt_exjail =3D=3D NULL.. Then when you look at directories within= the > > snapshot, you get the mp of the file system that .zfs exists in, which = does > > have mnt_exjail set non-NULL. > > > > There is this snippet of code in zfsctl_snapdir_lookup(): > > /* > > * Fix up the root vnode mounted on .zfs/snapshot/. > > * > > * This is where we lie about our v_vfsp in order to > > * make .zfs/snapshot/ accessible over NFS > > * without requiring manual mounts of . > > */ > > ASSERT3P(VTOZ(*vpp)->z_zfsvfs, !=3D, zfsvfs); > > VTOZ(*vpp)->z_zfsvfs->z_parent =3D zfsvfs; > > > > /* Clear the root flag (set via VFS_ROOT) as well. */ > > (*vpp)->v_vflag &=3D ~VV_ROOT; > > which seems to set the mp to that of the parent, but it > > seems this does not happen for the initial lookup of > > the ? > > > > I'll note that there is code before this in > > zfsctl_snapdir_lookup() for handling cases > > like "." and ".." that return without doing this. > > > > Now, why does this work without the mnt_exjail > > check (as in 13.2)? > > I am not quite sure, but there is this "cheat" in the > > NFS server (it has been there for years, maybe decades): > > /* > > * Allow a Lookup, Getattr, GetFH, Secinfo on an > > * non-exported directory if > > * nfs_rootfhset. Do I need to allow any other Ops? > > * (You can only have a non-exported vpnes if > > * nfs_rootfhset is true. See nfsd_fhtovp()) > > * Allow AUTH_SYS to be used for file systems > > * exported GSS only for certain Ops, to allow > > * clients to do mounts more easily. > > */ > > if (nfsv4_opflag[op].needscfh && vp) { > > if (!NFSVNO_EXPORTED(&vpnes) && > > op !=3D NFSV4OP_LOOKUP && > > op !=3D NFSV4OP_GETATTR && > > op !=3D NFSV4OP_GETFH && > > op !=3D NFSV4OP_ACCESS && > > op !=3D NFSV4OP_READLINK && > > op !=3D NFSV4OP_SECINFO && > > op !=3D NFSV4OP_SECINFONONAME) > > nd->nd_repstat =3D NFSERR_NOFILEHANDLE; > > This allows certain operations to be done on > > non-exported file systems and I think that is enough > > to allow this to work when mnt_exjail is not checked. > > (Note that NFSV4OP_LOOKUPP is not in the list, > > which might explain why it is the one that fails for > > Garrett. I don't think it can be added to this list > > safely, since that would allow a client to move above > > the exported file system into "uncharted territory".) > > > >> Just curious. Also, what is mnt_exjail > >> normally set to for file systems not in a jail? > > mnt_exjail is set to the credentials of the thread/process > > that exported the file system (usually mountd(8)). > > When not in a jail, cr_prison for these credentials > > points to prison0. > > > > Btw, I checked and the "other mp that has mnt_exjail =3D=3D NULL > > is in the mountlist, so the idea of checking "not in mountlist" > > is a dead end. > > > > I am looking for something "unique" about this other mp, > > but haven't found anything yet. > > Alternately, it might be necessary to add code to > > zfsctl_snapdir_lookup() to "cheat and change the mp" > > in more cases, such as "." and ".." lookups? > > It seems to me that if ZFS is creating an additional mount structure, > it should be responsible for setting it up correctly. That could > involve a vfs-level routine to do some of the cloning. In any case, > it seems to me that mnt_exjail should be set properly, e.g. by duping > the one in the original mount structure. Probably ZFS is the only > file system type that would need this added. I've created a patch that I think does this. It seemed to test ok for my ca= se. It's in D42672 on reviews.freebsd.org. I have also attached it here (this diff doesn't use -U999999, so it might b= e easier to apply). Hopefully this fixes the problem. Sorry for the breakage. rick > > Mike > > > rick > > ps: I added all the cc's back in because I want the > > ZFS folk to hopefully chime in. > > > >> > >> Mike > >> > >>> Do I need to search mountlist for it? > >>> > >>> rick > >>> ps: The hack patch attached should fix the problem, but can only be > >>> safely used if mountd/nfsd are not run in any jails. > >>> > >>>> > >>>> I will try and get a test setup going here, which leads me to.. > >>>> how do I create a ZFS snapshot? (I do have a simple ZFS pool running > >>>> on a test machine, but I've never done a snapshot.) > >>>> > >>>> Although this problem is not in 13.2, it will have shipped in 14.0. > >>>> > >>>> Any help with be appreciated, rick > >>>> > >>>>> > >>>>> Mike > >>>>>> > >>>>>>> rick > >>>>>>> > >>>>>>> On Fri, Nov 17, 2023 at 6:14=E2=80=AFPM Mike Karels wrote: > >>>>>>>> > >>>>>>>> CAUTION: This email originated from outside of the University of= Guelph. Do not click links or open attachments unless you recognize the se= nder and know the content is safe. If in doubt, forward suspicious emails t= o IThelp@uoguelph.ca. > >>>>>>>> > >>>>>>>> > >>>>>>>> Rick, have you been following this thread on freebsd-stable? I = have been able > >>>>>>>> to reproduce this using a 13-stable server from Oct 7 and a 15-c= urrent system > >>>>>>>> that is up to date using NFSv3. I did not reproduce with a 13.2= server. The > >>>>>>>> client was running 13.2. Any ideas? A full bisect seems fairly= painful, but > >>>>>>>> maybe you have an idea of points to try. Fortunately, these are= all test > >>>>>>>> systems that I can reboot at will. > >>>>>>>> > >>>>>>>> Mike > >>>>>>>> > >>>>>>>> Forwarded message: > >>>>>>>> > >>>>>>>>> From: Garrett Wollman > >>>>>>>>> To: Mike Karels > >>>>>>>>> Cc: freebsd-stable@freebsd.org > >>>>>>>>> Subject: Re: NFS exports of ZFS snapshots broken > >>>>>>>>> Date: Fri, 17 Nov 2023 17:35:04 -0500 > >>>>>>>>> > >>>>>>>>> < said: > >>>>>>>>> > >>>>>>>>>> I have not run into this, so I tried it just now. I had no pr= oblem. > >>>>>>>>>> The server is 13.2, fully patched, the client is up-to-date -c= urrent, > >>>>>>>>>> and the mount is v4. > >>>>>>>>> > >>>>>>>>> On my 13.2 client and 13-stable server, I see: > >>>>>>>>> > >>>>>>>>> 25034 ls CALL open(0x237d32f9a000,0x120004) > >>>>>>>>> 25034 ls NAMI "/mnt/tools/.zfs/snapshot/weekly-2023-45" > >>>>>>>>> 25034 ls RET open 4 > >>>>>>>>> 25034 ls CALL fcntl(0x4,F_ISUNIONSTACK,0x0) > >>>>>>>>> 25034 ls RET fcntl 0 > >>>>>>>>> 25034 ls CALL getdirentries(0x4,0x237d32faa000,0x1000,0= x237d32fa7028) > >>>>>>>>> 25034 ls RET getdirentries -1 errno 5 Input/output err= or > >>>>>>>>> 25034 ls CALL close(0x4) > >>>>>>>>> 25034 ls RET close 0 > >>>>>>>>> 25034 ls CALL exit(0) > >>>>>>>>> > >>>>>>>>> Certainly a libc bug here that getdirentries(2) returning [EIO] > >>>>>>>>> results in ls(1) returning EXIT_SUCCESS, but the [EIO] error is > >>>>>>>>> consistent across both FreeBSD and Linux clients. > >>>>>>>>> > >>>>>>>>> Looking at this from the RPC side: > >>>>>>>>> > >>>>>>>>> (PUTFH, GETATTR, LOOKUP(snapshotname), GETFH, GETATTR) > >>>>>>>>> [NFS4_OK for all ops] > >>>>>>>>> (PUTFH, GETATTR) > >>>>>>>>> [NFS4_OK, NFS4_OK] > >>>>>>>>> (PUTFH, ACCESS(0x3f), GETATTR) > >>>>>>>>> [NFS4_OK, NFS4_OK, rights =3D 0x03, NFS4_OK] > >>>>>>>>> (PUTFH, GETATTR, LOOKUPP, GETFH, GETATTR) > >>>>>>>>> [NFS4_OK, NFS4_OK, NFS4ERR_NOFILEHANDLE] > >>>>>>>>> > >>>>>>>>> and at this point the [EIO] is returned. > >>>>>>>>> > >>>>>>>>> It seems that clients always do a LOOKUPP before calling READDI= R, and > >>>>>>>>> this is failing when the subject file handle is the snapshot. = The > >>>>>>>>> client is perfectly able to *traverse into* the snapshot: if I = try to > >>>>>>>>> list a subdirectory I know exists in the snapshot, the client i= s able to > >>>>>>>>> LOOKUP(dirname) just fine, but LOOKUPP still fails with > >>>>>>>>> NFS4ERR_NOFILEHANDLE *on the subndirectory*. > >>>>>>>>> > >>>>>>>>> -GAWollman > >>>>>>>> --00000000000071e6ce060a78d585 Content-Type: application/octet-stream; name="zfssnap.patch" Content-Disposition: attachment; filename="zfssnap.patch" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_lp4wse5y0 LS0tIHN5cy9jb250cmliL29wZW56ZnMvaW5jbHVkZS9vcy9mcmVlYnNkL3NwbC9zeXMvdmZzLmgu emZzc25hcAkyMDIzLTExLTE4IDE3OjMyOjAwLjQ0OTYxODAwMCAtMDgwMAorKysgc3lzL2NvbnRy aWIvb3Blbnpmcy9pbmNsdWRlL29zL2ZyZWVic2Qvc3BsL3N5cy92ZnMuaAkyMDIzLTExLTE4IDE3 OjMxOjQ0LjUzOTcwNTAwMCAtMDgwMApAQCAtMTAxLDcgKzEwMSw3IEBAIHZvaWQgdmZzX3NldG1u dG9wdCh2ZnNfdCAqdmZzcCwgY29uc3QgY2hhciAqbmFtZSwgY29ucwogdm9pZCB2ZnNfY2xlYXJt bnRvcHQodmZzX3QgKnZmc3AsIGNvbnN0IGNoYXIgKm5hbWUpOwogaW50IHZmc19vcHRpb25pc3Nl dChjb25zdCB2ZnNfdCAqdmZzcCwgY29uc3QgY2hhciAqb3B0LCBjaGFyICoqYXJncCk7CiBpbnQg bW91bnRfc25hcHNob3Qoa3RocmVhZF90ICp0ZCwgdm5vZGVfdCAqKnZwcCwgY29uc3QgY2hhciAq ZnN0eXBlLAotICAgIGNoYXIgKmZzcGF0aCwgY2hhciAqZnNwZWMsIGludCBmc2ZsYWdzKTsKKyAg ICBjaGFyICpmc3BhdGgsIGNoYXIgKmZzcGVjLCBpbnQgZnNmbGFncywgdmZzX3QgKnZmc3ApOwog CiB0eXBlZGVmCXVpbnQ2NF90CXZmc19mZWF0dXJlX3Q7CiAKLS0tIHN5cy9jb250cmliL29wZW56 ZnMvbW9kdWxlL29zL2ZyZWVic2Qvc3BsL3NwbF92ZnMuYy56ZnNzbmFwCTIwMjMtMTEtMTggMTY6 NDQ6MzUuMjkyNjU1MDAwIC0wODAwCisrKyBzeXMvY29udHJpYi9vcGVuemZzL21vZHVsZS9vcy9m cmVlYnNkL3NwbC9zcGxfdmZzLmMJMjAyMy0xMS0xOCAxODozMDowNS40MzgyMTgwMDAgLTA4MDAK QEAgLTEyMCw3ICsxMjAsNyBAQCB2ZnNfb3B0aW9uaXNzZXQoY29uc3QgdmZzX3QgKnZmc3AsIGNv bnN0IGNoYXIgKm9wdCwgY2gKIAogaW50CiBtb3VudF9zbmFwc2hvdChrdGhyZWFkX3QgKnRkLCB2 bm9kZV90ICoqdnBwLCBjb25zdCBjaGFyICpmc3R5cGUsIGNoYXIgKmZzcGF0aCwKLSAgICBjaGFy ICpmc3BlYywgaW50IGZzZmxhZ3MpCisgICAgY2hhciAqZnNwZWMsIGludCBmc2ZsYWdzLCB2ZnNf dCAqcGFyZW50X3Zmc3ApCiB7CiAJc3RydWN0IHZmc2NvbmYgKnZmc3A7CiAJc3RydWN0IG1vdW50 ICptcDsKQEAgLTIxOSw2ICsyMTksMTIgQEAgbW91bnRfc25hcHNob3Qoa3RocmVhZF90ICp0ZCwg dm5vZGVfdCAqKnZwcCwgY29uc3QgY2hhCiAJCXZmc19mcmVlb3B0cyhtcC0+bW50X29wdCk7CiAJ bXAtPm1udF9vcHQgPSBtcC0+bW50X29wdG5ldzsKIAkodm9pZCkgVkZTX1NUQVRGUyhtcCwgJm1w LT5tbnRfc3RhdCk7CisKKwkvKgorCSAqIElmIHRoZSBwYXJlbnQgbW91bnQgaGFzIG1udF9leGph aWwgc2V0LCBzZXQgbW50X2V4amFpbCB0byB0aGUKKwkgKiBzYW1lIGNyZWRlbnRpYWxzLgorCSAq LworCXZmc19leGphaWxfY2xvbmUocGFyZW50X3Zmc3AsIG1wKTsKIAogCS8qCiAJICogUHJldmVu dCBleHRlcm5hbCBjb25zdW1lcnMgb2YgbW91bnQgb3B0aW9ucyBmcm9tIHJlYWRpbmcKLS0tIHN5 cy9jb250cmliL29wZW56ZnMvbW9kdWxlL29zL2ZyZWVic2QvemZzL3pmc19jdGxkaXIuYy56ZnNz bmFwCTIwMjMtMTEtMTggMTg6MDE6NTMuNjYxNjgzMDAwIC0wODAwCisrKyBzeXMvY29udHJpYi9v cGVuemZzL21vZHVsZS9vcy9mcmVlYnNkL3pmcy96ZnNfY3RsZGlyLmMJMjAyMy0xMS0xOCAxODow Mjo0OC41MDkzNTYwMDAgLTA4MDAKQEAgLTEwMjYsNyArMTAyNiw4IEBAIHpmc2N0bF9zbmFwZGly X2xvb2t1cChzdHJ1Y3Qgdm9wX2xvb2t1cF9hcmdzICphcCkKIAkgICAgIiVzLyIgWkZTX0NUTERJ Ul9OQU1FICIvc25hcHNob3QvJXMiLAogCSAgICBkdnAtPnZfdmZzcC0+bW50X3N0YXQuZl9tbnRv bm5hbWUsIG5hbWUpOwogCi0JZXJyID0gbW91bnRfc25hcHNob3QoY3VydGhyZWFkLCB2cHAsICJ6 ZnMiLCBtb3VudHBvaW50LCBmdWxsbmFtZSwgMCk7CisJZXJyID0gbW91bnRfc25hcHNob3QoY3Vy dGhyZWFkLCB2cHAsICJ6ZnMiLCBtb3VudHBvaW50LCBmdWxsbmFtZSwgMCwKKwkgICAgZHZwLT52 X3Zmc3ApOwogCWttZW1fZnJlZShtb3VudHBvaW50LCBtb3VudHBvaW50X2xlbik7CiAJaWYgKGVy ciA9PSAwKSB7CiAJCS8qCi0tLSBzeXMva2Vybi92ZnNfbW91bnQuYy56ZnNzbmFwCTIwMjMtMTEt MTggMTc6Mzc6MjIuOTc2NTQ0MDAwIC0wODAwCisrKyBzeXMva2Vybi92ZnNfbW91bnQuYwkyMDIz LTExLTE4IDE4OjUwOjI3LjA5ODI1NDAwMCAtMDgwMApAQCAtMzE0MSwzICszMTQxLDI5IEBAIHJl c3VtZV9hbGxfZnModm9pZCkKIAl9CiAJbXR4X3VubG9jaygmbW91bnRsaXN0X210eCk7CiB9CisK Ky8qCisgKiBDbG9uZSB0aGUgbW50X2V4amFpbCBmaWVsZCB0byBhIG5ldyBtb3VudCBwb2ludC4K KyAqLwordm9pZAordmZzX2V4amFpbF9jbG9uZShzdHJ1Y3QgbW91bnQgKmlubXAsIHN0cnVjdCBt b3VudCAqb3V0bXApCit7CisJc3RydWN0IHVjcmVkICpjcjsKKworCU1OVF9JTE9DSyhpbm1wKTsK KwljciA9IGlubXAtPm1udF9leGphaWw7CisJaWYgKGNyICE9IE5VTEwpIHsKKwkJY3Job2xkKGNy KTsKKwkJTU5UX0lVTkxPQ0soaW5tcCk7CisJCU1OVF9JTE9DSyhvdXRtcCk7CisJCWlmIChvdXRt cC0+bW50X2V4amFpbCA9PSBOVUxMKSB7CisJCQlvdXRtcC0+bW50X2V4amFpbCA9IGNyOworCQkJ YXRvbWljX2FkZF9pbnQoJmNyLT5jcl9wcmlzb24tPnByX2V4cG9ydGNudCwgMSk7CisJCQljciA9 IE5VTEw7CisJCX0KKwkJTU5UX0lVTkxPQ0sob3V0bXApOworCQlpZiAoY3IgIT0gTlVMTCkKKwkJ CWNyZnJlZShjcik7CisJfSBlbHNlCisJCU1OVF9JVU5MT0NLKGlubXApOworfQotLS0gc3lzL3N5 cy9tb3VudC5oLnpmc3NuYXAJMjAyMy0xMS0xOCAxNjo0MDoyOC4yMjg4NTkwMDAgLTA4MDAKKysr IHN5cy9zeXMvbW91bnQuaAkyMDIzLTExLTE4IDE4OjA2OjM0LjI1MzM1NDAwMCAtMDgwMApAQCAt MTAxNyw2ICsxMDE3LDcgQEAgaW50CXZmc19zZXRwdWJsaWNmcwkJCSAgICAvKiBzZXQgcHVibGlj bHkgZXhwb3J0ZWQgZnMgCiAJICAgIChzdHJ1Y3QgbW91bnQgKiwgc3RydWN0IG5ldGV4cG9ydCAq LCBzdHJ1Y3QgZXhwb3J0X2FyZ3MgKik7CiB2b2lkCXZmc19wZXJpb2RpYyhzdHJ1Y3QgbW91bnQg KiwgaW50KTsKIGludAl2ZnNfYnVzeShzdHJ1Y3QgbW91bnQgKiwgaW50KTsKK3ZvaWQJdmZzX2V4 amFpbF9jbG9uZShzdHJ1Y3QgbW91bnQgKiwgc3RydWN0IG1vdW50ICopOwogdm9pZAl2ZnNfZXhq YWlsX2RlbGV0ZShzdHJ1Y3QgcHJpc29uICopOwogaW50CXZmc19leHBvcnQJCQkgLyogcHJvY2Vz cyBtb3VudCBleHBvcnQgaW5mbyAqLwogCSAgICAoc3RydWN0IG1vdW50ICosIHN0cnVjdCBleHBv cnRfYXJncyAqLCBib29sKTsK --00000000000071e6ce060a78d585-- From nobody Sun Nov 19 04:10:16 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SXxxw4wMQz519S2 for ; Sun, 19 Nov 2023 04:10:28 +0000 (UTC) (envelope-from mike@karels.net) Received: from mail2.karels.net (mail2.karels.net [3.19.118.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "freebsd", Issuer "freebsd" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SXxxv5f1Mz4DX6; Sun, 19 Nov 2023 04:10:27 +0000 (UTC) (envelope-from mike@karels.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=karels.net header.s=mail2 header.b=cETMVQeR; spf=pass (mx1.freebsd.org: domain of mike@karels.net designates 3.19.118.201 as permitted sender) smtp.mailfrom=mike@karels.net; dmarc=none Received: from mail2.karels.net (localhost [IPv6:0:0:0:0:0:0:0:1]) by mail2.karels.net (8.17.1/8.17.1) with ESMTP id 3AJ4AHif062239; Sat, 18 Nov 2023 22:10:17 -0600 (CST) (envelope-from mike@karels.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=karels.net; s=mail2; t=1700367017; bh=Kj9VXN0ec4NtdXdmaivIfegRSxeTPaWPpPLK4LAt3Fc=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=cETMVQeReoUR2hEB1MBViT8msRhLhX6rE8ym4p7yvUbCTrcIPTp35W8fjOEMuOE5k yx2Ma7J9JCQJ2eCDlKk7q/Pi9wcI7a6DYn8L5Qeqo0GiytK5ulXM5GKbVj0sSPuMgC VeXZ+Shu2ImjBjYnIeSyOv5awAjZSO1GefqGrQyrEn0XjhQza8ygOkxgjsrKck3pyC YtzO4QLX0VKGVFIgAQ+fTaQ5yaePlOR6N4yMRXTf8XAGBIHLAqtRy+ffHnoo9HROIH F2zoD18FIfupoAmR4KG9aeg6k1hfP5OZ4o0XA/3ua8m8x5YuxEzUXsugGCAu3L9j73 Ew95gpJhXuKAw== Received: from [10.0.2.130] ([73.62.165.147]) by mail2.karels.net with ESMTPSA id A77GGKmKWWUd8wAAs/W3XQ (envelope-from ); Sat, 18 Nov 2023 22:10:17 -0600 From: Mike Karels To: Rick Macklem Cc: FreeBSD CURRENT , Alexander Motin , Martin Matuska , Garrett Wollman Subject: Re: NFS exports of ZFS snapshots broken Date: Sat, 18 Nov 2023 22:10:16 -0600 X-Mailer: MailMate (1.14r5964) Message-ID: <7C7920E0-B0FC-4255-AD5C-ECBDDC18CF43@karels.net> In-Reply-To: References: <25943.60056.880614.452966@hergotha.csail.mit.edu> <91988E23-ED50-4379-AA5F-4B069E08D80F@karels.net> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-2.00 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[karels.net:s=mail2]; R_SPF_ALLOW(-0.20)[+ip4:3.19.118.201]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_TO(0.00)[gmail.com]; ASN(0.00)[asn:16509, ipnet:3.16.0.0/14, country:US]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[karels.net:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[mike]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DMARC_NA(0.00)[karels.net]; TAGGED_RCPT(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4SXxxv5f1Mz4DX6 X-Spamd-Bar: - On 18 Nov 2023, at 21:19, Rick Macklem wrote: > On Sat, Nov 18, 2023 at 4:43=E2=80=AFPM Mike Karels w= rote: >> >> On 18 Nov 2023, at 17:23, Rick Macklem wrote: >> >>> On Sat, Nov 18, 2023 at 2:27=E2=80=AFPM Mike Karels = wrote: >>>> >>>> On 18 Nov 2023, at 15:58, Rick Macklem wrote: >>>> >>>>> On Sat, Nov 18, 2023 at 8:09=E2=80=AFAM Rick Macklem wrote: >>>>>> >>>>>> On Fri, Nov 17, 2023 at 8:19=E2=80=AFPM Mike Karels wrote: >>>>>>> >>>>>>> On 17 Nov 2023, at 22:14, Mike Karels wrote: >>>>>>> >>>>>>>> On 17 Nov 2023, at 21:24, Rick Macklem wrote: >>>>>>>> >>>>>>>>> Most of the changes in stable/13 that are not in releng/13.2 >>>>>>>>> are the "make it work in a jail" stuff. Unfortunately, they are= >>>>>>>>> a large # of changes (mostly trivial edits adding vnet macros),= >>>>>>>>> but it also includes export check changes. >>>>>>>>> >>>>>>>>> I have attached a trivial patch that I think disables the expor= t >>>>>>>>> checks for jails. If either of you can try it and see if it fix= es >>>>>>>>> the problem, that would be great. >>>>>>>>> (Note that this is only for testing, although it probably does = not >>>>>>>>> matter unless you are running nfsd(8) in vnet jails.) >>>>>>>> >>>>>>>> Yes, I can see snapshots with the patch. This system is just a = test >>>>>>>> system that doesn't normally run ZFS or NFS, so no problem messi= ng >>>>>>>> with permissions. It's a bhyve VM, so I just added a small disk= and >>>>>>>> enabled ZFS for testing. >>>>>>> >>>>>>> btw, you might try to get mm@ or maybe mav@ to help out from the = ZFS >>>>>>> side. It must be doing something differently inside a snapshot t= han >>>>>>> outside, maybe with file handles or something like that. >>>>>> Yes. I've added freebsd-current@ (although Garrett is not on it, h= e is >>>>>> cc'd) and these guys specifically... >>>>>> >>>>>> So, here's what appears to be the problem... >>>>>> Commit 88175af (in main and stable/13, but not 13.2) added checks = for >>>>>> nfsd(8) running in jails by filling in mnt_exjail with a reference= to the cred >>>>>> used when the file system is exported. >>>>>> When mnt_exjail is found NULL, the current nfsd code assumes that = there >>>>>> is no access allowed for the mount. >>>>>> >>>>>> My vague understanding is that when a ZFS snapshot is accessed, it= is >>>>>> "pseudo-mounted" by zfsctl_snapdir_lookup() and I am guessing that= >>>>>> mnt_exjail is NULL as a result. >>>>>> Since I do not know the ZFS code and don't even have an easy way t= o >>>>>> test this (thankfully Mike can test easily), I do not know what to= do from >>>>>> here? >>>>>> >>>>>> Is there a "struct mount" constructed for this pseudo mount >>>>>> (or it actually appears to be the lookup of ".." that fails, so it= >>>>>> might be the parent of the snapshot subdir?)? >>>>>> >>>>>> One thought is that I can check to see if the mount pointer is in = the >>>>>> mountlist (I don't think the snapshot's mount is in the mountlist)= and >>>>>> avoid the jail test for this case. This would assume that snapsho= ts are >>>>>> always within the file system(s) exported via that jail (which inc= ludes >>>>>> the case of prison0, of course), so that they do not need a separa= te >>>>>> jail check. >>>>>> >>>>>> If this doesn't work, there will need to be some sort of messing a= bout >>>>>> in ZFS to set mnt_exjail for these. >>>>> Ok, so now onto the hard part... >>>>> Thanks to Mike and others, I did create a snapshot under .zfs and I= can >>>>> see the problem. It is that mnt_exjail =3D=3D NULL. >>>>> Now, is there a way that this "struct mount" can be recognized as "= special" >>>>> for snapshots, so I can avoid the mnt_exjail =3D=3D NULL test? >>>>> (I had hoped that "mp->mnt_list.tqe_prev" would be NULL, but that i= s not >>>>> the case.) >>>> >>>> Dumb question, is the mount point (mp presumably) different between = the >>>> snapshot and the main file system? >>> Not a dump question and the answer is rather interesting... >>> It is "sometimes" or "usually" according to my printf(). >>> It seems that when you first "cd >> where mnt_exjail =3D=3D NULL.. Then when you look at directories with= in the >>> snapshot, you get the mp of the file system that .zfs exists in, whic= h does >>> have mnt_exjail set non-NULL. >>> >>> There is this snippet of code in zfsctl_snapdir_lookup(): >>> /* >>> * Fix up the root vnode mounted on .zfs/snapshot/. >>> * >>> * This is where we lie about our v_vfsp in order to >>> * make .zfs/snapshot/ accessible over NFS >>> * without requiring manual mounts of . >>> */ >>> ASSERT3P(VTOZ(*vpp)->z_zfsvfs, !=3D, zfsvfs); >>> VTOZ(*vpp)->z_zfsvfs->z_parent =3D zfsvfs; >>> >>> /* Clear the root flag (set via VFS_ROOT) as well. */ >>> (*vpp)->v_vflag &=3D ~VV_ROOT; >>> which seems to set the mp to that of the parent, but it >>> seems this does not happen for the initial lookup of >>> the ? >>> >>> I'll note that there is code before this in >>> zfsctl_snapdir_lookup() for handling cases >>> like "." and ".." that return without doing this. >>> >>> Now, why does this work without the mnt_exjail >>> check (as in 13.2)? >>> I am not quite sure, but there is this "cheat" in the >>> NFS server (it has been there for years, maybe decades): >>> /* >>> * Allow a Lookup, Getattr, GetFH, Secinfo on an >>> * non-exported directory if >>> * nfs_rootfhset. Do I need to allow any other Ops? >>> * (You can only have a non-exported vpnes if >>> * nfs_rootfhset is true. See nfsd_fhtovp()) >>> * Allow AUTH_SYS to be used for file systems >>> * exported GSS only for certain Ops, to allow >>> * clients to do mounts more easily. >>> */ >>> if (nfsv4_opflag[op].needscfh && vp) { >>> if (!NFSVNO_EXPORTED(&vpnes) && >>> op !=3D NFSV4OP_LOOKUP && >>> op !=3D NFSV4OP_GETATTR && >>> op !=3D NFSV4OP_GETFH && >>> op !=3D NFSV4OP_ACCESS && >>> op !=3D NFSV4OP_READLINK && >>> op !=3D NFSV4OP_SECINFO && >>> op !=3D NFSV4OP_SECINFONONAME) >>> nd->nd_repstat =3D NFSERR_NOFILEHANDLE; >>> This allows certain operations to be done on >>> non-exported file systems and I think that is enough >>> to allow this to work when mnt_exjail is not checked. >>> (Note that NFSV4OP_LOOKUPP is not in the list, >>> which might explain why it is the one that fails for >>> Garrett. I don't think it can be added to this list >>> safely, since that would allow a client to move above >>> the exported file system into "uncharted territory".) >>> >>>> Just curious. Also, what is mnt_exjail >>>> normally set to for file systems not in a jail? >>> mnt_exjail is set to the credentials of the thread/process >>> that exported the file system (usually mountd(8)). >>> When not in a jail, cr_prison for these credentials >>> points to prison0. >>> >>> Btw, I checked and the "other mp that has mnt_exjail =3D=3D NULL >>> is in the mountlist, so the idea of checking "not in mountlist" >>> is a dead end. >>> >>> I am looking for something "unique" about this other mp, >>> but haven't found anything yet. >>> Alternately, it might be necessary to add code to >>> zfsctl_snapdir_lookup() to "cheat and change the mp" >>> in more cases, such as "." and ".." lookups? >> >> It seems to me that if ZFS is creating an additional mount structure, >> it should be responsible for setting it up correctly. That could >> involve a vfs-level routine to do some of the cloning. In any case, >> it seems to me that mnt_exjail should be set properly, e.g. by duping >> the one in the original mount structure. Probably ZFS is the only >> file system type that would need this added. > I've created a patch that I think does this. It seemed to test ok for m= y case. > It's in D42672 on reviews.freebsd.org. > I have also attached it here (this diff doesn't use -U999999, so it mig= ht be > easier to apply). It works for me in 13-stable, and looks plausible. Mike > Hopefully this fixes the problem. Sorry for the breakage. > > rick > >> >> Mike >> >>> rick >>> ps: I added all the cc's back in because I want the >>> ZFS folk to hopefully chime in. >>> >>>> >>>> Mike >>>> >>>>> Do I need to search mountlist for it? >>>>> >>>>> rick >>>>> ps: The hack patch attached should fix the problem, but can only be= >>>>> safely used if mountd/nfsd are not run in any jails. >>>>> >>>>>> >>>>>> I will try and get a test setup going here, which leads me to.. >>>>>> how do I create a ZFS snapshot? (I do have a simple ZFS pool runni= ng >>>>>> on a test machine, but I've never done a snapshot.) >>>>>> >>>>>> Although this problem is not in 13.2, it will have shipped in 14.0= =2E >>>>>> >>>>>> Any help with be appreciated, rick >>>>>> >>>>>>> >>>>>>> Mike >>>>>>>> >>>>>>>>> rick >>>>>>>>> >>>>>>>>> On Fri, Nov 17, 2023 at 6:14=E2=80=AFPM Mike Karels wrote: >>>>>>>>>> >>>>>>>>>> CAUTION: This email originated from outside of the University = of Guelph. Do not click links or open attachments unless you recognize th= e sender and know the content is safe. If in doubt, forward suspicious em= ails to IThelp@uoguelph.ca. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Rick, have you been following this thread on freebsd-stable? = I have been able >>>>>>>>>> to reproduce this using a 13-stable server from Oct 7 and a 15= -current system >>>>>>>>>> that is up to date using NFSv3. I did not reproduce with a 13= =2E2 server. The >>>>>>>>>> client was running 13.2. Any ideas? A full bisect seems fair= ly painful, but >>>>>>>>>> maybe you have an idea of points to try. Fortunately, these a= re all test >>>>>>>>>> systems that I can reboot at will. >>>>>>>>>> >>>>>>>>>> Mike >>>>>>>>>> >>>>>>>>>> Forwarded message: >>>>>>>>>> >>>>>>>>>>> From: Garrett Wollman >>>>>>>>>>> To: Mike Karels >>>>>>>>>>> Cc: freebsd-stable@freebsd.org >>>>>>>>>>> Subject: Re: NFS exports of ZFS snapshots broken >>>>>>>>>>> Date: Fri, 17 Nov 2023 17:35:04 -0500 >>>>>>>>>>> >>>>>>>>>>> < said: >>>>>>>>>>> >>>>>>>>>>>> I have not run into this, so I tried it just now. I had no = problem. >>>>>>>>>>>> The server is 13.2, fully patched, the client is up-to-date = -current, >>>>>>>>>>>> and the mount is v4. >>>>>>>>>>> >>>>>>>>>>> On my 13.2 client and 13-stable server, I see: >>>>>>>>>>> >>>>>>>>>>> 25034 ls CALL open(0x237d32f9a000,0x120004) >>>>>>>>>>> 25034 ls NAMI "/mnt/tools/.zfs/snapshot/weekly-2023-4= 5" >>>>>>>>>>> 25034 ls RET open 4 >>>>>>>>>>> 25034 ls CALL fcntl(0x4,F_ISUNIONSTACK,0x0) >>>>>>>>>>> 25034 ls RET fcntl 0 >>>>>>>>>>> 25034 ls CALL getdirentries(0x4,0x237d32faa000,0x1000= ,0x237d32fa7028) >>>>>>>>>>> 25034 ls RET getdirentries -1 errno 5 Input/output e= rror >>>>>>>>>>> 25034 ls CALL close(0x4) >>>>>>>>>>> 25034 ls RET close 0 >>>>>>>>>>> 25034 ls CALL exit(0) >>>>>>>>>>> >>>>>>>>>>> Certainly a libc bug here that getdirentries(2) returning [EI= O] >>>>>>>>>>> results in ls(1) returning EXIT_SUCCESS, but the [EIO] error = is >>>>>>>>>>> consistent across both FreeBSD and Linux clients. >>>>>>>>>>> >>>>>>>>>>> Looking at this from the RPC side: >>>>>>>>>>> >>>>>>>>>>> (PUTFH, GETATTR, LOOKUP(snapshotname), GETFH, GETATTR) >>>>>>>>>>> [NFS4_OK for all ops] >>>>>>>>>>> (PUTFH, GETATTR) >>>>>>>>>>> [NFS4_OK, NFS4_OK] >>>>>>>>>>> (PUTFH, ACCESS(0x3f), GETATTR) >>>>>>>>>>> [NFS4_OK, NFS4_OK, rights =3D 0x03, NFS4_OK] >>>>>>>>>>> (PUTFH, GETATTR, LOOKUPP, GETFH, GETATTR) >>>>>>>>>>> [NFS4_OK, NFS4_OK, NFS4ERR_NOFILEHANDLE] >>>>>>>>>>> >>>>>>>>>>> and at this point the [EIO] is returned. >>>>>>>>>>> >>>>>>>>>>> It seems that clients always do a LOOKUPP before calling READ= DIR, and >>>>>>>>>>> this is failing when the subject file handle is the snapshot.= The >>>>>>>>>>> client is perfectly able to *traverse into* the snapshot: if = I try to >>>>>>>>>>> list a subdirectory I know exists in the snapshot, the client= is able to >>>>>>>>>>> LOOKUP(dirname) just fine, but LOOKUPP still fails with >>>>>>>>>>> NFS4ERR_NOFILEHANDLE *on the subndirectory*. >>>>>>>>>>> >>>>>>>>>>> -GAWollman >>>>>>>>>> From nobody Sun Nov 19 15:51:18 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SYFVl4S62z50xfg for ; Sun, 19 Nov 2023 15:51:27 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "www.zefox.com", Issuer "www.zefox.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SYFVk4wDLz3dkh for ; Sun, 19 Nov 2023 15:51:26 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of fbsd@www.zefox.net has no SPF policy when checking 50.1.20.27) smtp.mailfrom=fbsd@www.zefox.net; dmarc=none Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.17.1/8.15.2) with ESMTPS id 3AJFpJvw020932 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Sun, 19 Nov 2023 07:51:19 -0800 (PST) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.17.1/8.15.2/Submit) id 3AJFpJdG020931 for freebsd-current@freebsd.org; Sun, 19 Nov 2023 07:51:19 -0800 (PST) (envelope-from fbsd) Date: Sun, 19 Nov 2023 07:51:18 -0800 From: bob prohaska To: freebsd-current@freebsd.org Subject: How much survives an install/reboot cycle? Message-ID: List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Result: default: False [1.71 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; AUTH_NA(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.982]; NEURAL_SPAM_LONG(0.79)[0.787]; MID_RHS_WWW(0.50)[]; WWW_DOT_DOMAIN(0.50)[]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[zefox.net]; R_SPF_NA(0.00)[no SPF record]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; MIME_TRACE(0.00)[0:+] X-Rspamd-Queue-Id: 4SYFVk4wDLz3dkh X-Spamd-Bar: + How much of a running system's state survives a reboot? I used to think the answer was "nothing", but from time to time a second reboot behaves a little differently from the previous one. The most recent example was an update to bpf.c: Prior to the update an armv7 system had been inclined to drop ssh connections left up for days. After updating and running a build/install cycle the behavior persisted, but since a second reboot with no intentional changes it has stopped. I've not tampered with nextboot, so I don't think that's it. Maybe I'm just imagining imagining things.... Thanks for reading, bob prohaska From nobody Sun Nov 19 18:17:49 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SYJlc3QLhz517yG for ; Sun, 19 Nov 2023 18:17:48 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SYJlc1j6Vz4Q7W for ; Sun, 19 Nov 2023 18:17:48 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x536.google.com with SMTP id 4fb4d7f45d1cf-53e2308198eso5187580a12.1 for ; Sun, 19 Nov 2023 10:17:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1700417867; x=1701022667; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=i9ylEbd6rybsbbRPe5S+SZy2/rqKu766YiFlTf78Zvg=; b=W7pNml3SfVEQrsoFW+SbZDfR5ca0TJtgwD9AgSUSHgz5ZYAKtZ/c4qT6faqzcs8QGm b0WjDyayNC+08CsbKWpm/VoKpC+lO/xsdUAtZcscAw7vatH6d6JK+XiYjoRBStnxQ8wY bRaR3F4HIYr/pcQ+QaxT14tTO6YM+1MFlkAsKFUi7uvPXZkeZ1E5947Qlmam9x2qjPrc afC8cafXAzn5OLlr3kHPySVxEfYnzgHPlIA3rBApBaKv2k58yuzeP8DYGq23RnlPkTZi tZfsDzQGjTK8AELvGEVYFLWo6eaV6HkDvHOtoJ8Ra8W4eYkRifzH1dW1WDULIbRneGAT p1Xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700417867; x=1701022667; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=i9ylEbd6rybsbbRPe5S+SZy2/rqKu766YiFlTf78Zvg=; b=Vdu0UNLV8S1r6pPg0MrSjvYFmA6TIKwV0i9RWafceKGxaluWJWuZu0xMkI0+OwRJJm lffua1p/vPniPJNwNmMiyZ+rxEc+J/kANUK7vFMMF0MqALOtQFmO887Fy1iwxcL3tyvt COcURbpNcbbiXcGQZGfNiWy+JyrQzM+onhP8GcHhSzekg1AaJWCEIbBWF7w8KgaXmNQ/ 6EC+EoAkHBaSSLZY0mMHvUP3dtGVvgoWZXDbhCpVSh6e33paff/Kn2VL6/KbqusbgXTC 1gflhj5XSwyxYN0+uegWdjNFWBntQy+WpC14dT1wuuXa335yEaqyJ0oMudU5OyCQA0b3 dciQ== X-Gm-Message-State: AOJu0Yy75qFW2cRSfu8iuMuf3GSOjzt7KLk3Ztp3dPks5qDohi/TUZY9 oDbOXfxNCK6AkM6Xe0MqwMQAi97jW59u9bKDHDnf6fag2nX5qw9Tbhw= X-Google-Smtp-Source: AGHT+IFpt7gD8ReVYKkgTh6UbBtMXbGDf54gTsSEoEvn0OKSf38Kspt5XS54KbAhczEZyI/mhq8k31KyJCXWqQgkE+o= X-Received: by 2002:a05:6402:1a5a:b0:548:b096:6c0d with SMTP id bf26-20020a0564021a5a00b00548b0966c0dmr922142edb.38.1700417866959; Sun, 19 Nov 2023 10:17:46 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Sun, 19 Nov 2023 11:17:49 -0700 Message-ID: Subject: Re: How much survives an install/reboot cycle? To: bob prohaska Cc: freebsd-current@freebsd.org Content-Type: multipart/alternative; boundary="000000000000ee22fb060a8562f4" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4SYJlc1j6Vz4Q7W --000000000000ee22fb060a8562f4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Nov 19, 2023 at 8:51=E2=80=AFAM bob prohaska w= rote: > How much of a running system's state survives a reboot? I used to think > the answer was "nothing", but from time to time a second reboot behaves > a little differently from the previous one. > > The most recent example was an update to bpf.c: Prior to the update an > armv7 system had been inclined to drop ssh connections left up for days. > After updating and running a build/install cycle the behavior persisted, > but since a second reboot with no intentional changes it has stopped. > > I've not tampered with nextboot, so I don't think that's it. Maybe I'm > just imagining imagining things.... > Generally, nothing survives reboot. Nothing in RAM that is. Everything on disk should survie. swap survives boot, but we ignore it unless there's a core dump in there. Dirty pages in the buffer cache will be flushed to disk, unless the reboot is a crash. And sometimes, RAM will survive a reboot. It all depends on what the BIOS does. I've had dmesg survie several reboots because the kernel was the same and the msgbuf wound up in the same place and we recovered (And appended to= ) that (though saying that now, I'm not sure how FreeBSD does that). Some machines don't destructively memory test all RAM and/or clear RAM on a warm reboot. ssh dropping connections, though, may be a feature of ssh. Check that the following options haven't change and/or aren't causing you problems: ServerAliveCountMax, ServerAliveInterval, or TCPKeepAlive (for the client) and ChanelTimeout, UnusedConnectionTimeout, ClientAliveCountMax, ClientAliveInterval, or TCPKeepAlive (for the server). I hit this with ForwardX11Timeout which I had set to 99999w but now have to set it to 999w since sshd rejected such a long time. The default values of these change at random (at least seemingly so for people like me that can't be bothered to read the release notes). I'm not completely sure this is your problem, but if it is weird, it may be. Warner --000000000000ee22fb060a8562f4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sun, Nov 19, 2023 at 8:51=E2=80=AF= AM bob prohaska <fbsd@www.zefox.ne= t> wrote:
How much of a running system's state survives a reboot? I used to thin= k
the answer was "nothing", but from time to time a second reboot b= ehaves
a=C2=A0 little differently from the previous one.

The most recent example was an update to bpf.c: Prior to the update an
armv7 system had been inclined to drop ssh connections left up for days. After updating and running a build/install cycle the behavior persisted, but since a second reboot with no intentional changes it has stopped.

I've not tampered with nextboot, so I don't think that's it. Ma= ybe I'm
just imagining imagining things....

Ge= nerally, nothing survives reboot. Nothing in RAM that is. Everything
<= div>on disk should survie. swap survives boot, but we ignore it unless
there's a core dump in there.=C2=A0 Dirty pages in the buffer cac= he will be
flushed to disk, unless the reboot is a crash.

And sometimes, RAM will survive a reboot. It all depends = on what the BIOS
does. I've had dmesg survie several reboots = because the kernel was the same
and the msgbuf wound up in the sa= me place and we recovered (And appended to)
that (though saying t= hat now, I'm not sure how FreeBSD does that). Some machines
d= on't destructively memory test all RAM and/or clear RAM on a warm reboo= t.

ssh dropping connections, though, may be a feat= ure of ssh. Check that the
following options haven't change a= nd/or aren't causing you problems:
ServerAliveCountMax, Serve= rAliveInterval, or TCPKeepAlive (for the client) and
ChanelTimeou= t, UnusedConnectionTimeout, ClientAliveCountMax, ClientAliveInterval,
=
or TCPKeepAlive (for the server). I hit this with ForwardX11Timeout wh= ich
I had set to 99999w but now have to set it to 999w since sshd= rejected such
a long time. The default values of these change at= random (at least seemingly
so for people like me that can't = be bothered to read the release notes). I'm not
completely su= re this is your problem, but if it is weird, it may be.

Warner
--000000000000ee22fb060a8562f4-- From nobody Mon Nov 20 01:12:45 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SYTyZ07gnz51dWm for ; Mon, 20 Nov 2023 01:12:54 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SYTyY6hyBz3fXv; Mon, 20 Nov 2023 01:12:53 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700442773; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VUhcT3TBwsrLe9X0lBJPmCa1csaw453LoyuhQsKNskA=; b=veJkgOo1HrMHhrdHButhO6EEnr3HEZhhCVkiI81QC6lnj3Lh25zpz6xyhXgJhKqr+5cqDo LAw+DlCJemMJS1sbyAsyqz/5T+6iLt/D5OdM0NUQ6h2OUkzRt964PrhzS+p/EBuJiVyJ/n +N8aWS3N9laS7sWWInbOhlIAoOUymn81RCP8DS7JJtmHsE0N5Skf9BdKLEsu9cJve3limV DfuN+NUounnhEWqW5RvQeuHabhgEFhIYHAvRyPpbSbPZpeWB9XOcLz/FPectBWKOpypks6 itq9cHZAkRq3vgxif34mhxMhb7OCd22D7GnKTSNk2WcEbEi7YQM7U0vEMKq2gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700442773; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VUhcT3TBwsrLe9X0lBJPmCa1csaw453LoyuhQsKNskA=; b=s9VjuAIE21AUzfEI8dLBLUmHkCQT9gNjXKFL25/aA2tpMtdRglTKLhis8xxWzijwRreh1E EdLqP2zkdkH9izMMDrDnt1aHu1STJQpWOoBFNaoghdkfCDru9NHPCmC3U4Q3+hvXmELfTb hXRnK1eazow6Lye1aWpcfWurnV8qTkPRT+MW4Sh02MdemHaECCMfSGAb58CWrRwzYsSBY7 KwS4ZXWVf8zS1wdCuvbFg5Ayppt6tD8tM2kNp/UE1QJsh5KVNXn7PAakJ4yOjv+hzCTtj8 BREA/bdlaQbr1FG/J2TDtF4a9tvXk62Dy8Vqh0ZInA956gZ5T3jQGde3pXquKw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1700442773; a=rsa-sha256; cv=none; b=KrOv0msnzhGcljKJRM329W5y9YJIsFxX26iOvXxMdZWk9i4RQa4uGq/FcdcPOwVeIGhg/9 vdnqfdyz6HtbNMg5928pMTD+Iq8Kd0endQebDRj9qOOTKg9j933jeue4Bawomz4M5LHrnM cYCfCDxkgFnpeMvcAF4eSFhRMZAd2Dkf5bu2jJZfSEPld19W/9aG7C+p29cgZWbLA8W5Nc zyxhk45MKB/ygvltB5Vr+XcbKtK+JT0MaGiy70jEatr4pY8Dn97PCKoCkcjU1BTkEmGqqW yIB+UZkhz5wDnHPwfoLoi+t6s6/xlyiMqVI+uQux/RzDDwe0b1GZjTrS03rSvw== Received: from smtpclient.apple (ns1.oxydns.net [45.32.91.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4SYTyX502bz6yg; Mon, 20 Nov 2023 01:12:52 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.4\)) Subject: Re: How much survives an install/reboot cycle? From: Zhenlei Huang In-Reply-To: Date: Mon, 20 Nov 2023 09:12:45 +0800 Cc: freebsd-current@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: To: bob prohaska X-Mailer: Apple Mail (2.3696.120.41.1.4) > On Nov 19, 2023, at 11:51 PM, bob prohaska wrote: >=20 > How much of a running system's state survives a reboot? I used to = think > the answer was "nothing", but from time to time a second reboot = behaves > a little differently from the previous one.=20 Warner has a good description about that. I totally agree. >=20 > The most recent example was an update to bpf.c: Prior to the update an > armv7 system had been inclined to drop ssh connections left up for = days. > After updating and running a build/install cycle the behavior = persisted, > but since a second reboot with no intentional changes it has stopped. The most recent change to bpf.c is 7a974a649848 (bpf: Make dead_bpf_if = const) . It is not a functional change, and I do not think it will affect ssh. There could be issues under the earth. Anyway please do not hesitate to report if you get recovered by = reverting 7a974a649848. >=20 > I've not tampered with nextboot, so I don't think that's it. Maybe I'm > just imagining imagining things....=20 >=20 > Thanks for reading, >=20 > bob prohaska >=20 >=20 Best regards, Zhenlei From nobody Mon Nov 20 03:48:48 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SYYQY2VkWz51q02 for ; Mon, 20 Nov 2023 03:48:53 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SYYQX3Prqz4Rmy for ; Mon, 20 Nov 2023 03:48:52 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=UOo9Ggl5; spf=pass (mx1.freebsd.org: domain of grahamperrin@gmail.com designates 2a00:1450:4864:20::332 as permitted sender) smtp.mailfrom=grahamperrin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-40836ea8cbaso10960085e9.0 for ; Sun, 19 Nov 2023 19:48:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700452130; x=1701056930; darn=freebsd.org; h=in-reply-to:autocrypt:content-language:references:to:subject:from :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=bm2x6zKOgkilo0qdFGuVfPWYCUzPAFgp1Kywkzzpoj4=; b=UOo9Ggl5qt0P8pMLHddSm0sbceD41GsaU9AFn9hTRkrEV1WgXaZY1EF9HIppcN1O1Z wjBs+pJySK8X0J7akx/rQ5ErOAZv7NOuOS210laBPWKgoy8Yy8g12RBFzB8aH1kV+U7a qChJyrUPTrO70JKFuEG3BXgDFKk6DniOeHOg3ZDUGL47VL3D2uF9Adl37xca3vF6AJVX cPt3AIrRp6nRKWbJeOv1wKjKxT9m7jrR9Pv1n/q2ka0ql1Enw2kdH7FAhj8du2dQQh50 9SXccj38y6rRHJX8/YmNCBsIQ2lvz+1atlT75R1qAprpV5Orqs9hxRWFwPrnJBvNdA4N x8Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700452130; x=1701056930; h=in-reply-to:autocrypt:content-language:references:to:subject:from :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=bm2x6zKOgkilo0qdFGuVfPWYCUzPAFgp1Kywkzzpoj4=; b=theJ/JZt8spOq4ZZiLPP6twXMnbkDvaPCkg8cslr0dgO/M4asfFQnsFTGIn2D0qxRu dRjG3cmtUnINu8/KGD2NVsDc7If/JRCMNgRUE3iPp5F3He8QNNvBHZMoE72YFcMaENJo a46fnBXcTBY1ilCXbD7ZId78UCDsWeKZ0GQp2eCm3pBuT/HtUMDQ3IHA6d4fo46MTjAx TBL6labWr/IZe+qJUcaFYtDkadGqyo5eKnOtDHlyU+K78esg0/60q9WjtCfBnD7MVCk/ OhOsHlRfPUsF+puNhGE7Eu9v9yS5R4BJoPHI4ECewm6ogR9C+hUCdxOeFNUCc3RVMVIR JlAA== X-Gm-Message-State: AOJu0YwvimxtBFPizHJNaSDHFjcBTde8nRRbNvmEt2SrnItLTuOvKs0G zboqibPCil6dkhN3UiQQ29fgZTYMV9nEFA== X-Google-Smtp-Source: AGHT+IHmp05JV8Q4bVfVVIEyGenNU+LqY4syIAeFIhN44R1qj/dIvYJK5ZnQe4+aFHBDgs3rinTH8Q== X-Received: by 2002:a05:600c:1c84:b0:405:4776:735a with SMTP id k4-20020a05600c1c8400b004054776735amr5147084wms.2.1700452130194; Sun, 19 Nov 2023 19:48:50 -0800 (PST) Received: from [192.168.1.10] (80-42-66-93.dynamic.dsl.as9105.com. [80.42.66.93]) by smtp.gmail.com with ESMTPSA id v12-20020a05600c214c00b004065daba6casm16026503wml.46.2023.11.19.19.48.49 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 19 Nov 2023 19:48:49 -0800 (PST) Content-Type: multipart/alternative; boundary="------------OIWRdicYxwljNgwvBHqCO5T9" Message-ID: Date: Mon, 20 Nov 2023 03:48:48 +0000 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Graham Perrin Subject: Re: something got wrong with 32bit ld between 1500002 and 1500003 To: freebsd-current@freebsd.org References: Content-Language: en-US Autocrypt: addr=grahamperrin@gmail.com; keydata= xsFNBGKYt7ABEAClu83dJ3ZKfVgPOk9YKRv0Z+dl2b88+k9R4vwAmElgguYdKE7yhnQNhhWM v9vi6AFrBMc2oJdVHJ2OrXfwpELBFIgiSMEWNsC4e+Z3HtSajcl+pFZsP7ciiSoycj/w3wIV kAZoVGbhyIbNG7fbCEJ8q81TbfsGypV3bRmbZVvGNecBguYiooBtz2Qht1p3itXMkIA6P9pS YDl+6QddZLyUUAjAnFv2QDoYSHLnaDUWw4oONZsB0SKVu8jMIBh4uJZoYEOvdvc9jQQdOpA2 CAgA6ulfm42Ikr9lKBUUCtjqiWAhJ7iXOTyHAIdR4Mf8alCE6tdTq6dHdIt+GktTY7oYNyL2 3aD3C7I5waU0SFXvJcOMG10QLfwYQMOQoYQ9XJ0U5A28WYiDcylDdUWT7SappP1e1ZMeJWWO y14mxxNzHaJSI4rK8P/p5tp3Q7SSC4k5gMh9zKba3K2ApCWNbVLGvXsJeQkZZNvu70tE81ey AHI5iZcB6D7WaHysBUmsKaEpbcmm1ZThTnGL0SHEl5to5Jab5Fg6O+Cnly5sVz5lX/v8Aosx kKNei7SCVqXOVtteQeGxWbXWbhPgbMyc0Gi3DuxBI/yvJ43k/rJysQlLGLWfJx/UXprwLluC PDK9EvKEB+fD1Z349uzp1sKr3ihpySbyKI8fpudftnAz4EsoCwARAQABzSZHcmFoYW0gUGVy cmluIDxncmFoYW1wZXJyaW5AZ21haWwuY29tPsLBlAQTAQoAPhYhBFk/5bLDBwftvJcvCrdn SG9KGNQLBQJimMMBAhsDBQkFo5qABQsJCAcDBRUKCQgLBRYDAgEAAh4FAheAAAoJELdnSG9K GNQLbHAQAJi998y42bEbq5HmABYovmAEtQj33YSUWyc9QRmAHpN8Er3lTKsgmZcVChB5Fu/d go2oYynDjlVpA7+wiSmg4AG78mOYbg/e19XMhrH0keDKqZXFkU+G7agR0mF09qvpQZ9MTJYZ 2u7FtytZK665UfipOdV8eGn2hFC/WynjUwEzKyryBgbbLAEbfOPeZNry4h2ZPWbtTvx/PE/V X3Vh2oGqYx69DCGz+0xEhy62ZKbkX5SL8LUf/1WViyCVzsHasFxmFxYPWIfBy8ayQ7xapz7M cSXSQyu4oDT4qh9eZiGP9/aAcZKHcV6t9y77JGhUJ/5O1sANKMa3YhgimE+Z86LHYa1IH774 PHj1nAXBwS+Cj/1l/NQoQcyjvOj8zuCsMJVaLMb6B46YsReP4+3yBLpyeBC//t6zWPbgAkWW VjROC0dXUAMTFpnA6NZe3UghG+Nc4fnCLGOhc2nyWFYHIaYV6Hv1ITFSem9DdeNnR1CFm1VM TJ7i7TuqYM+WZTkoUsTf4c46hS/ZNJZSCxh0s9yYr+BYk3XBbd+ElaZ1dJE6cuSVdw15+P2h DnprurxC4byl4YFkn+UAVvQsOgeq6aSHLOHX0weYu1OLoiPYsTdyGhne72+kDhEEdFD5aHdQ PFrbQIrqWLV0a04++0ZwGpNvXtgnWhDdAQJDwGsSSwbLzsFNBGKYt7ABEADRb1tZuh7DPYET 0wK6fe7owbYgM+RfKhmcrGgR2HI9M2q6+0WKF/ITnggWdIW2Ecc4z2boLz/cwvPGCS7/YxZM 61KklGCwuS7q1s04XnHDWHuFxfXQPzAdVmNO3bYoMZbJjHXs6sB2u5ksiwPwaMAWWaGkviSj c5pwvHCiTmX5vH5CBj/Vi+5ESyX38vK4JM5S/m4ouI/6M9biyFgimV+v3vVyCxJCT1gI9g4o GIh1qq5S433b1fihn4yHPf8XOKyBpA/QcwLONViBqJL5nnOxpsh344rNxn2R7CcRzzicOV+e 2IbMem4lwNWQlZKoRotKXZi9LqN5mynSBYqAUdoZum0QinWT9F22B0Qex5PH1zAt9i2W91Vd kcPB3LwkRXj07ycRtsSzpgPA6fLc6AsoWFslHl8kVOO5eJIA4xhjlPa+W8lguQHZ0iX+5uAv 2eAgXR2swADuHPuENNFStmsgAMl8OOOgtq75yA5TpyIzxMuXV9Nmp0VfIaUM/IdLdmxhc1pC c320l5fYMHVLFAReWEbSj2QH8YzWfpXHIegutWWYEbH9SiDXgS9KoKmCJV/Qa+x6/b8y3pOZ vnIbCDaynC2Yr50s8gRa9kb54JE8Z+p8r16U3SEsK3PtUi0RF0e51danCVHrrE6/Hat2XUO/ 6nnYgVgFOrLao6Gh/VMs8wARAQABwsF8BBgBCgAmFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsF AmKYt7ACGwwFCQWjmoAACgkQt2dIb0oY1Av7qg//YjCZg8VXyMzXssgIQpROKKqh5V0UBSQl rM3tq4tWhyg0HVMugQj0Om+iNPsEEOGHkm6tyhHMzlKGpAc/l0iAM+8twIyg44Yo5+DcfFXr OMTbTw9T9jDsWOkOBksxy29iYhgpqpWdDBnhXvrJp/FNAiX8CfzrIOZeFPydDoEiKBEXAxfe a9o5J/JeVnZiUeoiFe7i68nZGsb4JxhPczNfqW12t0Ll5/ibjszg5BgjXiLao0KqbWNh4bS5 CVwH90Or+5qqWgzWPeBiuz+rN2QXE/V/fL44GEj1YKASCqmaiYRgjoRFubz1aq1wCXMXY3Iq d4525rscUgS7HBxbblnyTodUPaamN/2nSzcmE/Pkx8MApDSgZCIhs0RTAg+/AoX4HULV1rSE TQwMrBEQt84Tw5W5rHsvXKr4ZEsJUpbPLWYTISsp23nHR+vZtL/Ug+OWCmHC7X7D21xk/xVJ 4sA1RLJBKdCHtnyA4Unv/kNS1KVGxHnITVyw1a71QJADu4qsdtM5u6CyYUhqhM1oseWtV6j+ Qi8KC/G4C3AgZf06fe2fVl42z2grTabL4bC6FQXMwTX2dsm5NakWjUCmUL8uwsQE7ZA4zKxo EYI1YV9q1birpzncYRupr1qnMoggMUHWq0IBYshFQrEO8PeVUZBw7/GfAeh3argdw2Qu748T Cyw= In-Reply-To: X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[grahamperrin]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::332:from]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4SYYQX3Prqz4Rmy X-Spamd-Bar: --- This is a multi-part message in MIME format. --------------OIWRdicYxwljNgwvBHqCO5T9 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 11/16/23 08:44, Dima Panov wrote: > Moin-moin! > > After upgrade to recent -current (1500003) I've got a ld error for > 32bit an aarch64 machine:( > > ELF binary type "9" not known. > eval: /libexec/ld-elf32.so.1: Exec format error > > Snapshot from 2023-10-13 was fine, no errors > > 15.0-CURRENT #0 main-6f38d2e7b0: Thu Nov 16 03:29:03 MSK 2023 is still > broken > Similarly, with poudriere-devel and with pkg on AMD64:  ldconfig: warning: /lib32: No such file or directory --------------OIWRdicYxwljNgwvBHqCO5T9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit


On 11/16/23 08:44, Dima Panov wrote:
Moin-moin!

After upgrade to recent -current (1500003) I've got a ld error for 32bit an aarch64 machine:(

ELF binary type "9" not known.
eval: /libexec/ld-elf32.so.1: Exec format error

Snapshot from 2023-10-13 was fine, no errors

15.0-CURRENT #0 main-6f38d2e7b0: Thu Nov 16 03:29:03 MSK 2023 is still broken

Similarly, with poudriere-devel and with pkg on AMD64: 

<http://paste.purplehat.org/view/32ae904b#L14>

<http://paste.purplehat.org/view/3a03c541#L15>

 ldconfig: warning: /lib32: No such file or directory

--------------OIWRdicYxwljNgwvBHqCO5T9-- From nobody Mon Nov 20 08:48:02 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SYh3p6KRgz50wDZ for ; Mon, 20 Nov 2023 08:48:06 +0000 (UTC) (envelope-from fluffy@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SYh3p5lT0z4sTM; Mon, 20 Nov 2023 08:48:06 +0000 (UTC) (envelope-from fluffy@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700470086; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=JSG11M6lxg07RYJEDG4RRy7+Gt2e1ZRGObdEXn9LuEA=; b=Y0k43Rvm1wEOrenfeFSyCXfkWNtKU+k+xhDR491SjC2FmrGvUww671HALi+w//byEL5aym nTH1EOvzQQokEsbPFacBz8RweO93PNUgGxrqwIR/Z0MGAUnYe6l4y/uWR9HXP+mitAfLSb 4mzzOnurR5M5iXJ/6eCno85Sg89B/ANOAF9eseLXNzUJTgn/aQZZjK6gNPiz2dQswsuiBK V5ynBPHReGNBUoIUHXbBl7EH9/2bpXjikiJ2Frb0zaMCe5Mg9jipvF39fDCNQDI187LZzu 0pjQTzRoQX3zkYmgXVumbNJO8bl8LuYT3hp7j/8m2k3aS26m49jsRPvbGJzbWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700470086; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=JSG11M6lxg07RYJEDG4RRy7+Gt2e1ZRGObdEXn9LuEA=; b=dTa8jE1x+Hxcr5cHfwWOZ6OaybjLhzDwCqPo/mATyj2wn29S97trQcu24OGefoqaF1F3qS GeluZQmaS3ilix861PZuN8bCnTLetT+GfTlGNNbbdflPAh9YI7hIO7T51gf0bDX7unNgqJ J6yX390kuL6P9kHRGgQA4Xh0KLYVnrEPthtoldd7GxjBZ+qprLTBGUBVzB6248FhdNDxQv lLRSAGBd33zpOSdGrSLLMkQ2lciX8Mf0MGVvVi5fcKNCM7SwRpuUM/kOc3d149nH1+uGod hNtZGXqIDQNdBui00VHcCqPPQPb/TGxyDcDiHjjQmc/kw30UB+zAyDJLNZ89RA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1700470086; a=rsa-sha256; cv=none; b=SaoJGACRFjIRfXg/QWw1/4FSTGhUj2P7MQZxUAV+eyqOL6IyYL2HnCDAY2k18vOdx+KhWC iR75X7deIMTAcnGPwiCiaYTWQEQJ9ecqLJuRh8nRBY9wn0ezb0BidFc5BeRZiGFDSLlMAE tBu+JQ2M46W2DBGtJZ6DDtUgLhaDd/KxFDB8oeZycg/5IQs+j3jolGm0euIZ1qC61Gh/ba Zt1GM/pVdeVIcAKq+Ame03YP8X0TC+eTbGTJ6U1E96OkcRKIrD4Gw5UXFcwd+RtpCtN/KW 7ezuUtUMqB9n/C/V8fBRrWYeJdUIkT76r8sz7e9kW7aG5YYbNsDVTikA0efMIQ== Received: from [10.216.0.102] (unknown [188.243.165.67]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: fluffy) by smtp.freebsd.org (Postfix) with ESMTPSA id 4SYh3p28V8zlv7; Mon, 20 Nov 2023 08:48:06 +0000 (UTC) (envelope-from fluffy@FreeBSD.org) Message-ID: <68db7c32-598e-4bd6-9ed4-e3e9a50f5aa3@FreeBSD.org> Date: Mon, 20 Nov 2023 11:48:02 +0300 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: something got wrong with 32bit ld between 1500002 and 1500003 To: Graham Perrin , freebsd-current@freebsd.org References: Content-Language: ru, en-GB, en-US From: Dima Panov Organization: FreeBSD.org In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------0AD7tn7tTDXvI0lrjdnAhArW" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------0AD7tn7tTDXvI0lrjdnAhArW Content-Type: multipart/mixed; boundary="------------NMzOSpoV2EMgQes0POK6X0jg"; protected-headers="v1" From: Dima Panov To: Graham Perrin , freebsd-current@freebsd.org Message-ID: <68db7c32-598e-4bd6-9ed4-e3e9a50f5aa3@FreeBSD.org> Subject: Re: something got wrong with 32bit ld between 1500002 and 1500003 References: In-Reply-To: --------------NMzOSpoV2EMgQes0POK6X0jg Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 DQoNCk9uIDIwLjExLjIwMjMgMDY6NDgsIEdyYWhhbSBQZXJyaW4gd3JvdGU6DQo+IA0KPiBP biAxMS8xNi8yMyAwODo0NCwgRGltYSBQYW5vdiB3cm90ZToNCj4+IE1vaW4tbW9pbiENCj4+ DQo+PiBBZnRlciB1cGdyYWRlIHRvIHJlY2VudCAtY3VycmVudCAoMTUwMDAwMykgSSd2ZSBn b3QgYSBsZCBlcnJvciBmb3IgMzJiaXQgYW4gYWFyY2g2NCBtYWNoaW5lOigNCj4+DQo+PiBF TEYgYmluYXJ5IHR5cGUgIjkiIG5vdCBrbm93bi4NCj4+IGV2YWw6IC9saWJleGVjL2xkLWVs ZjMyLnNvLjE6IEV4ZWMgZm9ybWF0IGVycm9yDQo+Pg0KPj4gU25hcHNob3QgZnJvbSAyMDIz LTEwLTEzIHdhcyBmaW5lLCBubyBlcnJvcnMNCj4+DQo+PiAxNS4wLUNVUlJFTlQgIzAgbWFp bi02ZjM4ZDJlN2IwOiBUaHUgTm92IDE2IDAzOjI5OjAzIE1TSyAyMDIzIGlzIHN0aWxsIGJy b2tlbg0KPj4NCj4gU2ltaWxhcmx5LCB3aXRoIHBvdWRyaWVyZS1kZXZlbCBhbmQgd2l0aCBw a2cgb24gQU1ENjQ6DQo+IA0KPiA8aHR0cDovL3Bhc3RlLnB1cnBsZWhhdC5vcmcvdmlldy8z MmFlOTA0YiNMMTQ+DQo+IA0KPiA8aHR0cDovL3Bhc3RlLnB1cnBsZWhhdC5vcmcvdmlldy8z YTAzYzU0MSNMMTU+DQo+IA0KPiAgwqBsZGNvbmZpZzogd2FybmluZzogL2xpYjMyOiBObyBz dWNoIGZpbGUgb3IgZGlyZWN0b3J5DQo+IA0KDQpJdCBpcyByZWxhdGVkIHRvIGh0dHBzOi8v Y2dpdC5mcmVlYnNkLm9yZy9zcmMvY29tbWl0Lz9pZD05OTEzMmRhZjZmNzBjYjBjYzk2OWM1 NTVkMzYxMjU0N2ZhM2NmMWRiDQoNCg0KLS0gDQpTaW5jZXJlbHksDQpEaW1hIChmbHVmZnlA RnJlZUJTRC5vcmcsIGh0dHBzOi8vdC5tZS9GbHVmZnlCU0QpDQooZGVza3RvcCwga2RlLCB4 MTEsIG9mZmljZSwgcG9ydHMtc2VjdGVhbSlARnJlZUJTRCB0ZWFtDQo= --------------NMzOSpoV2EMgQes0POK6X0jg-- --------------0AD7tn7tTDXvI0lrjdnAhArW Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature.asc" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEELTAsy5mEEwxvh7r8+4ugndU5jykFAmVbHUIFAwAAAAAACgkQ+4ugndU5jymX +hAAmrs05qMqfULe3k5T+7SUvieKy3uU3uUMkFEJqc6pJDmKti3XvQYpJaW4r/oFy+5cQ3ilZEMt Ojw6q+iTusspKksQ6J6Cd+Vivsw5E3/mLlZfy8RhwYqkiM2kCdSS5A8zMxIqegyu9XuHYZv0GFu6 46xj8ytElDNY52yR5K4OnRyl12J1RmytdUGc32qQg9yTrN0K+yUPnuUB0/KPeGAlCd2+1sjB0vL0 xX9ZWNqmUeN60yZFe3BQ7iX/UHKk3i/Ia8mBVkaEAIZyVX49KicE/A2HxwoF4nEA6e36uxk3CK+4 4e0O7+FJn8+iRwesjoElE0GqBkWUVBe2VYAj5DiiY3MVcRJ/E5XosEvlp4nvcVWplHRwJOJAVo+e MojmcS9eG2dqMdwKbQmvxnQXrvEshhaqa5ZodM5BiNqGAT1bUlGqr4C9vF2V3zqrR1gkinykfMWu xczskVYxpsmgU+LnBtl4eXGDq7O66PeKjwCiM72GIuMUpOy8o8OShj8pcEILLBYFdfCS0TglOvkt RwnrZuvONizW6BXH8oyBb+xm6e77KZ1o/LUY7xopgcdpF55uXApsZn81VyuWmo7x9+EhJ93qSEI7 K6dvz9t9/kucjTgTElLB/h7olSeOcXqc2jMf/aOeqiJRv48aCBg9VHz+y96i+AkE6Hw+3exapN+f 0GE= =QjBV -----END PGP SIGNATURE----- --------------0AD7tn7tTDXvI0lrjdnAhArW-- From nobody Mon Nov 20 09:30:37 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SYj160HJvz50yw3 for ; Mon, 20 Nov 2023 09:30:50 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SYj156xhBz3Fdx for ; Mon, 20 Nov 2023 09:30:49 +0000 (UTC) (envelope-from eduardo@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700472650; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=HubP4v9TcMiYTxP2Gj1Lecom5PxUJW2mMkbY2Undszg=; b=oGTdMUKmXNTwT9qr8r4zK3rBBH0Cj/giPyMBJrMjCrAF2sXmbLXWZojRZKC9K68lea7Lg8 B2qvLbAy+P9gfQWM5kCG+tOjPVKxG1PzLUG7NkQ569QAQw6vXENLTYpkL9K7aF78/gwb2q GHJbqlQ2hbH2lRHmAeeMyI6Dd09LF8AEQXJiZsRL+wqn5NfOuQhyorGnZuTbciqVQFqfGG JzO/cJwDd068MvOI/VuK2LrMNqbawL6B6aCIqHPJPtKB+OrYDgGzf7bDms0kiXrGtnDX2p xsK5XoXf8QlH6xmNhrwl6OeZKwFqcNk9SsYP6wji6wUNMao3TAj/Tab9GuDvSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700472650; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=HubP4v9TcMiYTxP2Gj1Lecom5PxUJW2mMkbY2Undszg=; b=PZnH3Fv0/fM1lCrIBRFBMekILg5bICsJhamaIpsf+ejk8mAYPzORgFkeSB6tjFYYqKvv/a KzZWoDdKsBpwD0RMLq77kpxP8+BRbtkrtlLnOV+KI9uBuk9L39k1XCGmga/uyIzn4F3njl fcMEGaUShCLa9z9J/zRPsTIweuqH8D3C4qKV23POYQiuWnjPM5kcf06eKsOoz32/EYLCAg IOXk0nuMBoCHG41tf/32tcO4ip9eSHdopPYMoOK75E1z91ceRDDU68BlHSPbx7o76l7UJY qcoJyfvgdwc/AFWLczvcP8J1VvBfiIXM2zrq30sH8P2AmapVatWFCEnG7bvbtQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1700472650; a=rsa-sha256; cv=none; b=AzccDii7D88MGGXh0SSwqDFGW76wkj3U4EwrPwGrMGWICmkoe4ECgr0zZFJjMNI8C7tKSO r9YqSIRJxgAFchTsCHEo9VbTqJ/LyswtovQr6CiFJHQlJz0W95CBjkObaMzd3eHMeYcEPl Wc1a2vNiM4p7B+QdddboQLc7ALuW/zvO7nHLGnR+kg/QnGyeD96gv98kJdFk44CUOPmU9u WZELrvT7+DBpXH/7Gnh74GEvbTfOO5ESq3Kibme7aNSvyU4LW3A2NLPphY7qVhArrJGGEv FpqhIgKcsK1Y3wEZ+ai7ELE7VWaLj6M9Rc6nBztxX9yt4ymt9KNLleOdMW0QHw== Received: from mail-oo1-f48.google.com (mail-oo1-f48.google.com [209.85.161.48]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: eduardo) by smtp.freebsd.org (Postfix) with ESMTPSA id 4SYj1561RRzkwH for ; Mon, 20 Nov 2023 09:30:49 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: by mail-oo1-f48.google.com with SMTP id 006d021491bc7-581f78a0206so2415002eaf.2 for ; Mon, 20 Nov 2023 01:30:49 -0800 (PST) X-Gm-Message-State: AOJu0YxinXb6/Q0FdT/f3/hkQ5oFPllxODdEhw6RQsQS0DUXLbctxZ0g Vil0hbCqoLIjx4dpw0yVQ9cAaMXWEpD2iZur+d4= X-Google-Smtp-Source: AGHT+IEMxQsepynKpNSmJFymn1mLNmHMU5ocfKXdqVaFSgUjhdD7TvecNP1X8LIBDWH4WrnofcDy8BWCaKEfttYON5w= X-Received: by 2002:a05:6358:4194:b0:16b:70e4:8665 with SMTP id w20-20020a056358419400b0016b70e48665mr7890774rwc.15.1700472648845; Mon, 20 Nov 2023 01:30:48 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Nuno Teixeira Date: Mon, 20 Nov 2023 09:30:37 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: something got wrong with 32bit ld between 1500002 and 1500003 To: Graham Perrin Cc: freebsd-current@freebsd.org Content-Type: multipart/alternative; boundary="0000000000002f4bbc060a922422" --0000000000002f4bbc060a922422 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, I'm also checking it at @ports: https://lists.freebsd.org/archives/freebsd-ports/2023-November/004910.html Graham Perrin escreveu no dia segunda, 20/11/2023 =C3=A0(s) 03:50: > > On 11/16/23 08:44, Dima Panov wrote: > > Moin-moin! > > After upgrade to recent -current (1500003) I've got a ld error for 32bit > an aarch64 machine:( > > ELF binary type "9" not known. > eval: /libexec/ld-elf32.so.1: Exec format error > > Snapshot from 2023-10-13 was fine, no errors > > 15.0-CURRENT #0 main-6f38d2e7b0: Thu Nov 16 03:29:03 MSK 2023 is still > broken > > Similarly, with poudriere-devel and with pkg on AMD64: > > > > > > > > ldconfig: warning: /lib32: No such file or directory > --=20 Nuno Teixeira FreeBSD Committer (ports) --0000000000002f4bbc060a922422 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,


Graham Perrin <grahamperrin@gmail.com> escreveu no dia segunda, 20/11/2= 023 =C3=A0(s) 03:50:
=20 =20 =20


On 11/16/23 08:44, Dima Panov wrote:
Moin-moin!

After upgrade to recent -current (1500003) I've got a ld error fo= r 32bit an aarch64 machine:(

ELF binary type "9" not known.
eval: /libexec/ld-elf32.so.1: Exec format error

Snapshot from 2023-10-13 was fine, no errors

15.0-CURRENT #0 main-6f38d2e7b0: Thu Nov 16 03:29:03 MSK 2023 is still broken

Similarly, with poudriere-devel and with pkg on AMD64:=C2=A0

<http://paste.purplehat.org/view/32ae904b#L14>

<http://paste.purplehat.org/view/3a03c541#L15>

=C2=A0ldconfig: warning: /lib32: No such file or directory



--
Nuno Teixeira
FreeBSD Committ= er (ports)
--0000000000002f4bbc060a922422-- From nobody Mon Nov 20 10:23:58 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SYkBh38F5z513jb for ; Mon, 20 Nov 2023 10:24:12 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SYkBg30mdz3Lmk for ; Mon, 20 Nov 2023 10:24:11 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=ZZXm6TVU; spf=pass (mx1.freebsd.org: domain of grahamperrin@gmail.com designates 2607:f8b0:4864:20::d2c as permitted sender) smtp.mailfrom=grahamperrin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-io1-xd2c.google.com with SMTP id ca18e2360f4ac-7a67f447bf0so174015639f.2 for ; Mon, 20 Nov 2023 02:24:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700475850; x=1701080650; darn=freebsd.org; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=BZqrCCTaV0rrw6388BRCaAJbK+HMSISpRtOQMhg/SZA=; b=ZZXm6TVUppRgiEfpKuBAAYZbl60ivoyz/IYD/JNFQKtb92kqPM3yZs8A02MC4hsw3k e0q/+oZ6ZCdgjaJ+bojo8Jmkt/9x5heFfA/EJuklmM4vDAgMeYmkB9d98KhM39gWjt8z BB0BMPeIVB7onahhxYfvRHAx9aanyf9q0L74iZpsg+pKIy3o4jb57zdUhdLNLDLE/hyk caTztr64kerictiVuOaqhdZEX/dJEBSXfNCbegydqu1dmCrMIxlS9/4NEx7LN0b9x6Fu 1t1pHYTl/2Rtac289+4/4Y6wG7RvKPrAiq+awLpxwoZcqhaRVsQSa6N3DsQiQEmaMXI3 t6yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700475850; x=1701080650; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BZqrCCTaV0rrw6388BRCaAJbK+HMSISpRtOQMhg/SZA=; b=xAcPP8LOZaA3xaW02tOQA6lwDchngCsFJHV0FgdHZcrb3mM9DRjyltTTWhZTWzHYrE Qjn5WucBqAokGpln4ovIi0YNnm4OJhk6KYaWFRuLtqbod6vAPpDW1zPMEOokJMq3mppo KS6RGN5kBNO9/YRysX7ocS5ShfTCV56KOaF2g1y6OCy2NwBglCO5qpOlqyBELSh5VHCm 6+lQvzTCt38j0npXsVMNrrsGpLRIdw8vFTlBYM0NvBSkWmOC04iW0esABkDPGohsnx1f 6IeeZNsug+SES+8vLxqCWkzm6sNX7558FKGXyfVtf9fvOKC/gIHxhO9W133H8b0TYiy1 kGHA== X-Gm-Message-State: AOJu0YxeGUTcPX2aiXY3pQW9De+IWue/ZmT66ST//PE0HT3iYgAPjn1M tvTMO1pk316PuLMma7PqiRfgG08mlhzfp6r4fEaaaXIBCJqr0A== X-Google-Smtp-Source: AGHT+IHjMrbQ/a3MvtkD/ESp2RBK66pgDWBv1gm0mlftf598VEMhBnFRl+scEcLtUpIMRehsYYclZh0k7YCRUfEQ8nY= X-Received: by 2002:a05:6602:400c:b0:7af:f3d4:f7a0 with SMTP id bk12-20020a056602400c00b007aff3d4f7a0mr1896322iob.4.1700475849914; Mon, 20 Nov 2023 02:24:09 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <68db7c32-598e-4bd6-9ed4-e3e9a50f5aa3@FreeBSD.org> In-Reply-To: <68db7c32-598e-4bd6-9ed4-e3e9a50f5aa3@FreeBSD.org> From: Graham Perrin Date: Mon, 20 Nov 2023 10:23:58 +0000 Message-ID: Subject: 99132daf6f70 rc.d/ldconfig: Prepend rtld stdlib paths to ldconfig(32)_paths To: FreeBSD-CURRENT Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FREEFALL_USER(0.00)[grahamperrin]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::d2c:from]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org] X-Rspamd-Queue-Id: 4SYkBg30mdz3Lmk X-Spamd-Bar: --- On Mon, 20 Nov 2023 at 08:48, Dima Panov wrote: > On 20.11.2023 06:48, Graham Perrin wrote: > >> =E2=80=A6 >> ldconfig: warning: /lib32: No such file or directory >> > > It is related to https://cgit.freebsd.org/src/commit/?id=3D99132daf6f70cb= 0cc969c555d3612547fa3cf1db Thanks. Also noted, at the weekend, from beefy18 and ampere2: From nobody Mon Nov 20 15:47:04 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SYsMK0bB5z51SlC for ; Mon, 20 Nov 2023 15:47:09 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x82c.google.com (mail-qt1-x82c.google.com [IPv6:2607:f8b0:4864:20::82c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SYsMJ2kN9z4Sly; Mon, 20 Nov 2023 15:47:08 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b="BRiL/EfP"; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::82c as permitted sender) smtp.mailfrom=markjdb@gmail.com; dmarc=none Received: by mail-qt1-x82c.google.com with SMTP id d75a77b69052e-41ea8debcdaso26944811cf.1; Mon, 20 Nov 2023 07:47:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700495227; x=1701100027; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=uz898a8IkT2sfOFiedN8Wr4xGvaMU4kbqdSGZT9F2Yw=; b=BRiL/EfPbjW16WetRaNAvY5oMeP+R5B19hQhx5WA9hRDFiHqWWIVTv4k1zF5rv0sPx szjVePFcsON9VcRtZbY4le1ahbbR56JPEMoZlb/DBLhRN4E43ia9omaT5pm+gzn2xGt3 UTkfG3xYFav4mANgHhCpq1ropcgmIlgdoFmOEdv/HTUXjbV1r2jq9AE15qbIH0Tk2URF RLSwLmSur2kSGTIcSt3jticKfDUgs063PxMKqKZHmX0IAK/KeTmcHU/pBljgIZzOxEOF 9iknmw0go0lU4+XiJI03eZxd8odfCE59VPTaSht+/X3MD4a1QRccXMjPICA5KEAzsWLD 92hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700495227; x=1701100027; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uz898a8IkT2sfOFiedN8Wr4xGvaMU4kbqdSGZT9F2Yw=; b=RQ3t5KQFq5tUCMcWhrB+TqYSr7Z4ng+Q4jy/6K4syq34mYtgMwNWeD+AB54b4aJ9Xf HF6zjy8Tz330ghyv2yo2/f46dIxm2Ikad8igBMpR+w1zvZJpOheBBMNqRAlhj8d9BqHk TkA8bLdeIq82bhNVQUBJXl42kraRs5Udox3QS5hCGQJwNAMSfxUrOP1VizlNWWuRL54A MkhdjOmdhlurSzz8whKzanqhdXRh/+yyqlZUUBJse70ImqIx+yjs1tf9aZhihyYjoVuH 53uAe8vSTU7EOAKWqnFQwFzVHdKJd/+SXOo7LuUylT6HLUqYxk2x06QV0OLN6gQPEQ3h YqPA== X-Gm-Message-State: AOJu0Yz0QfoqNCokd//HHWdFBrRt56nXCTmQyE107e2C3+GrJvl9EGSI u/boL2x32QD9qewGI3jYFT+SHo3XSDY= X-Google-Smtp-Source: AGHT+IE/0Sx4/CweRE5ICSgpbcdCJ64zsZyvyiFrXDiWZGUFJD19tKYcXjb+9DFqpXPNMIo4f0X+0w== X-Received: by 2002:ad4:5aed:0:b0:679:e8e3:5f8a with SMTP id c13-20020ad45aed000000b00679e8e35f8amr248011qvh.58.1700495227214; Mon, 20 Nov 2023 07:47:07 -0800 (PST) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id p7-20020ad452c7000000b0066d20f29e5fsm3007577qvs.35.2023.11.20.07.47.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Nov 2023 07:47:06 -0800 (PST) Date: Mon, 20 Nov 2023 10:47:04 -0500 From: Mark Johnston To: "Bjoern A. Zeeb" Cc: current@freebsd.org, mhorne@freebsd.org Subject: Re: db> reset -> panic: acquiring blockable sleep lock with spinlock or critical section held ... Message-ID: References: <0q526858-7s33-27qo-6q80-8o1q10415q2s@yvfgf.mnoonqbm.arg> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <0q526858-7s33-27qo-6q80-8o1q10415q2s@yvfgf.mnoonqbm.arg> X-Spamd-Result: default: False [-2.70 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::82c:from]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[current@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[freebsd.org]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4SYsMJ2kN9z4Sly X-Spamd-Bar: -- On Thu, Nov 16, 2023 at 10:21:38PM +0000, Bjoern A. Zeeb wrote: > Hi, > > I seem to remember changes related to that a while ago but my cache > is miss for the actual change. Are we suppoed to handle this case? > > It would be nice if "reset" would reset again the first time ... We should perhaps make SCHEDULER_STOPPED() true when resetting from the debugger, since we can't make any assumptions about the state of the system (since we don't know how we got into DDB in the first place). Then the reset path here would behave as it does after a panic. > KDB: enter: Break to debugger > [ thread pid 11 tid 100005 ] > Stopped at kdb_alt_break_internal+0x180: str xzr, [x19, #896] > db> reset > panic: acquiring blockable sleep lock with spinlock or critical section held (sleep mutex) eventhandler @ /usr/src/sys/kern/subr_eventhandler.c:269 > cpuid = 2 > time = 307 > KDB: stack backtrace: > db_trace_self() at db_trace_self > db_trace_self_wrapper() at db_trace_self_wrapper+0x38 > vpanic() at vpanic+0x1a0 > panic() at panic+0x48 > witness_checkorder() at witness_checkorder+0xb4c > __mtx_lock_flags() at __mtx_lock_flags+0xac > eventhandler_find_list() at eventhandler_find_list+0x44 > kern_reboot() at kern_reboot+0x284 > db_reset() at db_reset+0xd8 > db_command() at db_command+0x2e4 > db_command_loop() at db_command_loop+0x58 > db_trap() at db_trap+0x100 > kdb_trap() at kdb_trap+0x364 > handle_el1h_sync() at handle_el1h_sync+0x18 > --- exception, esr 0xf2000000 > kdb_alt_break_internal() at kdb_alt_break_internal+0x180 > kdb_alt_break() at kdb_alt_break+0x10 > uart_intr_rxready() at uart_intr_rxready+0x8c > uart_intr() at uart_intr+0x120 > intr_event_handle() at intr_event_handle+0xf4 > intr_isrc_dispatch() at intr_isrc_dispatch+0x78 > arm_gic_v3_intr() at arm_gic_v3_intr+0x120 > intr_irq_handler() at intr_irq_handler+0x88 > handle_el1h_irq() at handle_el1h_irq+0x14 > --- interrupt > cpu_idle() at cpu_idle+0x78 > sched_idletd() at sched_idletd+0x4a0 > fork_exit() at fork_exit+0x78 > fork_trampoline() at fork_trampoline+0x18 > KDB: enter: panic > [ thread pid 11 tid 100005 ] > Stopped at kdb_enter+0x48: str xzr, [x19, #896] > db> reset > Uptime: 5m7s > INFO: PSCI Power Domain Map: > > > -- > Bjoern A. Zeeb r15:7 > From nobody Mon Nov 20 17:54:22 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SYwBH3Nrkz51d39 for ; Mon, 20 Nov 2023 17:54:31 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "www.zefox.com", Issuer "www.zefox.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SYwBH0JV5z3FC9; Mon, 20 Nov 2023 17:54:30 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Authentication-Results: mx1.freebsd.org; none Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.17.1/8.15.2) with ESMTPS id 3AKHsNlX027126 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 20 Nov 2023 09:54:23 -0800 (PST) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.17.1/8.15.2/Submit) id 3AKHsNpG027125; Mon, 20 Nov 2023 09:54:23 -0800 (PST) (envelope-from fbsd) Date: Mon, 20 Nov 2023 09:54:22 -0800 From: bob prohaska To: Zhenlei Huang Cc: freebsd-current@freebsd.org Subject: Re: How much survives an install/reboot cycle? Message-ID: References: List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US] X-Rspamd-Queue-Id: 4SYwBH0JV5z3FC9 On Mon, Nov 20, 2023 at 09:12:45AM +0800, Zhenlei Huang wrote: > > > > On Nov 19, 2023, at 11:51 PM, bob prohaska wrote: > > > > How much of a running system's state survives a reboot? I used to think > > the answer was "nothing", but from time to time a second reboot behaves > > a little differently from the previous one. > > Warner has a good description about that. I totally agree. > > > > > The most recent example was an update to bpf.c: Prior to the update an > > armv7 system had been inclined to drop ssh connections left up for days. > > After updating and running a build/install cycle the behavior persisted, > > but since a second reboot with no intentional changes it has stopped. > > The most recent change to bpf.c is 7a974a649848 (bpf: Make dead_bpf_if const) . > It is not a functional change, and I do not think it will affect ssh. > There could be issues under the earth. > That is most helpful. Very likely the change I saw is simply coincidence. > Anyway please do not hesitate to report if you get recovered by reverting 7a974a649848. In this case I don't want to revert, the new behavior is desirable. My only puzzle was the seeming delay in its appearance. The only consistent issue remaining is reported in Bug 273566 . It finally dawned on me that the garbage characters must be originated on the USB end, transmitted to the getty process watching the serial end and get stuck in the transmit buffer when the link goes down. When the serial link comes back up they appear on the receiving console display. Many thanks to you and Warner! bob prohaska From nobody Mon Nov 20 18:21:19 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SYwnG0Qpsz51fZ5 for ; Mon, 20 Nov 2023 18:21:22 +0000 (UTC) (envelope-from mhorne@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SYwnF6xfcz3HtP; Mon, 20 Nov 2023 18:21:21 +0000 (UTC) (envelope-from mhorne@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700504482; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=wBdUOWble66qXtKdb1cycjEO6T4N+YHb6L3OOFiwQVM=; b=jBGGCrsJDBj4W+wz95Yn4BcmEqlyrm77qz+CbnuGUWwWzHHvDi0dobNJGr3FUh0YYntago 1z49as091E+rqfV2PA3sOtaLajUhiLFnNxq1wKBxrwK3uMHIVS0I4iuGevnEMFlVmwM/oI HTnp1HZ85Qm1moxlekXEEoMyz4OyDr1yqwpXPqokQNQc2UzRJh0lprnK+9GkxPJ+Lr2aj7 /F59YY9XAYXHMQdvTuLoHs8oNtp+M1NKrW3T9pwjjkihQBX9zG3SYOu0Uv6ZbsoZPg5+eq D0GWOtMpo/FcXf9GuB81vXCvEUS45GPLnUuT5XlW7z7+S4fJKfOnrGk1p+Qs8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700504482; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=wBdUOWble66qXtKdb1cycjEO6T4N+YHb6L3OOFiwQVM=; b=Rb4EsXnzSczIy4QnczxR3I43P91PCkG65pzO4ZhJ10pq+E7eY63i7YrK2hyuKyGrc0eap9 6lKUcYhz88WdJkQK/xFVQhP7AidbVz/iCBIdjcOgBPTtHx275f2mPgDaRAw7YhGRaFw50/ Av8E3plqB9QNhpfdacTtDrib2aaGiE5pcHD1mET4jQGwLInVdEp8Ewj94Y1lKTdYnFRJAa qGrCEfFOhd1eMJ3LcX6YEg+pUQ1SfeWlJi6u6qWRRwVy8hx+4KvAdII3EHedBFSvyLH/oX xnVEZJvYBrzwI7SBKV8O+uKXBtII4pvNQXrzbozwn44sRZunXGtYI+6n+y1F3Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1700504482; a=rsa-sha256; cv=none; b=VufjfJ7PgwS2v75mu1Groxk9/RqFEIArj2uZXeWu91m/wdjNmFzSB/7IEfqz3K3TgOKIEt Eew0tVlTZObSrylXFoRRs4c2RaY688g6kaG5NAAZ5CaGprxT7WSNw+nPDdCr1MvVO9+DaF SMFU21lTcFsLMFiXBvOG1juzdLPOJZZIF3JDALTn3q5hyWtpt2mT2ib0Z3C5HSyt5HCZdl z82Puktbk4zSrqNg/EMlNiEDyqkrs4FEVYg3tJRdSzd6buuelL8QRo3r8cp7ulLHtovMoU u44cxBFjMG1y/bai3Roai4AvXK0umrfthRY8WJr64ATI789SWv4MFVC5T8LLLw== Received: from [192.168.1.151] (host-173-212-76-127.public.eastlink.ca [173.212.76.127]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: mhorne) by smtp.freebsd.org (Postfix) with ESMTPSA id 4SYwnF4pJVzx1w; Mon, 20 Nov 2023 18:21:21 +0000 (UTC) (envelope-from mhorne@freebsd.org) Message-ID: <2b0ef12f-a158-4de3-8d25-ad2e5fcde644@freebsd.org> Date: Mon, 20 Nov 2023 14:21:19 -0400 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: db> reset -> panic: acquiring blockable sleep lock with spinlock or critical section held ... Content-Language: en-CA To: "Bjoern A. Zeeb" , current@freebsd.org References: <0q526858-7s33-27qo-6q80-8o1q10415q2s@yvfgf.mnoonqbm.arg> From: Mitchell Horne Autocrypt: addr=mhorne@freebsd.org; keydata= xsBNBFyS2dQBCADdiXBG8hBVLmYbxu7aSzbwLwUf3HkGFz3rooS1kwyy+SfmjZ4UKNnl9WMx WKrJ7OAZpiNH6bLQ5nsqfx09OnpWL8c/QuPbhNdUywQoqqYpRI0K8GEn//nS9Gs0KTYwVpWb XlrzP+jf3Uh/9L5mcQmStLIH4zaaqMYHW+pMuPrvBmLIHTvLj2QjOkxslrcUdord9uvxe5Ht LU8RuTpQpHOKz705Z9/v7twFdi2HtKzpLwO6SzVyu351di1J+GihsVpcT5josQV5cHbIP3Un x+kmtKBEEc/jl/zBglF7ruWUtwgbryID+2ZPEaO1Mj+RResX4LFVMusq3uUpWRb5WJXxABEB AAHNI01pdGNoZWxsIEhvcm5lIDxtaG9ybmVARnJlZUJTRC5vcmc+wsCUBBMBCgA+AhsDBQsJ CAcCBhUKCQgLAgQWAgMBAh4BAheAFiEEkp/cYPcfabAiQvACi/gnTOdUid8FAmIyDpUFCQtC z0EACgkQi/gnTOdUid8IsQf+N8IptrrCgifT5Z0/WUVFfnHThFOKf4zBjaGswsIM8+VKsKnF 15jCWHODUHP6s+dcQ4nQi81PHPsnMfBSkGPvN/X3ess2/1KUVkH+6tAJbqXDjXhD8HT+i0NM QEFIXlLnotpgIKW3yOHjKv3ZvKw9LCvUjyNY9vOJmLk/6AbbkFh+INo65nXtQWb/hM5FVEHW S+zUoU8AqZRJoVAQfj9wmIfg/HdsxeDGKL0zkv5AwKpccvb8VJNGJbCVMgoy5uQYcUeXxcie cg0VlbFLshNQTfyhVQ85vyuHahARrUWs/k8KiYODoBnW1ChtyF8yM6VZTzSYx7pINqPq2YZy i/Htd87ATQRcktnUAQgA3zt4M4ecoQqfxpjliNLujt9klDqvmkJvWmzMuMXdzlPgGRJ0doio 9YIeEdkOt6xN0pPTK/ReCZ8WqFQ8zo23u1pwGuo0CnR58XF19wyxyUuKu/PHbt+56mC8tNHm AXsMyXQmlDqWvn/WzLY7euNRtNS4QQIwtxfM5EC4GGa5KQwxn0kM7dkUSOE/cxr+/kNbHHzb gagZR4cnNUqtPPr3dYXcibCTzgz96Lyt3/qMLXX9RTBRzu+O6E+byxWOe8ar/ZlwY2b4wTQG mhgNttkSxKtxMpZnd8+DGV/bI1P5Ct/K2GeCwNyupQGON5ymn6o7jTch+qmFX0ItkBWO4zn4 9QARAQABwsB8BBgBCgAmAhsMFiEEkp/cYPcfabAiQvACi/gnTOdUid8FAmIyDtwFCQtCz4gA CgkQi/gnTOdUid/i5gf/aQ75pJR4TJFM2vVVr6PDIwTdl0b5EchB4w4s4g/zE84XNbMOQanb BginLYEhAacLQVAvM3XdvUEhwrhaMQdjdSEB1krResL3/mbxrtKwdHSMbHA3IS3XdvxFWTB7 P5JjUSPsW6hqgoidbn4w3OxaNHhs45H2b0Nx5QiKcSyepmCZuB52gCEHnEnrdaz8TFQMXOLq 94WbTmZeIjChW3FB61m1gTf0UEFjoZAfTAUB+pbwoCa4AykIeZnDC19vjsruVU9Gy5rLglwd bjsZNfXIJGOZNEvdF8FOBwM7DlXx7SYvTJcUNoNJjOKtQ0bYGVgGqYOB/y2mTjVuKeU0eOkN Uw== In-Reply-To: <0q526858-7s33-27qo-6q80-8o1q10415q2s@yvfgf.mnoonqbm.arg> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 11/16/23 18:21, Bjoern A. Zeeb wrote: > Hi, > > I seem to remember changes related to that a while ago but my cache > is miss for the actual change.  Are we suppoed to handle this case? > > It would be nice if "reset" would reset again the first time ... > Hi Bjoern, This is still my fault, I am sorry to say. If you recall, I proposed a fix after your initial report (back in February!), see https://reviews.freebsd.org/D38656. However, I held off on committing it because I had some more work to do in the area, and believed there was a more correct way to fix this edge-case. I posted what I believe to be the better fix just now, see https://reviews.freebsd.org/D42684. I will commit this ASAP along with some other tweaks to shutdown hooks which should (loaded word) eliminate this type of recursive panic during debugger reset. At least, that is the goal of the series :) I apologize for the delay on this, my ability to finish some of the work I've started has been spotty this year. Cheers, Mitchell > > KDB: enter: Break to debugger > [ thread pid 11 tid 100005 ] > Stopped at      kdb_alt_break_internal+0x180:   str     xzr, [x19, #896] > db> reset > panic: acquiring blockable sleep lock with spinlock or critical section > held (sleep mutex) eventhandler @ /usr/src/sys/kern/subr_eventhandler.c:269 > cpuid = 2 > time = 307 > KDB: stack backtrace: > db_trace_self() at db_trace_self > db_trace_self_wrapper() at db_trace_self_wrapper+0x38 > vpanic() at vpanic+0x1a0 > panic() at panic+0x48 > witness_checkorder() at witness_checkorder+0xb4c > __mtx_lock_flags() at __mtx_lock_flags+0xac > eventhandler_find_list() at eventhandler_find_list+0x44 > kern_reboot() at kern_reboot+0x284 > db_reset() at db_reset+0xd8 > db_command() at db_command+0x2e4 > db_command_loop() at db_command_loop+0x58 > db_trap() at db_trap+0x100 > kdb_trap() at kdb_trap+0x364 > handle_el1h_sync() at handle_el1h_sync+0x18 > --- exception, esr 0xf2000000 > kdb_alt_break_internal() at kdb_alt_break_internal+0x180 > kdb_alt_break() at kdb_alt_break+0x10 > uart_intr_rxready() at uart_intr_rxready+0x8c > uart_intr() at uart_intr+0x120 > intr_event_handle() at intr_event_handle+0xf4 > intr_isrc_dispatch() at intr_isrc_dispatch+0x78 > arm_gic_v3_intr() at arm_gic_v3_intr+0x120 > intr_irq_handler() at intr_irq_handler+0x88 > handle_el1h_irq() at handle_el1h_irq+0x14 > --- interrupt > cpu_idle() at cpu_idle+0x78 > sched_idletd() at sched_idletd+0x4a0 > fork_exit() at fork_exit+0x78 > fork_trampoline() at fork_trampoline+0x18 > KDB: enter: panic > [ thread pid 11 tid 100005 ] > Stopped at      kdb_enter+0x48: str     xzr, [x19, #896] > db> reset > Uptime: 5m7s > INFO:    PSCI Power Domain Map: > > From nobody Mon Nov 20 19:40:47 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SYyY23KV7z51lhl for ; Mon, 20 Nov 2023 19:40:54 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:13b:39f::9f:25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE Root Certificate Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SYyY20qGnz3Qxh; Mon, 20 Nov 2023 19:40:53 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Authentication-Results: mx1.freebsd.org; none Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id E07368D4A22D; Mon, 20 Nov 2023 19:40:49 +0000 (UTC) Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 8F80C2D029D6; Mon, 20 Nov 2023 19:40:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id mVYKq-6VY2bV; Mon, 20 Nov 2023 19:40:48 +0000 (UTC) Received: from strong-iwl0.sbone.de (strong-iwl0.sbone.de [IPv6:fde9:577b:c1a9:4902:b66b:fcff:fef3:e3d2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 7C7E32D029D2; Mon, 20 Nov 2023 19:40:48 +0000 (UTC) Date: Mon, 20 Nov 2023 19:40:47 +0000 (UTC) From: "Bjoern A. Zeeb" To: Mitchell Horne cc: current@freebsd.org Subject: Re: db> reset -> panic: acquiring blockable sleep lock with spinlock or critical section held ... In-Reply-To: <2b0ef12f-a158-4de3-8d25-ad2e5fcde644@freebsd.org> Message-ID: References: <0q526858-7s33-27qo-6q80-8o1q10415q2s@yvfgf.mnoonqbm.arg> <2b0ef12f-a158-4de3-8d25-ad2e5fcde644@freebsd.org> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="1098556516-921959288-1700509248=:3600" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/32, country:DE] X-Rspamd-Queue-Id: 4SYyY20qGnz3Qxh This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --1098556516-921959288-1700509248=:3600 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8BIT On Mon, 20 Nov 2023, Mitchell Horne wrote: Hi Mitchell, > On 11/16/23 18:21, Bjoern A. Zeeb wrote: >> Hi, >> >> I seem to remember changes related to that a while ago but my cache >> is miss for the actual change.  Are we suppoed to handle this case? >> >> It would be nice if "reset" would reset again the first time ... >> > > Hi Bjoern, > > This is still my fault, I am sorry to say. If you recall, I proposed a fix > after your initial report (back in February!), see now that you say I do. I thought we had this all sorted. Cache miss, miss. Maybe I had a local patch and hadn't seen it for a while because of that. I likely dropped the ball on review and testing feedback. > I posted what I believe to be the better fix just now, see > https://reviews.freebsd.org/D42684. I will commit this ASAP along with some > other tweaks to shutdown hooks which should (loaded word) eliminate this type > of recursive panic during debugger reset. At least, that is the goal of the > series :) > > I apologize for the delay on this, my ability to finish some of the work I've > started has been spotty this year. Oh, no worries; I've been way worse this year. Thank you for stepping up working on this and the fixes. It is much appreciated! Bjoern -- Bjoern A. Zeeb r15:7 --1098556516-921959288-1700509248=:3600-- From nobody Tue Nov 21 01:21:24 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SZ6665cJcz51S1Q for ; Tue, 21 Nov 2023 01:21:34 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from anubis.delphij.net (anubis.delphij.net [IPv6:2001:470:1:117::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "anubis.delphij.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SZ6656R1mz4dTH; Tue, 21 Nov 2023 01:21:33 +0000 (UTC) (envelope-from delphij@delphij.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=delphij.net header.s=y07n header.b=g++LoIkb; dkim=pass header.d=delphij.net header.s=w44o header.b=XZRGg5Ob; spf=pass (mx1.freebsd.org: domain of delphij@delphij.net designates 2001:470:1:117::25 as permitted sender) smtp.mailfrom=delphij@delphij.net; dmarc=pass (policy=reject) header.from=delphij.net DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/simple; d=delphij.net; i=@delphij.net; q=dns/txt; s=y07n; t=1700529685; h=message-id : date : mime-version : reply-to : to : from : subject : content-type : from; bh=BaIxobL84Ul7lZoUA7lMMtSfgkqK/Cl11q/pAPlBDFM=; b=g++LoIkb25CF8or4WkDCU42VIu9vGIaVYb2/5RtxRqWzhKa0I99bmB1ELzI8caqWExo82 wqHCyYpqIaLAbqeDA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=delphij.net; i=@delphij.net; q=dns/txt; s=w44o; t=1700529685; h=message-id : date : mime-version : reply-to : to : from : subject : content-type : from; bh=BaIxobL84Ul7lZoUA7lMMtSfgkqK/Cl11q/pAPlBDFM=; b=XZRGg5ObPVU7prO7/2guVk7nYZ5RzFxmBtHix1WB9hqA99vCc9eZsLasunZZ1rBYPQLwt R3UEGPIn+44o54vmvyqKOfg9lhL2g/lfFm/uGdFbVVqx9s3ZiUpjzAo2VSEfcR6IvQ4TA4c fbDfqPqMiCsEYAF9/6sZmL2gumamAqyi9lD1Ien9j5kcuiD3ndkNnRZhyuArWYOG/eS4DVA f9YEhhScEKudKHBeEEzdRZNPUNCjZI7vh8Yp+p3Se1tGY95pqBQ6lnV3iHiR9l7fffsNYjS gzOemyFmaz4AS506o3okl9S4QMMIAvWce48tze/aMXvnn9FMlkKIsTwE0n1w== Received: from xins-laptop (unknown [IPv6:2601:646:9a00:3b0a:99cb:7d01:fdec:d8e4]) by anubis.delphij.net (Postfix) with ESMTPSA id 9E8342BC85; Mon, 20 Nov 2023 17:21:25 -0800 (PST) Message-ID: Date: Mon, 20 Nov 2023 17:21:24 -0800 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Reply-To: d@delphij.net Content-Language: en-US To: freebsd-current@freebsd.org, rcm@FreeBSD.org, Warner Losh From: Xin Li Subject: HEADSUP: panic: running without device atpic requires a local APIC on UEFI systems after 0b01d45783c3 Autocrypt: addr=delphij@delphij.net; keydata= xjMEZPbDoRYJKwYBBAHaRw8BAQdAsUNmxEWz6QiGdFbBrVVEpjNpgQV9FXjDWsLsY0UwRPvN HFhpbiBMSSA8ZGVscGhpakBkZWxwaGlqLm5ldD7ClgQTFgoAPhYhBLskk2pXNatsapeNzxED 4uuXWeTFBQJk9sRMAhsDBQkKBDXmBQsJCAcDBRUKCQgLBRYCAwEAAh4FAheAAAoJEBED4uuX WeTF6yIA/2Ls3Rb/qC8mQZ6D2S0UO5vblPghJfboFJLNJFw3i4GYAQCsTmQg3ahgbNEJu/vU xgtro2kTxa6kKnZ35IbqPqPcCc44BGT2w6ESCisGAQQBl1UBBQEBB0Cxji+sQgVPajLNA/Lw yHx0ogSalPQszdkfVgeg3iR3FAMBCAfCeAQYFgoAIBYhBLskk2pXNatsapeNzxED4uuXWeTF BQJk9sOhAhsMAAoJEBED4uuXWeTF3BQBAIx/gPCTFN2DPBrKLkE3oC/+j9EkmNLMUCGidlP/ Zb6HAP4nL1kStTsOldIGhi/3m1LvU7r3Kel3MnlIK8/9BlLPAg== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------i6ZeVA2eG98NOj627td4iDEv" X-Spamd-Result: default: False [-4.89 / 15.00]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_BASE64_TEXT_BOGUS(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[delphij.net,reject]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[delphij.net:s=y07n,delphij.net:s=w44o]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; ONCE_RECEIVED(0.10)[]; MIME_BASE64_TEXT(0.10)[]; XM_UA_NO_VERSION(0.01)[]; FREEFALL_USER(0.00)[delphij]; FROM_HAS_DN(0.00)[]; HAS_REPLYTO(0.00)[d@delphij.net]; REPLYTO_DOM_EQ_FROM_DOM(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; HAS_ATTACHMENT(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[delphij.net:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4SZ6656R1mz4dTH X-Spamd-Bar: ---- This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------i6ZeVA2eG98NOj627td4iDEv Content-Type: multipart/mixed; boundary="------------9HPXI3PJPGrz399hY15BO7Ph"; protected-headers="v1" From: Xin Li Reply-To: d@delphij.net To: freebsd-current@freebsd.org, rcm@FreeBSD.org, Warner Losh Message-ID: Subject: HEADSUP: panic: running without device atpic requires a local APIC on UEFI systems after 0b01d45783c3 --------------9HPXI3PJPGrz399hY15BO7Ph Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 SGksDQoNCkl0IHNlZW1zIHRoYXQgdGhlIHJlY2VudCBpbXByb3ZlbWVudHMgb2YgQUNQSSBk ZXRlY3Rpb24gKGUwZjNkYzgyNzI3ZiANCmFuZCAwYjAxZDQ1NzgzYzMpIHdvdWxkIGxlYXZl IHRoZSBzeXN0ZW0gaW4gYW4gdW5ib290YWJsZSBzdGF0ZSBpZiB0aGUgDQpVRUZJIGZpbGVz IGFyZSBub3QgYmVpbmcgdXBkYXRlZCBhdCB0aGUgc2FtZSB0aW1lIG9mICJtYWtlIA0KaW5z dGFsbHdvcmxkIi4gIEF0IGVhcmx5IGJvb3QgdGhlIGtlcm5lbCB3b3VsZCBwYW5pYyB3aXRo Og0KDQpwYW5pYzogcnVubmluZyB3aXRob3V0IGRldmljZSBhdHBpYyByZXF1aXJlcyBhIGxv Y2FsIEFQSUMgb24gVUVGSSBzeXN0ZW1zDQoNClRvIHJlY292ZXIgYSBzeXN0ZW0gaW4gdGhp cyBzdGF0ZSwgYXQgbG9hZGVyIHByb21wdCwgdXNlOg0KDQp1bnNldCBoaW50LmFjcGkuMC5k aXNhYmxlZA0KYm9vdA0KDQooSSB0aGluayBjb3JlLmx1YSBzaG91bGQgYmUgbW9kaWZpZWQg dG8gYmUgY29tcGF0aWJsZSB3aXRoIGFuIG9sZGVyIFVFRkkgDQpwYXlsb2FkLCBwb3NzaWJs eSBpc3N1aW5nIGEgd2FybmluZyB0aGF0IGdldHMgbG9nZ2VkOyBhbmQgdGhpcyBzaG91bGQg YmUgDQptZW50aW9uZWQgaW4gVVBEQVRJTkcpDQoNCkNoZWVycywNCg== --------------9HPXI3PJPGrz399hY15BO7Ph-- --------------i6ZeVA2eG98NOj627td4iDEv Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature.asc" -----BEGIN PGP SIGNATURE----- wnsEABYIACMWIQS7JJNqVzWrbGqXjc8RA+Lrl1nkxQUCZVwGFAUDAAAAAAAKCRARA+Lrl1nkxVex AP0VNZdI1rXKCflNmDKqs99RRTXSnDYYS5KV2zcFgA6xMgEAm/guikR12IQh4cm3eWwo6JKlkcQa sRCF2Pe4G4kDiQU= =slOQ -----END PGP SIGNATURE----- --------------i6ZeVA2eG98NOj627td4iDEv-- From nobody Tue Nov 21 02:16:26 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SZ7Kj3zGlz51XFV for ; Tue, 21 Nov 2023 02:16:41 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SZ7Kh679qz3GvJ for ; Tue, 21 Nov 2023 02:16:40 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x632.google.com with SMTP id a640c23a62f3a-a002562bd8bso206692866b.0 for ; Mon, 20 Nov 2023 18:16:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1700532998; x=1701137798; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=YaqSpnp0uQeoyfayzGhzU5RoG3ncYWb61Ig8x9q35vk=; b=zd2pcyyrDfEQjKPQYmmG33CV/XzxuHluH4ks+5bPkKo0JGygeTQwWvpuqRUOpUA+vW Hwb6srTjqXBdd1Kh8CHs8Ratm23O/Vh2lVop83tAKd+fp5Zj1ISC3lSb/SJgzGsLfw2j de5kTjwS054HrMRkNeiYZiSplwCeiHGBU3+GltI12u/qwh7XhB5DbixE02XjmqiDupSL Cg5A7AYmz2UwfnI+hUYtXZuwNAHPf0BbAi/oijFbHCYX0GmYd6u9y5K3+gOrMWjfgxh/ fQ9rwFtuZnPreunAoEaH2H1+KXMIyzEcPL22f3JdbWslyYbVHbcdEYYG9pG1cIkjzGoF fE+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700532998; x=1701137798; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YaqSpnp0uQeoyfayzGhzU5RoG3ncYWb61Ig8x9q35vk=; b=i7ZBN0SJRm/l6GpMre3EdbgChWYgR4FEqQajq+h2TE88WnVvr+JizeIyXz1ITI/mpu 3RhafI1fqEbpXvg7fE46a47J3ea2ogzD/DRYu+AfegStA9CTkhLmindS1XCtpBUVR9g4 d0RcT347hr2678Nuxuuu6QjJKQHsHKWhF5mxOVyFDcJPtSEF9SiXNZFiU7j8ciHueTcv 0VfYyAEQfvw/63Y+3Acg0xLNKWvT4mFjcYpzQ2nqkjBUHN1DX55mCrxZ8mgmlDRow/nr AKZpzGn+5h37QXYPZ2zhHKv3KKk4QXYHYATFBnhOAQzvqpr8uO0UUvT2wpf905mtjvIr GdBQ== X-Gm-Message-State: AOJu0YzhBwUU9Ob0tB+arOUW3QKdCJ1vPv4FaYw/Sq8JjZ/S5KOQA7hE iuLyO0n5Ppxr7FP+0cJbrQu2OP/ilsp4pGOiw+LouA0wx44+wwRw X-Google-Smtp-Source: AGHT+IFLSOA2z1kAR/opr8YyjIUnDlLkgDiP/CGUTepxV2Wbh/pR+ZpbQDDawG0AxISz9UaE9Y4CNcLbUgtx9QC7llM= X-Received: by 2002:a17:907:b97:b0:9b2:be2f:e31a with SMTP id ey23-20020a1709070b9700b009b2be2fe31amr1110890ejc.31.1700532997832; Mon, 20 Nov 2023 18:16:37 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Mon, 20 Nov 2023 19:16:26 -0700 Message-ID: Subject: Re: HEADSUP: panic: running without device atpic requires a local APIC on UEFI systems after 0b01d45783c3 To: Xin LI Cc: FreeBSD Current , rcm@freebsd.org Content-Type: multipart/alternative; boundary="00000000000043ce5c060aa0311e" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4SZ7Kh679qz3GvJ --00000000000043ce5c060aa0311e Content-Type: text/plain; charset="UTF-8" I have a patch cooking for this... I was almost done when I had to leave for karate. Warner On Mon, Nov 20, 2023, 6:21 PM Xin Li wrote: > Hi, > > It seems that the recent improvements of ACPI detection (e0f3dc82727f > and 0b01d45783c3) would leave the system in an unbootable state if the > UEFI files are not being updated at the same time of "make > installworld". At early boot the kernel would panic with: > > panic: running without device atpic requires a local APIC on UEFI systems > > To recover a system in this state, at loader prompt, use: > > unset hint.acpi.0.disabled > boot > > (I think core.lua should be modified to be compatible with an older UEFI > payload, possibly issuing a warning that gets logged; and this should be > mentioned in UPDATING) > > Cheers, > --00000000000043ce5c060aa0311e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable --00000000000043ce5c060aa0311e-- From nobody Tue Nov 21 03:33:29 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SZ92b6Hxjz51fjW for ; Tue, 21 Nov 2023 03:33:43 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-lf1-x129.google.com (mail-lf1-x129.google.com [IPv6:2a00:1450:4864:20::129]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SZ92b4S8Rz3R7X for ; Tue, 21 Nov 2023 03:33:43 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x129.google.com with SMTP id 2adb3069b0e04-5098e423ba2so7122094e87.2 for ; Mon, 20 Nov 2023 19:33:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1700537620; x=1701142420; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=BotDw3yRO3dcMRFaD3UZxnOVZecWLTOEColpPfvQcjA=; b=kq/DdG1vdKiRWehSqSdyWnaelDhxZIwRxRCH28BxtJcwghT3fGrGrz/EPpB9GHjL1b suCAweVtVPBLzfLzRGwIpCy9gH5JMSH02vC4Jf5YyYvJZu5Qu5TwgTvK4r16/q4EeFta 9km2718m6sk7CIbSfCIJpYOUu9i8/gXUHONyP5GQgZcxubDY0JJYpjENTIWY/86qmtJq x36dLKhWpiA6g4HOefmX0RgnLrGY3P3eYu45qwV65eg4FJdk58m/GHe335/YfReING/X Zq1e+6QPd/mJW9yMfWHTixLauAc/F7VxuTw5dhHkUEJIpx7dvBSMtUZJd2ObzgjACJjX 6w1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700537620; x=1701142420; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=BotDw3yRO3dcMRFaD3UZxnOVZecWLTOEColpPfvQcjA=; b=SQGZeKnt7r52yEV8WWKXk9WpmOygYPVo7Dv/DdWmr0VZ9iYlu7rliCv12k3JYW/rmu /fAudm7C5mBYUxD7D5T2J03pgUqsmNpSwAR3nuL+qBcvzEgasjmpCNTOzhwR/VaFnYgo uz3ieMqRxVX3Kjk9cbhO873ITNTz16+A7ziPXSQ56GRwi8RCzmcnr0YNQChwYMxIJmkg JPt/ya6OAsDNyY4vafhh8Mzesfudk53ep7DIqJjVubITxVGCRPM4IFsfVO5ivfxq8eqm 3Z7CSeG5wx2nuZqYLg4o9HZ9baO/KTKcgfwl6JhfNv1HxENtO5phTlPNcLrnVA4tTnHf zYAw== X-Gm-Message-State: AOJu0YzyIrUpMwvbcfofXT4/DO5ypFF1QoW5S0aVhZRvV1OtrA4Xx01U LsBF6JBbsWktOo3UuyQq9hmd+KyXaWUT5fr2Ajj2ePTKIY7n+0GGWR4= X-Google-Smtp-Source: AGHT+IEhivOWUMm1qqRnowgdr5fanSzBOsjwPS9iEYWZ65QqTfXe8lt9NWlJI6SqTnUtJkErivpXwVBAtWM0gUrgJU0= X-Received: by 2002:ac2:434a:0:b0:509:39e3:5268 with SMTP id o10-20020ac2434a000000b0050939e35268mr5000282lfl.59.1700537620290; Mon, 20 Nov 2023 19:33:40 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Mon, 20 Nov 2023 20:33:29 -0700 Message-ID: Subject: Re: HEADSUP: panic: running without device atpic requires a local APIC on UEFI systems after 0b01d45783c3 To: d@delphij.net Cc: freebsd-current@freebsd.org, rcm@freebsd.org Content-Type: multipart/alternative; boundary="000000000000c8fccd060aa144fd" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4SZ92b4S8Rz3R7X --000000000000c8fccd060aa144fd Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Nov 20, 2023 at 6:21=E2=80=AFPM Xin Li wrote: > Hi, > > It seems that the recent improvements of ACPI detection (e0f3dc82727f > and 0b01d45783c3) would leave the system in an unbootable state if the > UEFI files are not being updated at the same time of "make > installworld". At early boot the kernel would panic with: > > panic: running without device atpic requires a local APIC on UEFI systems > > To recover a system in this state, at loader prompt, use: > > unset hint.acpi.0.disabled > boot > > (I think core.lua should be modified to be compatible with an older UEFI > payload, possibly issuing a warning that gets logged; and this should be > mentioned in UPDATING) > I just pushed https://cgit.freebsd.org/src/commit/?id=3Df213da893ca8c7c76e1656b36d3a10f93= f9a1760 which should fix the issue for x86, with an UPDATING entry for aarch64. This is at best a stop-gap kludge. The real solution would be for loader.efi to publish a list of interfaces it implements and then the lua code can cope with old/new better. Warner --000000000000c8fccd060aa144fd Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Nov 20, 2023 at 6:21=E2=80=AF= PM Xin Li <delphij@delphij.net> wrote:
Hi,=

It seems that the recent improvements of ACPI detection (e0f3dc82727f
and 0b01d45783c3) would leave the system in an unbootable state if the
UEFI files are not being updated at the same time of "make
installworld".=C2=A0 At early boot the kernel would panic with:

panic: running without device atpic requires a local APIC on UEFI systems
To recover a system in this state, at loader prompt, use:

unset hint.acpi.0.disabled
boot

(I think core.lua should be modified to be compatible with an older UEFI payload, possibly issuing a warning that gets logged; and this should be mentioned in UPDATING)

I just pushed=C2= =A0https://cgit.freebsd.org/src/commit/?id=3Df213da893c= a8c7c76e1656b36d3a10f93f9a1760 which should fix the issue for x86, with= an UPDATING entry for aarch64.

This is at best a = stop-gap kludge. The real solution would be for loader.efi to publish a lis= t of interfaces it implements and then the lua code can cope with old/new b= etter.

Warner=C2=A0
--000000000000c8fccd060aa144fd-- From nobody Tue Nov 21 03:47:59 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SZ9M62Kdmz51ggT for ; Tue, 21 Nov 2023 03:48:02 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from anubis.delphij.net (anubis.delphij.net [IPv6:2001:470:1:117::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "anubis.delphij.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SZ9M605hXz3Sg7; Tue, 21 Nov 2023 03:48:02 +0000 (UTC) (envelope-from delphij@delphij.net) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/simple; d=delphij.net; i=@delphij.net; q=dns/txt; s=y07n; t=1700538480; h=message-id : date : mime-version : reply-to : subject : to : cc : references : from : in-reply-to : content-type : from; bh=jtEwT4CBQDrGn+IsmPN1fAVBLP7yQTm6Ivbb8bg/PuM=; b=n04Y5t6L/nFK0ngj56vMwzbIAA+AlNWry3AmlSntSzGSl35cE82qbXekmXxzDotAUqZsb //XdxHxEILg8BzFCA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=delphij.net; i=@delphij.net; q=dns/txt; s=w44o; t=1700538480; h=message-id : date : mime-version : reply-to : subject : to : cc : references : from : in-reply-to : content-type : from; bh=jtEwT4CBQDrGn+IsmPN1fAVBLP7yQTm6Ivbb8bg/PuM=; b=BMjDqJoorPc0t59PSU0OtYrTcJ8Cpla6CbAnCYNAL6Eg/nWZr03cHnji89rjih7yPxycZ 4LNT9RPB7/b7xwoizwEptLwNQq1lpl0dUtOKlJQrlQ3I+zBUSk7kihq7EZYT5mLft+8a40o dW4g7p9oZv+mQiNG4tBgcfTTBkBVcziDRlCHnYVtyu/W0bYIYanQ+2zxXQJHbOKz+zX9OP+ 0jZNjgoknhcbA0LyIfwJs047BowFWN8HXQ3R0huIJiHODu5OLm/6OT06rI86MPDjWZ4jwOd zYwDYXhQHNp3iVyrNpR7dc+MPynuANi/k/YFvomTQ6QtA7F+fawpHE+T+WDQ== Received: from xins-laptop (unknown [IPv6:2601:646:9a00:3b0a:99cb:7d01:fdec:d8e4]) by anubis.delphij.net (Postfix) with ESMTPSA id 2EC1F2B969; Mon, 20 Nov 2023 19:48:00 -0800 (PST) Message-ID: <303be2a6-5b9b-4079-919e-03abbe33c98d@delphij.net> Date: Mon, 20 Nov 2023 19:47:59 -0800 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Reply-To: d@delphij.net Subject: Re: HEADSUP: panic: running without device atpic requires a local APIC on UEFI systems after 0b01d45783c3 Content-Language: en-US To: Warner Losh , d@delphij.net Cc: freebsd-current@freebsd.org, rcm@freebsd.org References: From: Xin Li Autocrypt: addr=delphij@delphij.net; keydata= xjMEZPbDoRYJKwYBBAHaRw8BAQdAsUNmxEWz6QiGdFbBrVVEpjNpgQV9FXjDWsLsY0UwRPvN HFhpbiBMSSA8ZGVscGhpakBkZWxwaGlqLm5ldD7ClgQTFgoAPhYhBLskk2pXNatsapeNzxED 4uuXWeTFBQJk9sRMAhsDBQkKBDXmBQsJCAcDBRUKCQgLBRYCAwEAAh4FAheAAAoJEBED4uuX WeTF6yIA/2Ls3Rb/qC8mQZ6D2S0UO5vblPghJfboFJLNJFw3i4GYAQCsTmQg3ahgbNEJu/vU xgtro2kTxa6kKnZ35IbqPqPcCc44BGT2w6ESCisGAQQBl1UBBQEBB0Cxji+sQgVPajLNA/Lw yHx0ogSalPQszdkfVgeg3iR3FAMBCAfCeAQYFgoAIBYhBLskk2pXNatsapeNzxED4uuXWeTF BQJk9sOhAhsMAAoJEBED4uuXWeTF3BQBAIx/gPCTFN2DPBrKLkE3oC/+j9EkmNLMUCGidlP/ Zb6HAP4nL1kStTsOldIGhi/3m1LvU7r3Kel3MnlIK8/9BlLPAg== In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------Q6qtFCrYc53I0DDD2eNTXIfT" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Queue-Id: 4SZ9M605hXz3Sg7 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------Q6qtFCrYc53I0DDD2eNTXIfT Content-Type: multipart/mixed; boundary="------------XgWchPOlg21ekKSBaPNlIJ4s"; protected-headers="v1" From: Xin Li Reply-To: d@delphij.net To: Warner Losh , d@delphij.net Cc: freebsd-current@freebsd.org, rcm@freebsd.org Message-ID: <303be2a6-5b9b-4079-919e-03abbe33c98d@delphij.net> Subject: Re: HEADSUP: panic: running without device atpic requires a local APIC on UEFI systems after 0b01d45783c3 References: In-Reply-To: --------------XgWchPOlg21ekKSBaPNlIJ4s Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 T24gMjAyMy0xMS0yMCAxOTozMywgV2FybmVyIExvc2ggd3JvdGU6DQo+IA0KPiANCj4gT24g TW9uLCBOb3YgMjAsIDIwMjMgYXQgNjoyMeKAr1BNIFhpbiBMaSA8ZGVscGhpakBkZWxwaGlq Lm5ldCANCj4gPG1haWx0bzpkZWxwaGlqQGRlbHBoaWoubmV0Pj4gd3JvdGU6DQo+IA0KPiAg ICAgSGksDQo+IA0KPiAgICAgSXQgc2VlbXMgdGhhdCB0aGUgcmVjZW50IGltcHJvdmVtZW50 cyBvZiBBQ1BJIGRldGVjdGlvbiAoZTBmM2RjODI3MjdmDQo+ICAgICBhbmQgMGIwMWQ0NTc4 M2MzKSB3b3VsZCBsZWF2ZSB0aGUgc3lzdGVtIGluIGFuIHVuYm9vdGFibGUgc3RhdGUgaWYg dGhlDQo+ICAgICBVRUZJIGZpbGVzIGFyZSBub3QgYmVpbmcgdXBkYXRlZCBhdCB0aGUgc2Ft ZSB0aW1lIG9mICJtYWtlDQo+ICAgICBpbnN0YWxsd29ybGQiLsKgIEF0IGVhcmx5IGJvb3Qg dGhlIGtlcm5lbCB3b3VsZCBwYW5pYyB3aXRoOg0KPiANCj4gICAgIHBhbmljOiBydW5uaW5n IHdpdGhvdXQgZGV2aWNlIGF0cGljIHJlcXVpcmVzIGEgbG9jYWwgQVBJQyBvbiBVRUZJDQo+ ICAgICBzeXN0ZW1zDQo+IA0KPiAgICAgVG8gcmVjb3ZlciBhIHN5c3RlbSBpbiB0aGlzIHN0 YXRlLCBhdCBsb2FkZXIgcHJvbXB0LCB1c2U6DQo+IA0KPiAgICAgdW5zZXQgaGludC5hY3Bp LjAuZGlzYWJsZWQNCj4gICAgIGJvb3QNCj4gDQo+ICAgICAoSSB0aGluayBjb3JlLmx1YSBz aG91bGQgYmUgbW9kaWZpZWQgdG8gYmUgY29tcGF0aWJsZSB3aXRoIGFuIG9sZGVyDQo+ICAg ICBVRUZJDQo+ICAgICBwYXlsb2FkLCBwb3NzaWJseSBpc3N1aW5nIGEgd2FybmluZyB0aGF0 IGdldHMgbG9nZ2VkOyBhbmQgdGhpcw0KPiAgICAgc2hvdWxkIGJlDQo+ICAgICBtZW50aW9u ZWQgaW4gVVBEQVRJTkcpDQo+IA0KPiANCj4gSSBqdXN0IHB1c2hlZCANCj4gaHR0cHM6Ly9j Z2l0LmZyZWVic2Qub3JnL3NyYy9jb21taXQvP2lkPWYyMTNkYTg5M2NhOGM3Yzc2ZTE2NTZi MzZkM2ExMGY5M2Y5YTE3NjAgPGh0dHBzOi8vY2dpdC5mcmVlYnNkLm9yZy9zcmMvY29tbWl0 Lz9pZD1mMjEzZGE4OTNjYThjN2M3NmUxNjU2YjM2ZDNhMTBmOTNmOWExNzYwPiB3aGljaCBz aG91bGQgZml4IHRoZSBpc3N1ZSBmb3IgeDg2LCB3aXRoIGFuIFVQREFUSU5HIGVudHJ5IGZv ciBhYXJjaDY0Lg0KPiANCj4gVGhpcyBpcyBhdCBiZXN0IGEgc3RvcC1nYXAga2x1ZGdlLiBU aGUgcmVhbCBzb2x1dGlvbiB3b3VsZCBiZSBmb3IgDQo+IGxvYWRlci5lZmkgdG8gcHVibGlz aCBhIGxpc3Qgb2YgaW50ZXJmYWNlcyBpdCBpbXBsZW1lbnRzIGFuZCB0aGVuIHRoZSANCj4g bHVhIGNvZGUgY2FuIGNvcGUgd2l0aCBvbGQvbmV3IGJldHRlci4NCg0KWWVhaCBJIHRoaW5r IGl0IChJIGFzc3VtZSB5b3UgbWVhbiANCmh0dHBzOi8vY2dpdC5mcmVlYnNkLm9yZy9zcmMv Y29tbWl0Lz9pZD0wYWJlMDVhZWFjMjlkOTk3ODY0MDFiOTA3OGU5N2RjZWFkMzVmN2YzIA0K KSBzaG91bGQgYmUgc3VmZmljaWVudCBmb3IgeDg2IHN5c3RlbXMgdG8gYm9vdC4gIFRoYW5r cyENCg0KQ2hlZXJzLA0KDQo= --------------XgWchPOlg21ekKSBaPNlIJ4s-- --------------Q6qtFCrYc53I0DDD2eNTXIfT Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature.asc" -----BEGIN PGP SIGNATURE----- wnsEABYIACMWIQS7JJNqVzWrbGqXjc8RA+Lrl1nkxQUCZVwobwUDAAAAAAAKCRARA+Lrl1nkxffo AP9e0W/AvxUewzw8e2zKmMne/YIapaYRAONrHSfnMRIqqgEAxI2XFm7iw/IgWqStpKBGveE+/Qp5 TeasAUSSRuOKkQQ= =pV7Y -----END PGP SIGNATURE----- --------------Q6qtFCrYc53I0DDD2eNTXIfT-- From nobody Tue Nov 21 03:49:07 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SZ9Nb4kctz51gkf for ; Tue, 21 Nov 2023 03:49:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SZ9Nb2sbxz3Tss for ; Tue, 21 Nov 2023 03:49:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x52b.google.com with SMTP id 4fb4d7f45d1cf-548a2c20f50so2822883a12.1 for ; Mon, 20 Nov 2023 19:49:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1700538558; x=1701143358; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=pYTs3e9R+MZL9HwMjxHXzNGCRRo1we8umCCfjex3OtU=; b=hIh2m0bRl749Uea+COtiHchGHChbLbcQeQpnaWDUcmfl9ud0k/wO1hPC7hiUx2p3AP bdzgLk6gUgHzl2XajIfCCH+uck7ycjAfniztXYCqk0pUeqI4wQjReua7jzNQWUSH1SHI WBWy563Acie0jliouuNPQ4F86OVIOb0g5w8oebONvpTy/kdjBwe0tnBJ8aH8g/16uUBA jSEDQiw8zL+l57goJiAh/evQKF/1iz7jDp65dWv2EzZuCsRU4f1IjYSOIs6kck7Tnkn3 bDXLEIhmZd7dOHjCI5DLVIJi5yNtC+kKV9VWeLa9vJDvQEaRX6RIQCe71o1OpuVs1I4R we/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700538558; x=1701143358; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pYTs3e9R+MZL9HwMjxHXzNGCRRo1we8umCCfjex3OtU=; b=VmyPW4FMjMTu9s5SJmT4SAd2DiRZs91LS+17C6KCqvQnNpu4t6MuiQlnJnpMpnKc6+ uNHePqFsRZizcYxzE/doPhL3h6z26gtX93E9UzQSv8S621YKJ2+hbeeOVNf2SuPexj5u XPb2aKS3emmo8un1pX3Slyi537AhcuCtQMjm1IHhnJfcQ5Vsx4S3tuUn1AGF8TnJL0S2 ifUsckfJlOIHcHMPDompio4SKfOzdz7t8bSqPXEkBqimF+6Xc+OXCIsqQFihxFRV4Vk5 AipGR5VGLIRBNsCLbK2MB6R8UPCBjpO9U82mY2srJYhC8U++NOqkMHdKW3LfQJUJF9XO JWiA== X-Gm-Message-State: AOJu0Yy12+Y4W1M/QOtmgQVHhPQdjQveULjX4VH/ph1+ldA9cAc6+yBn mrefhUkYIEofIvPD4FKO7HIlQFp0da1dAru4ek3AHPcuL8kv+d3r X-Google-Smtp-Source: AGHT+IHt1w8/5Rzn7NbeEcGW3Jx4vwtPf8mFFr013fZOnRPRFBVWvU/7AeHJbZk0aRBMbGo++89NynRYjS7ZZUJlB7I= X-Received: by 2002:a50:ed08:0:b0:548:694c:fc52 with SMTP id j8-20020a50ed08000000b00548694cfc52mr784846eds.26.1700538557957; Mon, 20 Nov 2023 19:49:17 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <303be2a6-5b9b-4079-919e-03abbe33c98d@delphij.net> In-Reply-To: <303be2a6-5b9b-4079-919e-03abbe33c98d@delphij.net> From: Warner Losh Date: Mon, 20 Nov 2023 20:49:07 -0700 Message-ID: Subject: Re: HEADSUP: panic: running without device atpic requires a local APIC on UEFI systems after 0b01d45783c3 To: d@delphij.net Cc: freebsd-current@freebsd.org, rcm@freebsd.org Content-Type: multipart/alternative; boundary="000000000000ac98f2060aa17c93" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4SZ9Nb2sbxz3Tss --000000000000ac98f2060aa17c93 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Nov 20, 2023 at 8:48=E2=80=AFPM Xin Li wrote: > On 2023-11-20 19:33, Warner Losh wrote: > > > > > > On Mon, Nov 20, 2023 at 6:21=E2=80=AFPM Xin Li > > wrote: > > > > Hi, > > > > It seems that the recent improvements of ACPI detection (e0f3dc8272= 7f > > and 0b01d45783c3) would leave the system in an unbootable state if > the > > UEFI files are not being updated at the same time of "make > > installworld". At early boot the kernel would panic with: > > > > panic: running without device atpic requires a local APIC on UEFI > > systems > > > > To recover a system in this state, at loader prompt, use: > > > > unset hint.acpi.0.disabled > > boot > > > > (I think core.lua should be modified to be compatible with an older > > UEFI > > payload, possibly issuing a warning that gets logged; and this > > should be > > mentioned in UPDATING) > > > > > > I just pushed > > > https://cgit.freebsd.org/src/commit/?id=3Df213da893ca8c7c76e1656b36d3a10f= 93f9a1760 > < > https://cgit.freebsd.org/src/commit/?id=3Df213da893ca8c7c76e1656b36d3a10f= 93f9a1760> > which should fix the issue for x86, with an UPDATING entry for aarch64. > > > > This is at best a stop-gap kludge. The real solution would be for > > loader.efi to publish a list of interfaces it implements and then the > > lua code can cope with old/new better. > > Yeah I think it (I assume you mean > > https://cgit.freebsd.org/src/commit/?id=3D0abe05aeac29d99786401b9078e97dc= ead35f7f3 > ) should be sufficient for x86 systems to boot. Thanks! > Yes. Clicked on the wrong commit. Kyle and I will come up with something to allow easier transitions in the future. Warner > Cheers, > > --000000000000ac98f2060aa17c93 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Nov 20, 2023 at 8:48=E2=80=AF= PM Xin Li <delphij@delphij.net> wrote:
On = 2023-11-20 19:33, Warner Losh wrote:
>
>
> On Mon, Nov 20, 2023 at 6:21=E2=80=AFPM Xin Li <delphij@delphij.net
> <mailto:de= lphij@delphij.net>> wrote:
>
>=C2=A0 =C2=A0 =C2=A0Hi,
>
>=C2=A0 =C2=A0 =C2=A0It seems that the recent improvements of ACPI detec= tion (e0f3dc82727f
>=C2=A0 =C2=A0 =C2=A0and 0b01d45783c3) would leave the system in an unbo= otable state if the
>=C2=A0 =C2=A0 =C2=A0UEFI files are not being updated at the same time o= f "make
>=C2=A0 =C2=A0 =C2=A0installworld".=C2=A0 At early boot the kernel = would panic with:
>
>=C2=A0 =C2=A0 =C2=A0panic: running without device atpic requires a loca= l APIC on UEFI
>=C2=A0 =C2=A0 =C2=A0systems
>
>=C2=A0 =C2=A0 =C2=A0To recover a system in this state, at loader prompt= , use:
>
>=C2=A0 =C2=A0 =C2=A0unset hint.acpi.0.disabled
>=C2=A0 =C2=A0 =C2=A0boot
>
>=C2=A0 =C2=A0 =C2=A0(I think core.lua should be modified to be compatib= le with an older
>=C2=A0 =C2=A0 =C2=A0UEFI
>=C2=A0 =C2=A0 =C2=A0payload, possibly issuing a warning that gets logge= d; and this
>=C2=A0 =C2=A0 =C2=A0should be
>=C2=A0 =C2=A0 =C2=A0mentioned in UPDATING)
>
>
> I just pushed
> https://cgit.= freebsd.org/src/commit/?id=3Df213da893ca8c7c76e1656b36d3a10f93f9a1760 &= lt;https://cgit.fr= eebsd.org/src/commit/?id=3Df213da893ca8c7c76e1656b36d3a10f93f9a1760>= which should fix the issue for x86, with an UPDATING entry for aarch64. >
> This is at best a stop-gap kludge. The real solution would be for
> loader.efi to publish a list of interfaces it implements and then the =
> lua code can cope with old/new better.

Yeah I think it (I assume you mean
https://cgit.freeb= sd.org/src/commit/?id=3D0abe05aeac29d99786401b9078e97dcead35f7f3
) should be sufficient for x86 systems to boot.=C2=A0 Thanks!

Yes. Clicked on the wrong commit.=C2=A0 Kyle and I w= ill come up with something to allow easier transitions in the future.
=

Warner
=C2=A0
Cheers,

--000000000000ac98f2060aa17c93-- From nobody Tue Nov 21 11:46:12 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SZMz31Lc2z51SbG for ; Tue, 21 Nov 2023 11:46:23 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SZMz16HQ4z3PMV for ; Tue, 21 Nov 2023 11:46:21 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of phk@critter.freebsd.dk designates 130.225.244.222 as permitted sender) smtp.mailfrom=phk@critter.freebsd.dk; dmarc=none Received: from critter.freebsd.dk (unknown [192.168.55.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by phk.freebsd.dk (Postfix) with ESMTPS id 4231E8928F for ; Tue, 21 Nov 2023 11:46:13 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.17.1/8.16.1) with ESMTPS id 3ALBkC9m079789 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Tue, 21 Nov 2023 11:46:12 GMT (envelope-from phk@critter.freebsd.dk) Received: (from phk@localhost) by critter.freebsd.dk (8.17.1/8.16.1/Submit) id 3ALBkCSA079788; Tue, 21 Nov 2023 11:46:12 GMT (envelope-from phk) Message-Id: <202311211146.3ALBkCSA079788@critter.freebsd.dk> To: current@freebsd.org Subject: bl[ao]cklistd, some first impressions... From: Poul-Henning Kamp List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-ID: <79786.1700567172.1@critter.freebsd.dk> Content-Transfer-Encoding: quoted-printable Date: Tue, 21 Nov 2023 11:46:12 +0000 X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; FORGED_SENDER(0.30)[phk@phk.freebsd.dk,phk@critter.freebsd.dk]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; MLMMJ_DEST(0.00)[current@freebsd.org]; DMARC_NA(0.00)[freebsd.dk]; R_DKIM_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FREEFALL_USER(0.00)[phk]; ARC_NA(0.00)[]; ASN(0.00)[asn:1835, ipnet:130.225.0.0/16, country:EU]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[current@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; FROM_NEQ_ENVFROM(0.00)[phk@phk.freebsd.dk,phk@critter.freebsd.dk] X-Rspamd-Queue-Id: 4SZMz16HQ4z3PMV X-Spamd-Bar: -- I have played around with bl[ao]cklistd on 13.2 and=C2=A0I am not terribly= impressed: A) It would be nice to be able to specify in bl[ao]cklistd.conf that violations of a particular rule should get the IP banned for any trafic, not just the one they happened to trigger first. This is particular necessary/useful for instance when you run a "HTTP-sandwich", and spot the problem in the inner layers, but want to block access to the exterior port 443 from a process which does not hold that socket. (ie: varnish behind hitch) B) The OaM commands for pf take obscurity to a new level: pfctl -a blacklistd/80 -sr -v pfctl -a blacklistd/80 -t port80 -T show and that is just for one specific port: repeat manually for all of 22, 25, 443 etc. Going to a default "totally block IP's" policy would simplify this. C) Anchor-wildcards do not work in pfctl and the diagnostics are criminal: root@test-net:~ # pfctl -a 'blacklistd/*' -t port80 -T show pfctl: Unknown error: -1. root@test-net:~ # pfctl -t port80 -T show pfctl: Unknown error: -1. D) It would be nice to be able to have bl[ao]cklistd cooperate across machines, so that for instance all VM's on the same hardware could benefit from each others detections. E) It should be possible to inject an entry with bl[ao]cklistctl, so that simple text-based processing of logfiles can contribute too. Other than that, once you get it set up right, it seems to get the job don= e... -- = Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence= . From nobody Tue Nov 21 12:27:25 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SZNtY3FX2z51WMx for ; Tue, 21 Nov 2023 12:27:33 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from mx0.gid.co.uk (mx0.gid.co.uk [194.32.164.250]) by mx1.freebsd.org (Postfix) with ESMTP id 4SZNtX5wJlz3SVj for ; Tue, 21 Nov 2023 12:27:32 +0000 (UTC) (envelope-from rb@gid.co.uk) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (gw.br-thn-01.caladan.net.uk [80.71.4.69] (may be forged)) by mx0.gid.co.uk (8.14.2/8.14.2) with ESMTP id 3ALCRPX6057048; Tue, 21 Nov 2023 12:27:25 GMT (envelope-from rb@gid.co.uk) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.4\)) Subject: Re: bl[ao]cklistd, some first impressions... From: Bob Bishop In-Reply-To: <202311211146.3ALBkCSA079788@critter.freebsd.dk> Date: Tue, 21 Nov 2023 12:27:25 +0000 Cc: "current@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <79D1DBEE-06AD-4109-8A34-DD46A4805EDC@gid.co.uk> References: <202311211146.3ALBkCSA079788@critter.freebsd.dk> To: Poul-Henning Kamp X-Mailer: Apple Mail (2.3696.120.41.1.4) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:42831, ipnet:194.32.164.0/24, country:GB] X-Rspamd-Queue-Id: 4SZNtX5wJlz3SVj Hi, > On 21 Nov 2023, at 11:46, Poul-Henning Kamp = wrote: >=20 > I have played around with bl[ao]cklistd on 13.2 and I am not terribly = impressed: > [etc] We came up with a similar list of objections and built our own solution. -- Bob Bishop rb@gid.co.uk From nobody Wed Nov 22 05:43:50 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SZqtb5c4xz51sPN for ; Wed, 22 Nov 2023 05:44:07 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-8.consmr.mail.gq1.yahoo.com (sonic316-8.consmr.mail.gq1.yahoo.com [98.137.69.32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SZqtZ6W3wz4NWb for ; Wed, 22 Nov 2023 05:44:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=rOMLLKar; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.32 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1700631843; bh=sFEx/Ox+xeucOmZ/t7J7w5A97oGYKI8jj9WQ07lQuos=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=rOMLLKarsdc6xk//FNObdy9G/eOV1Q4r3OuhTiFdI+aJki5UXRx2bB8Y72HxPyrIUeBNBj2ptqM8OcgivZG8MU8EtEb6cMHKwY4Wk3Qaq2UFbBwa29WSUmD0ElqJ/fyZUChQGf/QT6bBaMHSuWjZgEgZ5tqcYc53wuuExwBjkkT75dEjvk826hKCPx58ZGnSS/95r/og1kJLXlND+dBS8qr+8CcahgLpsJZxCoiUmewW9U1YGRhEpYFwirukS94WyZKzCd6NGy/TvYiIz7k93xIwWBfynHehsuCdkY5cx02eY+niOj6QGDpGnQTgy94h9jS1jS7VieoUHxHRRYkL2g== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1700631843; bh=7th3hD+N9zw8qNX53FzbtPZosphj5vLlefFFWJ3tkhX=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=CKPigrJZ1T4EOKwhOzpiGIAZvzxoFUzgk0/wLCRTix6Gg8G4oqvEJKBnw0KpiBS0Au4EP+OdQtZpnyM7+eEMwaXF1NIjl3ZX2dWAPItzBosH8SF0PUOWwO/eyGa+A7sOzJ9ElzHHopms3rAOFhXCUkf1NSgc9mRA4NOAXamfM2HL21UJq3cV5SRIXcWBrkwJl0GCABt46wqtwCNcjZ9gUdq9T6SH0tmB3a/8A2l1UpA1wMxy3qeq/rE9+FmBwV72Upg0OA6k40fotxDvuIND25Y+G+A1EkDiJN6AXKREPUofVsiqy+34nuCI3CEsBqUaRrYhpCD3n+fH4ds9HAGfNg== X-YMail-OSG: X.mCi3cVM1npQzEMbXr3g1F_ge5rIOp.QhgA97YxghRyyznhMFN5eQhOocrmbW1 VKXD0WqHez7Znila4t4a2ascllcrkXXZJVb46egwMoyYm_B..g.8jQMzjK2oBdq2nu.R1eu5PTPg V1qArGu9oocy1pzcjpWvAyrP7CP7p08PewB0bPaKsZqZ1vEwmvCobY9zq0QfHTFxEeorQptVIN7B eMTkqx8cMWP6980QfKzj5fO8Hb7Xf7goFA7RpuHg.c0J3GisfA_4eCbNOY3zQMZayBIHak5f1vA8 NsKIe9E1F82RGD20PNqSd2QGahTzgZxsi4SsJsK6hBqRwhLt8WVE0LMa6geBlu1PTohOjeEBGisY BWiC0i3Vv_yZRbMbN_7CfNGyZGjR2_gky6MUDocA6Nd2XZ01w9NE9Zg50cfrSF__mtpvL19gjNql qjrPnjmjUBfQ_2IzRidPVS9fe.7fPnAVyKMrVB2DUH2DqHVcAH6Fk28d2Ue9lxDwlhEOGd4O1qhb XwrClOe52lNFmYT4Oi_AvdJ1KUsn2kCzL66g0U_zcJGTo_fgcnomZ_NlY9NHJDNM4yy7JFxY_E1B Sj7upAiewHLY3fgtZN2DfblYf2vD4uIRU29ApyvdXt9FFsn39pDS1iDcOCbcCGH96pTFcMcefwoV 5Iw_NAt9ZFaCs8VV_EGJgiSbAtIi.MeFddHj0EwUCcPa.Bwdq3Qmo6SF4BfN2uUD5EIpCrwuy4ue HeQHP9GkR3nRLWsgANxlbkG_naLVEVQu5snZb.urc5yOeKVGwUc4FJ75xSCi64VdNy129yAiAd3c vikPlKUQYU99d9oe7v53SMCj_XOJ3yxSD_9vMV_nyw89P8AXod9iyVqe0JElftVu2lPT61Hs_oaB T5TJmvNMnC7cTrd7fR0ERLcrDLIxPbpfNxAmCWjArkY0Cvkf.AY_kTaHfEMyNQ63HEPk88fxCTW8 tk0KvAo2ukBgxuQx4.lk4WJf4r6SPwk5_j0rOYtd.D6XNuVO15jDbN3KjDbH9Imhk5OELKvuz4gL l9fTJ1Qf5B7SSUeFkLobu37n6C1DTMxK5g2NS_VklRihDsUMjFOvNvoWfE3Nv20T38QG63S6y4kh yL2liaOn2W1KLlTHbYf3Fiv_rYDqm.f7wzxjcOfzXqAMuyzHCaVauyNeWWgfnOVMyZhxgSLvvP9N sfK5UyjS6HpG12QFk.B_3niQ8WT6fmil1TTFN0DGtICmrMz38caXhnK0sTVcUvM9DSaqECxsejOC t5MtWwJHooOhntAlKatywGgZs.DyLtZyK4Mw22tPicPAkAT6WM_L._vdDjNd3vHQ2Yx_ZDA8pLMK 2o1tjJILCxootmXY32fu6DWvvSAeL77o4DkQR2zLapQhNHp_qYo5i.XoCt1ENLgWTEjlmTKmm_rS 3YhGlBTGto.LpVia.ll.O1_ZQUHB_JJrIqqC71Bg6SZZWhbl8ARhBvKca6sn5pP4PiJYcLKaVSFM ir8VT_aJhLzatKz4ghuAq8b.E9CCExyRa1flbcOL7evh.kdscW0YuuaNgrsTLcDdTUS8P0GybL3b UQhWE3oy4jF7r_NJvjIwTZ3W433aIiPK06cZ2odRwImxnxIz06YtxWsJ4yYaamrybqbWNWEuUX_E YT0ayEks4ElIPV4L78LVV9sflVJy74nhTpOfK1sV7ofyVznFXQ7L3hA4m213hcEGEDghcjZ7JYk_ ibiXxnafM0LgKAHVQYNJv67ili1t.x2NsU_gcW39CNFZ.Z.jnpian8gCe7B_lCK85Ryj4Kwd8Pvc fpWm_MGVpOLha1Or0fbKqHCCWmJmU1CiNSmOQiqTnvjw_XCokIKVy.D3nmRyatJOw8x9tGr1DfMP IDDjPBNW3c3anoXaFMw0PPYDxQr6DYMHAx.IFPB9t77XbVnwPmzEiFGD9OKGYXLNl55DI7yl61mR WHwez6YciP6WLYrVEHu_lFpYO.G81dwl67NtajwGd6LjOo3UeoXw0kDKYK.5.reoS3X1tzfKuLOf 9SnL1p.91217uaTmDuKAR8aUqd2_0ubxOeUDkSLXP3STGlZ0Cbh0QD7eucVI85IXB9PtIhnivCLG eW6E6BhmrjT_rZSRNWZnxX7Bb5O.yqoZ8bhrhbzqjmSaZ.ioML.Hadp9kNcLqXlWj.eQwVXA46fN rUGhLxmx01mezMIzZScZQczkhTT.87JiN1.YmzwlIZhBKw9dDmfusRl_DxEB0_pBlVIfUa.JCMU. oWPH2Kge1Hyp1Lf1V7isOmSU.Hdr0L34EUfWSC4oap.AEcl9API5Tm71xeykA.pM- X-Sonic-MF: X-Sonic-ID: 2fe6e20a-f0c7-4de5-9795-a227eff5e784 Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Wed, 22 Nov 2023 05:44:03 +0000 Received: by hermes--production-gq1-6775bfb8fc-57kw2 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 70ee2b3a8508f3fb00c68955646ed6f8; Wed, 22 Nov 2023 05:44:01 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\)) Subject: "options MAXMEMDOM=2" vs. amd64 DBG kernel booting: 3000+ "kernel: Process (pid 1) got signal 5" notices during booting Message-Id: <6F60DA66-1333-4CF0-B433-F0CE56F5D301@yahoo.com> Date: Tue, 21 Nov 2023 21:43:50 -0800 To: Current FreeBSD X-Mailer: Apple Mail (2.3774.200.91.1.1) References: <6F60DA66-1333-4CF0-B433-F0CE56F5D301.ref@yahoo.com> X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.32:from]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.32:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4SZqtZ6W3wz4NWb X-Spamd-Bar: --- While my kernel/world build procedures build both DBG and NODBG kernels and worlds, I normally run the NODBG kernel and world, using DBG only when I need to for problem investigation. I recently had reason to use the DBG kernel and found it got the oddity of 3000+ instances of "kernel: Process (pid 1) got signal 5" during booting, as reported in /var/log/messages . An example is: . . . Nov 20 23:13:09 7950X3D-UFS shutdown[20174]: reboot by root:=20 Nov 20 23:13:09 7950X3D-UFS syslogd: exiting on signal 15 Nov 20 23:14:21 7950X3D-UFS syslogd: kernel boot file is = /boot/kernel/kernel Nov 20 23:14:21 7950X3D-UFS kernel: got signal 5 Nov 20 23:14:21 7950X3D-UFS kernel: Process (pid 1) got signal 5 Nov 20 23:14:21 7950X3D-UFS syslogd: last message repeated 3133 times Nov 20 23:14:21 7950X3D-UFS kernel: intsmb0: = at device 20.0 on pci0 . . . This stopped when I commented out the: options MAXMEMDOM=3D2 that I've had historically and built, installed, and booted the resulting DBG kernel. I'll note that I never had the messages for the NODDBG kernel, despite it also having that line. For reference: # uname -apKU FreeBSD 7950X3D-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT #2 = main-n266130-d521abdff236-dirty: Tue Nov 21 21:03:11 PST 2023 = root@7950X3D-UFS:/usr/obj/BUILDs/main-amd64-dbg-clang/usr/main-src/amd64.a= md64/sys/GENERIC-DBG amd64 amd64 1500002 1500002 # ~/fbsd-based-on-what-commit.sh -C /usr/main-src/ d521abdff236 (HEAD -> main, freebsd/main, freebsd/HEAD) Update ASLR = stack sysctl description in security.7 and mitigations.7 Author: Ed Maste Commit: Ed Maste CommitDate: 2023-10-24 22:29:25 +0000 branch: main merge-base: d521abdff2367a5c72a773a815fc3d99403274f5 merge-base: CommitDate: 2023-10-24 22:29:25 +0000 n266130 (--first-parent --count for merge-base) =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Wed Nov 22 20:23:56 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SbCQ02rGYz51pLc for ; Wed, 22 Nov 2023 20:24:08 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SbCPz53Vtz4S8Y; Wed, 22 Nov 2023 20:24:07 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=DhKoxZrK; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::536 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-x536.google.com with SMTP id 41be03b00d2f7-5c186f027cfso108404a12.3; Wed, 22 Nov 2023 12:24:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700684646; x=1701289446; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=BCnPP2Lt8KsVpERFQPUfwutkmn00JaUwN1kaZ7EjlSs=; b=DhKoxZrKyRzxFy3Shfjr/b9UTyybBJUG5Niki5U42k8do2r1OzRN720HLblOM1DpGH 40SY+Mkq1mUTq9EyU25ADoAg7/YwVMed+NAm183v/CsatwzjpeokiCDQvQvUcZ0SJIQ5 iE1BAm2d56yrwNZnGmGq1BnJ4l6qounapj9SUU+OTFB3CbjFjKgGfbH8qh6zy2UnJenS 1Vfz8fpWNbNTad9EMFehk8ztd6DrWJSbzea9tmJm3vNa9ZyspBuWYooG81/JEyXDA6cY r3X4sxxEKLKDxJoiNDAk1sgYqMYct3GoR7FA1GUZ5synWjXN0T3WflEDbRjBoc6EPtIP 49sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700684646; x=1701289446; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=BCnPP2Lt8KsVpERFQPUfwutkmn00JaUwN1kaZ7EjlSs=; b=xRkepDNp71u1QcPN5MkTcS/UR2j3QsfWkegbMwRlUM1OYvXDZrVL7ev7qt6C+mCbGx oZA5YAzAW4S/xkHbCM5j5Z02bQb+NBnspDgfg6F9DUEWtU7hrYTrN15+yBI3HfMhCNl9 jJkxpqrM6naA8lPMX2epzXwdBCPk68JhYN51Ug4775Ls3ztb/1UIUBSbIVcQWBqWGaIa 3pGFBVhgdrmSIS6QsHuf5CtRUpkdQjqBOCvTGbCcUOkxac3weORUyB7bBclRg65H3psW JqVIY8HBMaSmQzdXCk26QvupQ/LworN0h0D64TbLuoZbmRbl6orcwGujJUVMHoidTJpU yyDw== X-Gm-Message-State: AOJu0YyWQWH7zObKcO1NtaUy/7mXDNG3phYloNiy+A99RS4uPG+jm4hD YyyQsS3TqtHZgnVC6cqFrKvLmIdtDZiUI3e/HDi9SDU= X-Google-Smtp-Source: AGHT+IFs3YF8vDuB/fyVi0iYPV4EuZ13E0VIk+yBk8k5sB2HgrLvvywr+qnBmC6OVEfQFLQe8MeQhqAFuug8ui74Zkc= X-Received: by 2002:a05:6a20:a182:b0:187:f2f7:2383 with SMTP id r2-20020a056a20a18200b00187f2f72383mr2465485pzk.45.1700684646263; Wed, 22 Nov 2023 12:24:06 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 From: Rick Macklem Date: Wed, 22 Nov 2023 12:23:56 -0800 Message-ID: Subject: RFC: #f FreeBSD_version of #ifdef for OpenZFS pull request To: Martin Matuska , Alexander Motin , FreeBSD CURRENT , Garrett Wollman , Mike Karels Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.90 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.90)[-0.902]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCVD_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::536:from]; DKIM_TRACE(0.00)[gmail.com:+]; TAGGED_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; TO_DN_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4SbCPz53Vtz4S8Y X-Spamd-Bar: --- Hi, I have a patch currently under review at D42672 that fixes visibility of snapshots under .zfs/snapshot for NFS clients. It adds a new function called vfs_exjail_clone(), which the ZFS code needs to use to fill in the mnt_exjail field. Since the OpenZFS code is supposed to build for 12.2 or later, I can see two ways of doing this: (A) #if on the FreeBSD_versions, which will look something like: #if (__FreeBSD_version >= 1300xxx && __FreeBSD_version < 1400000) || (__FreeBSD_version >= 1400yyy && __FreeBSD_version < 1400500) || (__FreeBSD_version >= 1400zzz && __FreeBSD_version < 1500000) || __FreeBSD_version >= 1500wwww vfs_exjail_clone(); #endif The problem with this one is I do not know what www, xxx, yyy and zzz are until I have MFC'd the patch and bumped __FreeBSD_version. --> I cannot generate the OpenZFS pull request until after that and, since I am headed to Florida for a few weeks, it would be late December at the earliest. OR (B) add a line like #define VFS_SUPPORTS_EXJAIL_CLONE 1 to mount.h in the patch and then: #ifdef VFS_SUPPORTS_EXJAIL_CLONE vfs_exjail_clone(); #endif The adavntage of (B) is that I can do the pull request on OpenZFS right away and commit the patch to main, etc as soon as possible, So, which do you think is preferred? rick ps: Unless D42672 gets reviewed soon, it won't really matter w.r.t. timing. From nobody Wed Nov 22 21:30:09 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SbDtT736Hz522vB for ; Wed, 22 Nov 2023 21:30:25 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SbDtS4MJWz4d4Z for ; Wed, 22 Nov 2023 21:30:24 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20230601.gappssmtp.com header.s=20230601 header.b=YvpNOiOr; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2a00:1450:4864:20::535) smtp.mailfrom=wlosh@bsdimp.com; dmarc=none Received: by mail-ed1-x535.google.com with SMTP id 4fb4d7f45d1cf-54a95657df3so369704a12.0 for ; Wed, 22 Nov 2023 13:30:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1700688621; x=1701293421; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=xuPyxY3okCjv/Yyh9bKcbV2sxeLoQk7DHp+GdNFF7uM=; b=YvpNOiOr88lX6t7j5PIplmyu4e6GNTQ9fSrzdGdI1BgJH1Bh+usQ50+T3bdazth6WJ iYLODRjYUz8n37H2t+U9dNtk6j7t6mgt99dc0p9480ATnrEOAVsju+/K1gBR4x4unmQz ljQHANBqXIlPPj2/qnksuwzyKwW401QUkeaBBmvXh3xNZNqG6iK6GPEIWj8uGCnnQyOg TYuGO1v0mXD43nUVTqqNlZKGIQAgxflFihx9Huh4B7FQ4anyHyhF8QA3o7+pYwD1BAJh eTQ1inErjJy60dwWi8CUdVg6pv1q09rRt8NV6F0bbypnRbqjo7fb57tXads7JoJV5+4O YBug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700688621; x=1701293421; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=xuPyxY3okCjv/Yyh9bKcbV2sxeLoQk7DHp+GdNFF7uM=; b=o6hPKo6qsVHFRPne3YGnrC5y0LFtWvQQ4JD8ZemTn71dvvXI5BW6j7InwM4geaSN8v qeJ6K2WyZNKp+RU0mxCa/z4i7o4uTfsfF4dqSsZwwTSWG8D2h8ySINxq0G36BKQuZvDK kv3zvVHELLZ2WYAEH+WSvXgk282l8am1Aw98htMw2oK+UwNDpVbIoQfpbAA4HBWMBXvo 8yC3cLSsPEfxTmRiAWKdlCd8tPGTdOZWSF95Isozf9N5vQBnFencx0UnSHwP3Ef7AZIz djWDDE0m6OH3WhHQWlX9Y7LSSd1HqvqPMRYscbvwdbeFvCIcTh7hnOuLOjeXSYeFKfnI RqgQ== X-Gm-Message-State: AOJu0YyS6G4qqwejRjwavDmGXjeGdxeKztkwRV3TpKpfRIGPAfm/k4IY wmgbP61/fNVMUw3lAWmnWsEP517w3C6fLEidvh6aZw== X-Google-Smtp-Source: AGHT+IEzmX2MSwKPPV2LKNvJxp3HeV3wLsHPl/y8FKXUkkx6RrYBt3xyRdAAONl83Ab/a9aapmslz3q2p3y84GqbItU= X-Received: by 2002:aa7:c1cf:0:b0:53d:b751:ece1 with SMTP id d15-20020aa7c1cf000000b0053db751ece1mr2523960edp.41.1700688621237; Wed, 22 Nov 2023 13:30:21 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Wed, 22 Nov 2023 14:30:09 -0700 Message-ID: Subject: Re: RFC: #f FreeBSD_version of #ifdef for OpenZFS pull request To: Rick Macklem Cc: Martin Matuska , Alexander Motin , FreeBSD CURRENT , Garrett Wollman , Mike Karels Content-Type: multipart/alternative; boundary="000000000000248665060ac46d3e" X-Spamd-Result: default: False [-1.12 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.62)[-0.615]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20230601.gappssmtp.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::535:from]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_ALL(0.00)[]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; TAGGED_RCPT(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20230601.gappssmtp.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCPT_COUNT_FIVE(0.00)[6]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_NA(0.00)[no SPF record] X-Rspamd-Queue-Id: 4SbDtS4MJWz4d4Z X-Spamd-Bar: - --000000000000248665060ac46d3e Content-Type: text/plain; charset="UTF-8" On Wed, Nov 22, 2023, 1:24 PM Rick Macklem wrote: > Hi, > > I have a patch currently under review at D42672 that fixes visibility > of snapshots under .zfs/snapshot for NFS clients. > It adds a new function called vfs_exjail_clone(), which the ZFS > code needs to use to fill in the mnt_exjail field. > > Since the OpenZFS code is supposed to build for 12.2 or later, > I can see two ways of doing this: > (A) #if on the FreeBSD_versions, which will look something like: > > #if (__FreeBSD_version >= 1300xxx && __FreeBSD_version < 1400000) || > (__FreeBSD_version >= 1400yyy && __FreeBSD_version < 1400500) || > (__FreeBSD_version >= 1400zzz && __FreeBSD_version < 1500000) || > __FreeBSD_version >= 1500wwww > vfs_exjail_clone(); > #endif > > The problem with this one is I do not know what www, xxx, yyy and zzz are > until I have MFC'd the patch and bumped __FreeBSD_version. > --> I cannot generate the OpenZFS pull request until after that and, > since I am headed to Florida for a few weeks, it would be late > December > at the earliest. > OR > (B) add a line like > #define VFS_SUPPORTS_EXJAIL_CLONE 1 > to mount.h in the patch and then: > > #ifdef VFS_SUPPORTS_EXJAIL_CLONE > vfs_exjail_clone(); > #endif > > The adavntage of (B) is that I can do the pull request on OpenZFS > right away and commit the patch to main, etc as soon as possible, > > So, which do you think is preferred? rick > ps: Unless D42672 gets reviewed soon, it won't really matter w.r.t. timing. > I'd do B if I were doing this. With a comment for why I'm doing this define. Then version numbers don't matter unless we botch something badly and need them as a fallback. Warner --000000000000248665060ac46d3e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Wed, Nov 22, 2023, 1:24 PM Rick Macklem <rick.macklem@gmail.com> wrote:
Hi,

I have a patch currently under review at D42672 that fixes visibility
of snapshots under .zfs/snapshot for NFS clients.
It adds a new function called vfs_exjail_clone(), which the ZFS
code needs to use to fill in the mnt_exjail field.

Since the OpenZFS code is supposed to build for 12.2 or later,
I can see two ways of doing this:
(A) #if on the FreeBSD_versions, which will look something like:

#if (__FreeBSD_version >=3D 1300xxx && __FreeBSD_version < 14= 00000) ||
=C2=A0 =C2=A0 =C2=A0(__FreeBSD_version >=3D 1400yyy && __FreeBSD= _version < 1400500) ||
=C2=A0 =C2=A0 =C2=A0(__FreeBSD_version >=3D 1400zzz && __FreeBSD= _version < 1500000) ||
=C2=A0 =C2=A0 =C2=A0__FreeBSD_version >=3D 1500wwww
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0vfs_exjail_clone();
#endif

The problem with this one is I do not know what www, xxx, yyy and zzz are until I have MFC'd the patch and bumped __FreeBSD_version.
--> I cannot generate the OpenZFS pull request until after that and,
=C2=A0 =C2=A0 =C2=A0since I am headed to Florida for a few weeks, it would = be late December
=C2=A0 =C2=A0 =C2=A0at the earliest.
OR
(B) add a line like
#define VFS_SUPPORTS_EXJAIL_CLONE=C2=A0 =C2=A0 1
to mount.h in the patch and then:

#ifdef VFS_SUPPORTS_EXJAIL_CLONE
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0vfs_exjail_clone();
#endif

The adavntage of (B) is that I can do the pull request on OpenZFS
right away and commit the patch to main, etc as soon as possible,

So, which do you think is preferred? rick
ps: Unless D42672 gets reviewed soon, it won't really matter w.r.t. tim= ing.

I'd do B if I were doing this. With a comment for why I'm doing= this define. Then version numbers don't matter unless we botch somethi= ng badly and need them as a fallback.

Warner

--000000000000248665060ac46d3e-- From nobody Thu Nov 23 03:27:47 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SbNq75q1sz52KJZ for ; Thu, 23 Nov 2023 03:28:03 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SbNq749Vyz4K1r; Thu, 23 Nov 2023 03:28:03 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x102e.google.com with SMTP id 98e67ed59e1d1-2856437b584so147246a91.3; Wed, 22 Nov 2023 19:28:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700710079; x=1701314879; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=VgvPyroddqXhXZrDPQ4Srfg7cFhswXMAOFXr5phXtpY=; b=QkPxzaUcQlZHxX8pCW4aPk1yXHOSCLTW/NbXKstMk4XfpXGz8z1w0ugouVFCnR6eVK SPKEjbta6GNu9vXKHYGEdA3YF40+CUiUhvrhFl6xExmXJEmowzLaAsExHbf/MYDOloFj 1g3dKLgp3oITDDM4XTeUJ5uEml1faBuWLFUX37zx2RGQil8ArQP6cFQeADVewTrJVNeI X5JjW8HXYwtKJ8HuYkcxy3wX73aWCxHx0y/S8H2rSGRbJZfL9nHtM1GqJ8i4HyA6w0sc aLcikW8Kj4SeYwlH+buw4xVCLsHt21AEWH8m7y1iyrwKtL+LRmjSSydd9AVvtNPNVwkA QBSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700710079; x=1701314879; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VgvPyroddqXhXZrDPQ4Srfg7cFhswXMAOFXr5phXtpY=; b=sfieF7Pr3xPx960+QY6f8Q5kjTg3UGs9JMe4s2xQk5fq4jfzf9q1wng+2CMHzQFtUt vG0uDVsce7MkcY34k6IVPTwUigkR8bE2BZlaXON1XMI/T8Cyk8HT4DdAZRqkZ0zskPRD Mm6LHDPwDaAxuIgAKb/VlyceN8jlUObg0XJpSEGc23qzwI/M89E7i432CvCs2ZtA7Qog 7jiKKlzSru17V7RpvhyLiHp3jtVEObP4XoVmYLEfGk9mBzSsXR3i4qxyQStQjzugU9Wy Kk4OO/mw+02XAefi/FbfwUcF2hyl8nZPyXJcAXzHmO7jO8aMFHedlcqQm6/87zIwn8wn JQyw== X-Gm-Message-State: AOJu0YxbxuWm4Qq2n6ZfI/vhz0Qz4TErh9qrGlrLBXYYjPobB3xr4ZZ0 BH+QWoUXE9NATjpSI3DA46iwrN8hVGwmXrUqXA6/D+0= X-Google-Smtp-Source: AGHT+IHlhG+03MqYQ1wYEiTy8p7bpEbaAQrzg6EfkwzTJtJo3CSa28UQpMpAyEBaj+r+XLwp/ilKrDHcqBFI48YmhqU= X-Received: by 2002:a17:90b:3595:b0:27d:1c70:23d4 with SMTP id mm21-20020a17090b359500b0027d1c7023d4mr4408170pjb.44.1700710079200; Wed, 22 Nov 2023 19:27:59 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Wed, 22 Nov 2023 19:27:47 -0800 Message-ID: Subject: Re: RFC: #f FreeBSD_version of #ifdef for OpenZFS pull request To: Warner Losh Cc: Martin Matuska , Alexander Motin , FreeBSD CURRENT , Garrett Wollman , Mike Karels Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Queue-Id: 4SbNq749Vyz4K1r Thanks, I've gone with B, rick On Wed, Nov 22, 2023 at 1:30=E2=80=AFPM Warner Losh wrote: > > > > On Wed, Nov 22, 2023, 1:24 PM Rick Macklem wrote= : >> >> Hi, >> >> I have a patch currently under review at D42672 that fixes visibility >> of snapshots under .zfs/snapshot for NFS clients. >> It adds a new function called vfs_exjail_clone(), which the ZFS >> code needs to use to fill in the mnt_exjail field. >> >> Since the OpenZFS code is supposed to build for 12.2 or later, >> I can see two ways of doing this: >> (A) #if on the FreeBSD_versions, which will look something like: >> >> #if (__FreeBSD_version >=3D 1300xxx && __FreeBSD_version < 1400000) || >> (__FreeBSD_version >=3D 1400yyy && __FreeBSD_version < 1400500) || >> (__FreeBSD_version >=3D 1400zzz && __FreeBSD_version < 1500000) || >> __FreeBSD_version >=3D 1500wwww >> vfs_exjail_clone(); >> #endif >> >> The problem with this one is I do not know what www, xxx, yyy and zzz ar= e >> until I have MFC'd the patch and bumped __FreeBSD_version. >> --> I cannot generate the OpenZFS pull request until after that and, >> since I am headed to Florida for a few weeks, it would be late Dece= mber >> at the earliest. >> OR >> (B) add a line like >> #define VFS_SUPPORTS_EXJAIL_CLONE 1 >> to mount.h in the patch and then: >> >> #ifdef VFS_SUPPORTS_EXJAIL_CLONE >> vfs_exjail_clone(); >> #endif >> >> The adavntage of (B) is that I can do the pull request on OpenZFS >> right away and commit the patch to main, etc as soon as possible, >> >> So, which do you think is preferred? rick >> ps: Unless D42672 gets reviewed soon, it won't really matter w.r.t. timi= ng. > > > I'd do B if I were doing this. With a comment for why I'm doing this defi= ne. Then version numbers don't matter unless we botch something badly and n= eed them as a fallback. > > Warner > From nobody Thu Nov 23 16:13:26 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SbjpX6ktnz51plF for ; Thu, 23 Nov 2023 16:13:40 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SbjpX0q0wz4VkV; Thu, 23 Nov 2023 16:13:40 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=eJvmC81g; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::102d as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x102d.google.com with SMTP id 98e67ed59e1d1-2809b4d648bso866476a91.2; Thu, 23 Nov 2023 08:13:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700756018; x=1701360818; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=vh2UKPMNx1L/5N+6LNgkvUvX17L29P0s2Gceuwy/h9o=; b=eJvmC81gx3WMLrLUYGtWPRjVycsOBerHCcibQJHmPVF1Ai6YD+TZ82ltN/J650OD8U Fh6p/XHQ+qfnCQnoWzKnXgfhIN3GDvyF1FYPeR46ED2kdVLf2XmwTNN3yFQt4o+X4Li+ /wW2sRcZPyoMf5zp7mR0V2ps6ycQ55+fmMFMvbjpbR5l8I6nXPsGQTXlXYErrVYd5kDn A4mTxyw4J5yMZZP+wSpPoyd88NG7XXA76CjQGZjeklmKVYMUAzNeZEm/ZzLd9RdRBvWV he0XwNlfVVmOuNF+RNgao8+qMxMTIZ2LV+HrATwCMMDiTPOKjCrlkjd+zle/4RahsDZT y/oQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700756018; x=1701360818; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vh2UKPMNx1L/5N+6LNgkvUvX17L29P0s2Gceuwy/h9o=; b=UKWyRPVRlYc1xMXRF6qrSAydqFC9M2guOHZ3pB6egBAGYnSofe00sLBR60RvaLpwOR brREh6SJhDDEIEhF53b6b2AdMU6FWgYdI1BlOI7WdUZ7BtbMYykw4W2QQw05e3fdUCjJ uPojPjyF3HDXzB6RojWP+d6EUvRT8WVkwd7qI6HUfHNEuEqrTR7DNnj0wtqlgot6p89u SOdS8Z0ThFjgserlUPjHe/ZUExpoR1B5h1mclBSXOF/FkH/7f+wv+k+A+sPzfob8lMbY xPc19MqQQBqcwpLzZw2JeRzp8BggdtKHwQAjSVg1KVulJK0spBp5LQewaJ459vjKkHlt kdNA== X-Gm-Message-State: AOJu0YyU5qMR1Ub6Agw+cIm1UJwpsT9ffunvH0M813bp2k2O68X+6F3n 3KIU0SfNMsJEF4uZtBAcXTIukrys8VcSULO/cg== X-Google-Smtp-Source: AGHT+IFyqadu9mdrWTeXdi81RAvUirFiB/pF8LSJFiPp5tctA1P5dgGkLOb4BkElbtAn1zzW9CDQTzKLRcnUW5jyZfk= X-Received: by 2002:a17:90b:1649:b0:281:20a0:dae3 with SMTP id il9-20020a17090b164900b0028120a0dae3mr5472828pjb.40.1700756017952; Thu, 23 Nov 2023 08:13:37 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <25943.60056.880614.452966@hergotha.csail.mit.edu> <91988E23-ED50-4379-AA5F-4B069E08D80F@karels.net> <7C7920E0-B0FC-4255-AD5C-ECBDDC18CF43@karels.net> In-Reply-To: <7C7920E0-B0FC-4255-AD5C-ECBDDC18CF43@karels.net> From: Rick Macklem Date: Thu, 23 Nov 2023 08:13:26 -0800 Message-ID: Subject: Re: NFS exports of ZFS snapshots broken To: Mike Karels Cc: FreeBSD CURRENT , Alexander Motin , Martin Matuska , Garrett Wollman Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102d:from]; DKIM_TRACE(0.00)[gmail.com:+]; TAGGED_FROM(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; TO_DN_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4SbjpX0q0wz4VkV X-Spamd-Bar: --- On Sat, Nov 18, 2023 at 8:10=E2=80=AFPM Mike Karels wrote= : > > On 18 Nov 2023, at 21:19, Rick Macklem wrote: > > > On Sat, Nov 18, 2023 at 4:43=E2=80=AFPM Mike Karels w= rote: > >> > >> On 18 Nov 2023, at 17:23, Rick Macklem wrote: > >> > >>> On Sat, Nov 18, 2023 at 2:27=E2=80=AFPM Mike Karels = wrote: > >>>> > >>>> On 18 Nov 2023, at 15:58, Rick Macklem wrote: > >>>> > >>>>> On Sat, Nov 18, 2023 at 8:09=E2=80=AFAM Rick Macklem wrote: > >>>>>> > >>>>>> On Fri, Nov 17, 2023 at 8:19=E2=80=AFPM Mike Karels wrote: > >>>>>>> > >>>>>>> On 17 Nov 2023, at 22:14, Mike Karels wrote: > >>>>>>> > >>>>>>>> On 17 Nov 2023, at 21:24, Rick Macklem wrote: > >>>>>>>> > >>>>>>>>> Most of the changes in stable/13 that are not in releng/13.2 > >>>>>>>>> are the "make it work in a jail" stuff. Unfortunately, they are > >>>>>>>>> a large # of changes (mostly trivial edits adding vnet macros), > >>>>>>>>> but it also includes export check changes. > >>>>>>>>> > >>>>>>>>> I have attached a trivial patch that I think disables the expor= t > >>>>>>>>> checks for jails. If either of you can try it and see if it fix= es > >>>>>>>>> the problem, that would be great. > >>>>>>>>> (Note that this is only for testing, although it probably does = not > >>>>>>>>> matter unless you are running nfsd(8) in vnet jails.) > >>>>>>>> > >>>>>>>> Yes, I can see snapshots with the patch. This system is just a = test > >>>>>>>> system that doesn't normally run ZFS or NFS, so no problem messi= ng > >>>>>>>> with permissions. It's a bhyve VM, so I just added a small disk= and > >>>>>>>> enabled ZFS for testing. > >>>>>>> > >>>>>>> btw, you might try to get mm@ or maybe mav@ to help out from the = ZFS > >>>>>>> side. It must be doing something differently inside a snapshot t= han > >>>>>>> outside, maybe with file handles or something like that. > >>>>>> Yes. I've added freebsd-current@ (although Garrett is not on it, h= e is > >>>>>> cc'd) and these guys specifically... > >>>>>> > >>>>>> So, here's what appears to be the problem... > >>>>>> Commit 88175af (in main and stable/13, but not 13.2) added checks = for > >>>>>> nfsd(8) running in jails by filling in mnt_exjail with a reference= to the cred > >>>>>> used when the file system is exported. > >>>>>> When mnt_exjail is found NULL, the current nfsd code assumes that = there > >>>>>> is no access allowed for the mount. > >>>>>> > >>>>>> My vague understanding is that when a ZFS snapshot is accessed, it= is > >>>>>> "pseudo-mounted" by zfsctl_snapdir_lookup() and I am guessing that > >>>>>> mnt_exjail is NULL as a result. > >>>>>> Since I do not know the ZFS code and don't even have an easy way t= o > >>>>>> test this (thankfully Mike can test easily), I do not know what to= do from > >>>>>> here? > >>>>>> > >>>>>> Is there a "struct mount" constructed for this pseudo mount > >>>>>> (or it actually appears to be the lookup of ".." that fails, so it > >>>>>> might be the parent of the snapshot subdir?)? > >>>>>> > >>>>>> One thought is that I can check to see if the mount pointer is in = the > >>>>>> mountlist (I don't think the snapshot's mount is in the mountlist)= and > >>>>>> avoid the jail test for this case. This would assume that snapsho= ts are > >>>>>> always within the file system(s) exported via that jail (which inc= ludes > >>>>>> the case of prison0, of course), so that they do not need a separa= te > >>>>>> jail check. > >>>>>> > >>>>>> If this doesn't work, there will need to be some sort of messing a= bout > >>>>>> in ZFS to set mnt_exjail for these. > >>>>> Ok, so now onto the hard part... > >>>>> Thanks to Mike and others, I did create a snapshot under .zfs and I= can > >>>>> see the problem. It is that mnt_exjail =3D=3D NULL. > >>>>> Now, is there a way that this "struct mount" can be recognized as "= special" > >>>>> for snapshots, so I can avoid the mnt_exjail =3D=3D NULL test? > >>>>> (I had hoped that "mp->mnt_list.tqe_prev" would be NULL, but that i= s not > >>>>> the case.) > >>>> > >>>> Dumb question, is the mount point (mp presumably) different between = the > >>>> snapshot and the main file system? > >>> Not a dump question and the answer is rather interesting... > >>> It is "sometimes" or "usually" according to my printf(). > >>> It seems that when you first "cd >>> where mnt_exjail =3D=3D NULL.. Then when you look at directories with= in the > >>> snapshot, you get the mp of the file system that .zfs exists in, whic= h does > >>> have mnt_exjail set non-NULL. > >>> > >>> There is this snippet of code in zfsctl_snapdir_lookup(): > >>> /* > >>> * Fix up the root vnode mounted on .zfs/snapshot/. > >>> * > >>> * This is where we lie about our v_vfsp in order to > >>> * make .zfs/snapshot/ accessible over NFS > >>> * without requiring manual mounts of . > >>> */ > >>> ASSERT3P(VTOZ(*vpp)->z_zfsvfs, !=3D, zfsvfs); > >>> VTOZ(*vpp)->z_zfsvfs->z_parent =3D zfsvfs; > >>> > >>> /* Clear the root flag (set via VFS_ROOT) as well. */ > >>> (*vpp)->v_vflag &=3D ~VV_ROOT; > >>> which seems to set the mp to that of the parent, but it > >>> seems this does not happen for the initial lookup of > >>> the ? > >>> > >>> I'll note that there is code before this in > >>> zfsctl_snapdir_lookup() for handling cases > >>> like "." and ".." that return without doing this. > >>> > >>> Now, why does this work without the mnt_exjail > >>> check (as in 13.2)? > >>> I am not quite sure, but there is this "cheat" in the > >>> NFS server (it has been there for years, maybe decades): > >>> /* > >>> * Allow a Lookup, Getattr, GetFH, Secinfo on an > >>> * non-exported directory if > >>> * nfs_rootfhset. Do I need to allow any other Ops? > >>> * (You can only have a non-exported vpnes if > >>> * nfs_rootfhset is true. See nfsd_fhtovp()) > >>> * Allow AUTH_SYS to be used for file systems > >>> * exported GSS only for certain Ops, to allow > >>> * clients to do mounts more easily. > >>> */ > >>> if (nfsv4_opflag[op].needscfh && vp) { > >>> if (!NFSVNO_EXPORTED(&vpnes) && > >>> op !=3D NFSV4OP_LOOKUP && > >>> op !=3D NFSV4OP_GETATTR && > >>> op !=3D NFSV4OP_GETFH && > >>> op !=3D NFSV4OP_ACCESS && > >>> op !=3D NFSV4OP_READLINK && > >>> op !=3D NFSV4OP_SECINFO && > >>> op !=3D NFSV4OP_SECINFONONAME) > >>> nd->nd_repstat =3D NFSERR_NOFILEHANDLE; > >>> This allows certain operations to be done on > >>> non-exported file systems and I think that is enough > >>> to allow this to work when mnt_exjail is not checked. > >>> (Note that NFSV4OP_LOOKUPP is not in the list, > >>> which might explain why it is the one that fails for > >>> Garrett. I don't think it can be added to this list > >>> safely, since that would allow a client to move above > >>> the exported file system into "uncharted territory".) > >>> > >>>> Just curious. Also, what is mnt_exjail > >>>> normally set to for file systems not in a jail? > >>> mnt_exjail is set to the credentials of the thread/process > >>> that exported the file system (usually mountd(8)). > >>> When not in a jail, cr_prison for these credentials > >>> points to prison0. > >>> > >>> Btw, I checked and the "other mp that has mnt_exjail =3D=3D NULL > >>> is in the mountlist, so the idea of checking "not in mountlist" > >>> is a dead end. > >>> > >>> I am looking for something "unique" about this other mp, > >>> but haven't found anything yet. > >>> Alternately, it might be necessary to add code to > >>> zfsctl_snapdir_lookup() to "cheat and change the mp" > >>> in more cases, such as "." and ".." lookups? > >> > >> It seems to me that if ZFS is creating an additional mount structure, > >> it should be responsible for setting it up correctly. That could > >> involve a vfs-level routine to do some of the cloning. In any case, > >> it seems to me that mnt_exjail should be set properly, e.g. by duping > >> the one in the original mount structure. Probably ZFS is the only > >> file system type that would need this added. > > I've created a patch that I think does this. It seemed to test ok for m= y case. > > It's in D42672 on reviews.freebsd.org. > > I have also attached it here (this diff doesn't use -U999999, so it mig= ht be > > easier to apply). > > It works for me in 13-stable, and looks plausible. The patch has now been fixed so that there is no race between vfs_exjail_cl= one() and a jail dying. I believe the current patch fixes the problem. The patch can be found as an attachment to PR#275200 (that is a FreeBSD bug= zilla problem report, not an OpenZFS pull request). I will not close the PR until the above has all happened. This patch will be needed for systems running stable/13, stable/14 and 14.0= . The vfs_exjail_clone() part of the patch is now committed to main and will = be MFC'd in 3 days. The ZFS part of the patch is a pull request on OpenZFS. I cannot say how long the ZFS part takes or if/when an errata to 14.0 will happen. Sorry about the breakage and thanks for reporting it (and persevering when I said I knew nothing about ZFS, which is true). Thanks goes to Mike Karels= , who was able to determine that the problem was not in 13.2, but was in stable/13, which allowed me to conclude it was a "nfsd in jail" related breakage. rick > > Mike > > > Hopefully this fixes the problem. Sorry for the breakage. > > > > rick > > > >> > >> Mike > >> > >>> rick > >>> ps: I added all the cc's back in because I want the > >>> ZFS folk to hopefully chime in. > >>> > >>>> > >>>> Mike > >>>> > >>>>> Do I need to search mountlist for it? > >>>>> > >>>>> rick > >>>>> ps: The hack patch attached should fix the problem, but can only be > >>>>> safely used if mountd/nfsd are not run in any jails. > >>>>> > >>>>>> > >>>>>> I will try and get a test setup going here, which leads me to.. > >>>>>> how do I create a ZFS snapshot? (I do have a simple ZFS pool runni= ng > >>>>>> on a test machine, but I've never done a snapshot.) > >>>>>> > >>>>>> Although this problem is not in 13.2, it will have shipped in 14.0= . > >>>>>> > >>>>>> Any help with be appreciated, rick > >>>>>> > >>>>>>> > >>>>>>> Mike > >>>>>>>> > >>>>>>>>> rick > >>>>>>>>> > >>>>>>>>> On Fri, Nov 17, 2023 at 6:14=E2=80=AFPM Mike Karels wrote: > >>>>>>>>>> > >>>>>>>>>> CAUTION: This email originated from outside of the University = of Guelph. Do not click links or open attachments unless you recognize the = sender and know the content is safe. If in doubt, forward suspicious emails= to IThelp@uoguelph.ca. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Rick, have you been following this thread on freebsd-stable? = I have been able > >>>>>>>>>> to reproduce this using a 13-stable server from Oct 7 and a 15= -current system > >>>>>>>>>> that is up to date using NFSv3. I did not reproduce with a 13= .2 server. The > >>>>>>>>>> client was running 13.2. Any ideas? A full bisect seems fair= ly painful, but > >>>>>>>>>> maybe you have an idea of points to try. Fortunately, these a= re all test > >>>>>>>>>> systems that I can reboot at will. > >>>>>>>>>> > >>>>>>>>>> Mike > >>>>>>>>>> > >>>>>>>>>> Forwarded message: > >>>>>>>>>> > >>>>>>>>>>> From: Garrett Wollman > >>>>>>>>>>> To: Mike Karels > >>>>>>>>>>> Cc: freebsd-stable@freebsd.org > >>>>>>>>>>> Subject: Re: NFS exports of ZFS snapshots broken > >>>>>>>>>>> Date: Fri, 17 Nov 2023 17:35:04 -0500 > >>>>>>>>>>> > >>>>>>>>>>> < said: > >>>>>>>>>>> > >>>>>>>>>>>> I have not run into this, so I tried it just now. I had no = problem. > >>>>>>>>>>>> The server is 13.2, fully patched, the client is up-to-date = -current, > >>>>>>>>>>>> and the mount is v4. > >>>>>>>>>>> > >>>>>>>>>>> On my 13.2 client and 13-stable server, I see: > >>>>>>>>>>> > >>>>>>>>>>> 25034 ls CALL open(0x237d32f9a000,0x120004) > >>>>>>>>>>> 25034 ls NAMI "/mnt/tools/.zfs/snapshot/weekly-2023-4= 5" > >>>>>>>>>>> 25034 ls RET open 4 > >>>>>>>>>>> 25034 ls CALL fcntl(0x4,F_ISUNIONSTACK,0x0) > >>>>>>>>>>> 25034 ls RET fcntl 0 > >>>>>>>>>>> 25034 ls CALL getdirentries(0x4,0x237d32faa000,0x1000= ,0x237d32fa7028) > >>>>>>>>>>> 25034 ls RET getdirentries -1 errno 5 Input/output e= rror > >>>>>>>>>>> 25034 ls CALL close(0x4) > >>>>>>>>>>> 25034 ls RET close 0 > >>>>>>>>>>> 25034 ls CALL exit(0) > >>>>>>>>>>> > >>>>>>>>>>> Certainly a libc bug here that getdirentries(2) returning [EI= O] > >>>>>>>>>>> results in ls(1) returning EXIT_SUCCESS, but the [EIO] error = is > >>>>>>>>>>> consistent across both FreeBSD and Linux clients. > >>>>>>>>>>> > >>>>>>>>>>> Looking at this from the RPC side: > >>>>>>>>>>> > >>>>>>>>>>> (PUTFH, GETATTR, LOOKUP(snapshotname), GETFH, GETATTR) > >>>>>>>>>>> [NFS4_OK for all ops] > >>>>>>>>>>> (PUTFH, GETATTR) > >>>>>>>>>>> [NFS4_OK, NFS4_OK] > >>>>>>>>>>> (PUTFH, ACCESS(0x3f), GETATTR) > >>>>>>>>>>> [NFS4_OK, NFS4_OK, rights =3D 0x03, NFS4_OK] > >>>>>>>>>>> (PUTFH, GETATTR, LOOKUPP, GETFH, GETATTR) > >>>>>>>>>>> [NFS4_OK, NFS4_OK, NFS4ERR_NOFILEHANDLE] > >>>>>>>>>>> > >>>>>>>>>>> and at this point the [EIO] is returned. > >>>>>>>>>>> > >>>>>>>>>>> It seems that clients always do a LOOKUPP before calling READ= DIR, and > >>>>>>>>>>> this is failing when the subject file handle is the snapshot.= The > >>>>>>>>>>> client is perfectly able to *traverse into* the snapshot: if = I try to > >>>>>>>>>>> list a subdirectory I know exists in the snapshot, the client= is able to > >>>>>>>>>>> LOOKUP(dirname) just fine, but LOOKUPP still fails with > >>>>>>>>>>> NFS4ERR_NOFILEHANDLE *on the subndirectory*. > >>>>>>>>>>> > >>>>>>>>>>> -GAWollman > >>>>>>>>>> From nobody Thu Nov 23 16:26:00 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sbk525s9pz51rtk for ; Thu, 23 Nov 2023 16:26:14 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sbk5164C3z4Xyy; Thu, 23 Nov 2023 16:26:13 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=CHd+SzxA; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::1029 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x1029.google.com with SMTP id 98e67ed59e1d1-285196aaecaso913003a91.0; Thu, 23 Nov 2023 08:26:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700756772; x=1701361572; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=VJZF9+tmEGXTEGaoMG+9OuTsS8sX3q0QjFioNUQqV1g=; b=CHd+SzxAczBYrVJEIRXk3FO4dJbDelrvG2lcGXgvUP35CUF7RUfFRuBcfm1dIfk22x 1egoATBDv1L13qElN5kvUGtdfsBQ9JE0mGAkbWJvPBUGYGRugla0AI68eua3AKWO1EUx kqFkaHxW52n51gQpTNpO6GQ4WpCY3Oq85UPZKgl8tfD3U7+SdJDaIbzGQFidKLV+1yqo Q82f3pCdUezYnPXvZjmmn454mcmeF/M4NE+VbpiwLNJHu1CFIr0n2F5O2yjU4r1HLpCE gwqazz431LBbE4F4nZvEBbXF+t+tr9WpBjp8+Zne8Qwvxa30M8ImKhXOY1FXmlqMCwsU BwUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700756772; x=1701361572; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VJZF9+tmEGXTEGaoMG+9OuTsS8sX3q0QjFioNUQqV1g=; b=whkBh0bNeTspHxIHQHw35cty0Bh5sIVyZLYclrSbtxyDsqoq34ulHEs/v5NyFUt1OT LSUGQUnsMCoY/jFrh1yhuK0PWTZdQrkBlL4+yO8YfnMizDOvLAvchoOZ38M9CzaoMP1t IidDnNxScOJqf616WeRWo4LDbgtkwj+ueCnnNw1YW2KQsFv/BD6FANL7Gj5CFpRlDvoY NW37Ec5SFiuwQvVYmjBMYso2XZy/rT+fgNkhV9faH+0bmTUsjoJwVkiuBWoJSs3SmhXg M91bd2pwAfWyqsmJ3uHVYK/wOrOGSH9jBEGFpFZ8ERMX0R/+YpuR7WU7tLqutpbO4A5f xqag== X-Gm-Message-State: AOJu0YzF3ATIJPSZZvk+SGC8urPine94MyAHsdjiwi5CKq78aqtlDewp CdQ929hbwR/RGrYI1T9ke9sIJYXfx7G0A8wQFg== X-Google-Smtp-Source: AGHT+IGgVVzyZSxSHIs2MllnomA95uQYGCmOPHP/eIFi6LUEO0iqSUk3KLsLaC43wlgN0XfI44UsrZYsdax++dRbgLk= X-Received: by 2002:a17:90b:4a03:b0:283:21d3:11eb with SMTP id kk3-20020a17090b4a0300b0028321d311ebmr23642pjb.3.1700756772066; Thu, 23 Nov 2023 08:26:12 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <25943.60056.880614.452966@hergotha.csail.mit.edu> <91988E23-ED50-4379-AA5F-4B069E08D80F@karels.net> <7C7920E0-B0FC-4255-AD5C-ECBDDC18CF43@karels.net> In-Reply-To: From: Rick Macklem Date: Thu, 23 Nov 2023 08:26:00 -0800 Message-ID: Subject: Re: NFS exports of ZFS snapshots broken To: Mike Karels Cc: FreeBSD CURRENT , Alexander Motin , Martin Matuska , Garrett Wollman Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1029:from]; DKIM_TRACE(0.00)[gmail.com:+]; TAGGED_FROM(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; TO_DN_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4Sbk5164C3z4Xyy X-Spamd-Bar: --- Oh, and thanks to everyone who put up with my lame ZFS snapshot questions. rick On Thu, Nov 23, 2023 at 8:13=E2=80=AFAM Rick Macklem wrote: > > On Sat, Nov 18, 2023 at 8:10=E2=80=AFPM Mike Karels wro= te: > > > > On 18 Nov 2023, at 21:19, Rick Macklem wrote: > > > > > On Sat, Nov 18, 2023 at 4:43=E2=80=AFPM Mike Karels = wrote: > > >> > > >> On 18 Nov 2023, at 17:23, Rick Macklem wrote: > > >> > > >>> On Sat, Nov 18, 2023 at 2:27=E2=80=AFPM Mike Karels wrote: > > >>>> > > >>>> On 18 Nov 2023, at 15:58, Rick Macklem wrote: > > >>>> > > >>>>> On Sat, Nov 18, 2023 at 8:09=E2=80=AFAM Rick Macklem wrote: > > >>>>>> > > >>>>>> On Fri, Nov 17, 2023 at 8:19=E2=80=AFPM Mike Karels wrote: > > >>>>>>> > > >>>>>>> On 17 Nov 2023, at 22:14, Mike Karels wrote: > > >>>>>>> > > >>>>>>>> On 17 Nov 2023, at 21:24, Rick Macklem wrote: > > >>>>>>>> > > >>>>>>>>> Most of the changes in stable/13 that are not in releng/13.2 > > >>>>>>>>> are the "make it work in a jail" stuff. Unfortunately, they a= re > > >>>>>>>>> a large # of changes (mostly trivial edits adding vnet macros= ), > > >>>>>>>>> but it also includes export check changes. > > >>>>>>>>> > > >>>>>>>>> I have attached a trivial patch that I think disables the exp= ort > > >>>>>>>>> checks for jails. If either of you can try it and see if it f= ixes > > >>>>>>>>> the problem, that would be great. > > >>>>>>>>> (Note that this is only for testing, although it probably doe= s not > > >>>>>>>>> matter unless you are running nfsd(8) in vnet jails.) > > >>>>>>>> > > >>>>>>>> Yes, I can see snapshots with the patch. This system is just = a test > > >>>>>>>> system that doesn't normally run ZFS or NFS, so no problem mes= sing > > >>>>>>>> with permissions. It's a bhyve VM, so I just added a small di= sk and > > >>>>>>>> enabled ZFS for testing. > > >>>>>>> > > >>>>>>> btw, you might try to get mm@ or maybe mav@ to help out from th= e ZFS > > >>>>>>> side. It must be doing something differently inside a snapshot= than > > >>>>>>> outside, maybe with file handles or something like that. > > >>>>>> Yes. I've added freebsd-current@ (although Garrett is not on it,= he is > > >>>>>> cc'd) and these guys specifically... > > >>>>>> > > >>>>>> So, here's what appears to be the problem... > > >>>>>> Commit 88175af (in main and stable/13, but not 13.2) added check= s for > > >>>>>> nfsd(8) running in jails by filling in mnt_exjail with a referen= ce to the cred > > >>>>>> used when the file system is exported. > > >>>>>> When mnt_exjail is found NULL, the current nfsd code assumes tha= t there > > >>>>>> is no access allowed for the mount. > > >>>>>> > > >>>>>> My vague understanding is that when a ZFS snapshot is accessed, = it is > > >>>>>> "pseudo-mounted" by zfsctl_snapdir_lookup() and I am guessing th= at > > >>>>>> mnt_exjail is NULL as a result. > > >>>>>> Since I do not know the ZFS code and don't even have an easy way= to > > >>>>>> test this (thankfully Mike can test easily), I do not know what = to do from > > >>>>>> here? > > >>>>>> > > >>>>>> Is there a "struct mount" constructed for this pseudo mount > > >>>>>> (or it actually appears to be the lookup of ".." that fails, so = it > > >>>>>> might be the parent of the snapshot subdir?)? > > >>>>>> > > >>>>>> One thought is that I can check to see if the mount pointer is i= n the > > >>>>>> mountlist (I don't think the snapshot's mount is in the mountlis= t) and > > >>>>>> avoid the jail test for this case. This would assume that snaps= hots are > > >>>>>> always within the file system(s) exported via that jail (which i= ncludes > > >>>>>> the case of prison0, of course), so that they do not need a sepa= rate > > >>>>>> jail check. > > >>>>>> > > >>>>>> If this doesn't work, there will need to be some sort of messing= about > > >>>>>> in ZFS to set mnt_exjail for these. > > >>>>> Ok, so now onto the hard part... > > >>>>> Thanks to Mike and others, I did create a snapshot under .zfs and= I can > > >>>>> see the problem. It is that mnt_exjail =3D=3D NULL. > > >>>>> Now, is there a way that this "struct mount" can be recognized as= "special" > > >>>>> for snapshots, so I can avoid the mnt_exjail =3D=3D NULL test? > > >>>>> (I had hoped that "mp->mnt_list.tqe_prev" would be NULL, but that= is not > > >>>>> the case.) > > >>>> > > >>>> Dumb question, is the mount point (mp presumably) different betwee= n the > > >>>> snapshot and the main file system? > > >>> Not a dump question and the answer is rather interesting... > > >>> It is "sometimes" or "usually" according to my printf(). > > >>> It seems that when you first "cd > >>> where mnt_exjail =3D=3D NULL.. Then when you look at directories wi= thin the > > >>> snapshot, you get the mp of the file system that .zfs exists in, wh= ich does > > >>> have mnt_exjail set non-NULL. > > >>> > > >>> There is this snippet of code in zfsctl_snapdir_lookup(): > > >>> /* > > >>> * Fix up the root vnode mounted on .zfs/snapshot/. > > >>> * > > >>> * This is where we lie about our v_vfsp in order to > > >>> * make .zfs/snapshot/ accessible over NFS > > >>> * without requiring manual mounts of . > > >>> */ > > >>> ASSERT3P(VTOZ(*vpp)->z_zfsvfs, !=3D, zfsvfs); > > >>> VTOZ(*vpp)->z_zfsvfs->z_parent =3D zfsvfs; > > >>> > > >>> /* Clear the root flag (set via VFS_ROOT) as well. */ > > >>> (*vpp)->v_vflag &=3D ~VV_ROOT; > > >>> which seems to set the mp to that of the parent, but it > > >>> seems this does not happen for the initial lookup of > > >>> the ? > > >>> > > >>> I'll note that there is code before this in > > >>> zfsctl_snapdir_lookup() for handling cases > > >>> like "." and ".." that return without doing this. > > >>> > > >>> Now, why does this work without the mnt_exjail > > >>> check (as in 13.2)? > > >>> I am not quite sure, but there is this "cheat" in the > > >>> NFS server (it has been there for years, maybe decades): > > >>> /* > > >>> * Allow a Lookup, Getattr, GetFH, Secinfo on an > > >>> * non-exported directory if > > >>> * nfs_rootfhset. Do I need to allow any other Ops? > > >>> * (You can only have a non-exported vpnes if > > >>> * nfs_rootfhset is true. See nfsd_fhtovp()) > > >>> * Allow AUTH_SYS to be used for file systems > > >>> * exported GSS only for certain Ops, to allow > > >>> * clients to do mounts more easily. > > >>> */ > > >>> if (nfsv4_opflag[op].needscfh && vp) { > > >>> if (!NFSVNO_EXPORTED(&vpnes) && > > >>> op !=3D NFSV4OP_LOOKUP && > > >>> op !=3D NFSV4OP_GETATTR && > > >>> op !=3D NFSV4OP_GETFH && > > >>> op !=3D NFSV4OP_ACCESS && > > >>> op !=3D NFSV4OP_READLINK && > > >>> op !=3D NFSV4OP_SECINFO && > > >>> op !=3D NFSV4OP_SECINFONONAME) > > >>> nd->nd_repstat =3D NFSERR_NOFILEHANDLE; > > >>> This allows certain operations to be done on > > >>> non-exported file systems and I think that is enough > > >>> to allow this to work when mnt_exjail is not checked. > > >>> (Note that NFSV4OP_LOOKUPP is not in the list, > > >>> which might explain why it is the one that fails for > > >>> Garrett. I don't think it can be added to this list > > >>> safely, since that would allow a client to move above > > >>> the exported file system into "uncharted territory".) > > >>> > > >>>> Just curious. Also, what is mnt_exjail > > >>>> normally set to for file systems not in a jail? > > >>> mnt_exjail is set to the credentials of the thread/process > > >>> that exported the file system (usually mountd(8)). > > >>> When not in a jail, cr_prison for these credentials > > >>> points to prison0. > > >>> > > >>> Btw, I checked and the "other mp that has mnt_exjail =3D=3D NULL > > >>> is in the mountlist, so the idea of checking "not in mountlist" > > >>> is a dead end. > > >>> > > >>> I am looking for something "unique" about this other mp, > > >>> but haven't found anything yet. > > >>> Alternately, it might be necessary to add code to > > >>> zfsctl_snapdir_lookup() to "cheat and change the mp" > > >>> in more cases, such as "." and ".." lookups? > > >> > > >> It seems to me that if ZFS is creating an additional mount structure= , > > >> it should be responsible for setting it up correctly. That could > > >> involve a vfs-level routine to do some of the cloning. In any case, > > >> it seems to me that mnt_exjail should be set properly, e.g. by dupin= g > > >> the one in the original mount structure. Probably ZFS is the only > > >> file system type that would need this added. > > > I've created a patch that I think does this. It seemed to test ok for= my case. > > > It's in D42672 on reviews.freebsd.org. > > > I have also attached it here (this diff doesn't use -U999999, so it m= ight be > > > easier to apply). > > > > It works for me in 13-stable, and looks plausible. > > The patch has now been fixed so that there is no race between vfs_exjail_= clone() > and a jail dying. I believe the current patch fixes the problem. > The patch can be found as an attachment to PR#275200 (that is a FreeBSD b= ugzilla > problem report, not an OpenZFS pull request). I will not close the PR unt= il > the above has all happened. > > This patch will be needed for systems running stable/13, stable/14 and 14= .0. > > The vfs_exjail_clone() part of the patch is now committed to main and wil= l be > MFC'd in 3 days. The ZFS part of the patch is a pull request on OpenZFS. > I cannot say how long the ZFS part takes or if/when an errata to 14.0 wil= l > happen. > > Sorry about the breakage and thanks for reporting it (and persevering whe= n > I said I knew nothing about ZFS, which is true). Thanks goes to Mike Kare= ls, > who was able to determine that the problem was not in 13.2, but was in > stable/13, > which allowed me to conclude it was a "nfsd in jail" related breakage. > > rick > > > > > Mike > > > > > Hopefully this fixes the problem. Sorry for the breakage. > > > > > > rick > > > > > >> > > >> Mike > > >> > > >>> rick > > >>> ps: I added all the cc's back in because I want the > > >>> ZFS folk to hopefully chime in. > > >>> > > >>>> > > >>>> Mike > > >>>> > > >>>>> Do I need to search mountlist for it? > > >>>>> > > >>>>> rick > > >>>>> ps: The hack patch attached should fix the problem, but can only = be > > >>>>> safely used if mountd/nfsd are not run in any jails. > > >>>>> > > >>>>>> > > >>>>>> I will try and get a test setup going here, which leads me to.. > > >>>>>> how do I create a ZFS snapshot? (I do have a simple ZFS pool run= ning > > >>>>>> on a test machine, but I've never done a snapshot.) > > >>>>>> > > >>>>>> Although this problem is not in 13.2, it will have shipped in 14= .0. > > >>>>>> > > >>>>>> Any help with be appreciated, rick > > >>>>>> > > >>>>>>> > > >>>>>>> Mike > > >>>>>>>> > > >>>>>>>>> rick > > >>>>>>>>> > > >>>>>>>>> On Fri, Nov 17, 2023 at 6:14=E2=80=AFPM Mike Karels wrote: > > >>>>>>>>>> > > >>>>>>>>>> CAUTION: This email originated from outside of the Universit= y of Guelph. Do not click links or open attachments unless you recognize th= e sender and know the content is safe. If in doubt, forward suspicious emai= ls to IThelp@uoguelph.ca. > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Rick, have you been following this thread on freebsd-stable?= I have been able > > >>>>>>>>>> to reproduce this using a 13-stable server from Oct 7 and a = 15-current system > > >>>>>>>>>> that is up to date using NFSv3. I did not reproduce with a = 13.2 server. The > > >>>>>>>>>> client was running 13.2. Any ideas? A full bisect seems fa= irly painful, but > > >>>>>>>>>> maybe you have an idea of points to try. Fortunately, these= are all test > > >>>>>>>>>> systems that I can reboot at will. > > >>>>>>>>>> > > >>>>>>>>>> Mike > > >>>>>>>>>> > > >>>>>>>>>> Forwarded message: > > >>>>>>>>>> > > >>>>>>>>>>> From: Garrett Wollman > > >>>>>>>>>>> To: Mike Karels > > >>>>>>>>>>> Cc: freebsd-stable@freebsd.org > > >>>>>>>>>>> Subject: Re: NFS exports of ZFS snapshots broken > > >>>>>>>>>>> Date: Fri, 17 Nov 2023 17:35:04 -0500 > > >>>>>>>>>>> > > >>>>>>>>>>> < said: > > >>>>>>>>>>> > > >>>>>>>>>>>> I have not run into this, so I tried it just now. I had n= o problem. > > >>>>>>>>>>>> The server is 13.2, fully patched, the client is up-to-dat= e -current, > > >>>>>>>>>>>> and the mount is v4. > > >>>>>>>>>>> > > >>>>>>>>>>> On my 13.2 client and 13-stable server, I see: > > >>>>>>>>>>> > > >>>>>>>>>>> 25034 ls CALL open(0x237d32f9a000,0x120004) > > >>>>>>>>>>> 25034 ls NAMI "/mnt/tools/.zfs/snapshot/weekly-2023= -45" > > >>>>>>>>>>> 25034 ls RET open 4 > > >>>>>>>>>>> 25034 ls CALL fcntl(0x4,F_ISUNIONSTACK,0x0) > > >>>>>>>>>>> 25034 ls RET fcntl 0 > > >>>>>>>>>>> 25034 ls CALL getdirentries(0x4,0x237d32faa000,0x10= 00,0x237d32fa7028) > > >>>>>>>>>>> 25034 ls RET getdirentries -1 errno 5 Input/output= error > > >>>>>>>>>>> 25034 ls CALL close(0x4) > > >>>>>>>>>>> 25034 ls RET close 0 > > >>>>>>>>>>> 25034 ls CALL exit(0) > > >>>>>>>>>>> > > >>>>>>>>>>> Certainly a libc bug here that getdirentries(2) returning [= EIO] > > >>>>>>>>>>> results in ls(1) returning EXIT_SUCCESS, but the [EIO] erro= r is > > >>>>>>>>>>> consistent across both FreeBSD and Linux clients. > > >>>>>>>>>>> > > >>>>>>>>>>> Looking at this from the RPC side: > > >>>>>>>>>>> > > >>>>>>>>>>> (PUTFH, GETATTR, LOOKUP(snapshotname), GETFH, GETATTR= ) > > >>>>>>>>>>> [NFS4_OK for all ops] > > >>>>>>>>>>> (PUTFH, GETATTR) > > >>>>>>>>>>> [NFS4_OK, NFS4_OK] > > >>>>>>>>>>> (PUTFH, ACCESS(0x3f), GETATTR) > > >>>>>>>>>>> [NFS4_OK, NFS4_OK, rights =3D 0x03, NFS4_OK] > > >>>>>>>>>>> (PUTFH, GETATTR, LOOKUPP, GETFH, GETATTR) > > >>>>>>>>>>> [NFS4_OK, NFS4_OK, NFS4ERR_NOFILEHANDLE] > > >>>>>>>>>>> > > >>>>>>>>>>> and at this point the [EIO] is returned. > > >>>>>>>>>>> > > >>>>>>>>>>> It seems that clients always do a LOOKUPP before calling RE= ADDIR, and > > >>>>>>>>>>> this is failing when the subject file handle is the snapsho= t. The > > >>>>>>>>>>> client is perfectly able to *traverse into* the snapshot: i= f I try to > > >>>>>>>>>>> list a subdirectory I know exists in the snapshot, the clie= nt is able to > > >>>>>>>>>>> LOOKUP(dirname) just fine, but LOOKUPP still fails with > > >>>>>>>>>>> NFS4ERR_NOFILEHANDLE *on the subndirectory*. > > >>>>>>>>>>> > > >>>>>>>>>>> -GAWollman > > >>>>>>>>>> From nobody Thu Nov 23 16:56:05 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SbklW0Jmrz51ypV for ; Thu, 23 Nov 2023 16:56:07 +0000 (UTC) (envelope-from mhorne@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SbklV6vX0z4f51; Thu, 23 Nov 2023 16:56:06 +0000 (UTC) (envelope-from mhorne@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700758567; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=fwAbmcEsXNt11mvKJR9NsoZ5P58xm5Yvt96il3wovmM=; b=Mu05iuwc8YQL9GLbr5j67DUm/h9OeOJDO6YJx4Ia+UAcsKTqJH0a3FiOTyyGhJ8rhoA+26 Ir9LdFViNQBWv3iyqH3xfNfNC+B9YZ55s1HH0I/Iw2ibbiG12zH92nUGZ20KmYRwW+dDeB El4UWS9YZJwqj/bCZsTZxmTVCuSAmJqQ/GYIlw6hAi4oqOzmtA3c1PUdQqpbBxaA8zVxgB ef+F2gsCVaQZpN1ZzoE9vbOxj4mysd6e2GwdEKXZP1bvYR88mYrmcTL+HZLxFaq4CEDVD1 x13ayWJ2c65Mu7vZJmYFnYVZvoj03TPj39xKRVd8WX0PAV+Gf2f5iHCmsljdDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700758567; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=fwAbmcEsXNt11mvKJR9NsoZ5P58xm5Yvt96il3wovmM=; b=EC1QonCaAYTZEVFE5LbMezvbLO2ifirEFgaSepvwB9gAekX4in49FbtXEJMTy0SOCUGoTg teNPj65n2/dsh6oRmYfhQ8O/tUO0HjrWnaQzcXIOQnTEz1t+IHIECqI4xhRiif2nJCTwW4 uXGQtAGNiQxRVxSi/sVEMV/Qyrm/3NcQUxDVYzBI/UobSlDz0SCjTuoFEZmb6u4y0E4g7E zsAdifpR6NwQru+OHquJUUvPB630XaxF47FIMbf81spJUdd6WbV8RxdS4MsZI4ob1XFbGy H57PUAIsWuHTgZnzR34z2IuLr9ytlVPS8v86vcEwIKlCijnyVKFY2zA3NLBFtg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1700758567; a=rsa-sha256; cv=none; b=J+8oBx9IIdijzoKLUdtSLR0UT9KVFADcVyhhyxNJ/pB6T1fa2G/zb7fME7P0nJxx+h+Un/ xifuk8An1tBLCVcYd9gi6WVb9CcRDzIycVOW3I3gXzrGR7vduEhxhIdkMVhfTcYmsBChoU 4UMoGAG1jxCKfpgY4A5dxRbIpYZkjwaaaLvXs9f3BHOpb61Ab+vCz2MSIfslBMrZNz//Po y/xMpOn/zMAnyvYFREW99EGivRKUyal0iq+bbBGyupnI76ZbvWxaPF+Xz9blLynuGchcjB Fk4gowGDWhJraEUqR5jG07KM8qAS2nB01Sh6fgs/LU13hxB0ghs2zWDIpdrZ2A== Received: from [192.168.1.151] (host-173-212-76-127.public.eastlink.ca [173.212.76.127]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: mhorne) by smtp.freebsd.org (Postfix) with ESMTPSA id 4SbklV4m9zz14Jx; Thu, 23 Nov 2023 16:56:06 +0000 (UTC) (envelope-from mhorne@freebsd.org) Message-ID: <06ffebce-65fa-4876-bce3-6120489017be@freebsd.org> Date: Thu, 23 Nov 2023 12:56:05 -0400 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: db> reset -> panic: acquiring blockable sleep lock with spinlock or critical section held ... To: "Bjoern A. Zeeb" Cc: current@freebsd.org References: <0q526858-7s33-27qo-6q80-8o1q10415q2s@yvfgf.mnoonqbm.arg> <2b0ef12f-a158-4de3-8d25-ad2e5fcde644@freebsd.org> Content-Language: en-CA From: Mitchell Horne Autocrypt: addr=mhorne@freebsd.org; keydata= xsBNBFyS2dQBCADdiXBG8hBVLmYbxu7aSzbwLwUf3HkGFz3rooS1kwyy+SfmjZ4UKNnl9WMx WKrJ7OAZpiNH6bLQ5nsqfx09OnpWL8c/QuPbhNdUywQoqqYpRI0K8GEn//nS9Gs0KTYwVpWb XlrzP+jf3Uh/9L5mcQmStLIH4zaaqMYHW+pMuPrvBmLIHTvLj2QjOkxslrcUdord9uvxe5Ht LU8RuTpQpHOKz705Z9/v7twFdi2HtKzpLwO6SzVyu351di1J+GihsVpcT5josQV5cHbIP3Un x+kmtKBEEc/jl/zBglF7ruWUtwgbryID+2ZPEaO1Mj+RResX4LFVMusq3uUpWRb5WJXxABEB AAHNI01pdGNoZWxsIEhvcm5lIDxtaG9ybmVARnJlZUJTRC5vcmc+wsCUBBMBCgA+AhsDBQsJ CAcCBhUKCQgLAgQWAgMBAh4BAheAFiEEkp/cYPcfabAiQvACi/gnTOdUid8FAmIyDpUFCQtC z0EACgkQi/gnTOdUid8IsQf+N8IptrrCgifT5Z0/WUVFfnHThFOKf4zBjaGswsIM8+VKsKnF 15jCWHODUHP6s+dcQ4nQi81PHPsnMfBSkGPvN/X3ess2/1KUVkH+6tAJbqXDjXhD8HT+i0NM QEFIXlLnotpgIKW3yOHjKv3ZvKw9LCvUjyNY9vOJmLk/6AbbkFh+INo65nXtQWb/hM5FVEHW S+zUoU8AqZRJoVAQfj9wmIfg/HdsxeDGKL0zkv5AwKpccvb8VJNGJbCVMgoy5uQYcUeXxcie cg0VlbFLshNQTfyhVQ85vyuHahARrUWs/k8KiYODoBnW1ChtyF8yM6VZTzSYx7pINqPq2YZy i/Htd87ATQRcktnUAQgA3zt4M4ecoQqfxpjliNLujt9klDqvmkJvWmzMuMXdzlPgGRJ0doio 9YIeEdkOt6xN0pPTK/ReCZ8WqFQ8zo23u1pwGuo0CnR58XF19wyxyUuKu/PHbt+56mC8tNHm AXsMyXQmlDqWvn/WzLY7euNRtNS4QQIwtxfM5EC4GGa5KQwxn0kM7dkUSOE/cxr+/kNbHHzb gagZR4cnNUqtPPr3dYXcibCTzgz96Lyt3/qMLXX9RTBRzu+O6E+byxWOe8ar/ZlwY2b4wTQG mhgNttkSxKtxMpZnd8+DGV/bI1P5Ct/K2GeCwNyupQGON5ymn6o7jTch+qmFX0ItkBWO4zn4 9QARAQABwsB8BBgBCgAmAhsMFiEEkp/cYPcfabAiQvACi/gnTOdUid8FAmIyDtwFCQtCz4gA CgkQi/gnTOdUid/i5gf/aQ75pJR4TJFM2vVVr6PDIwTdl0b5EchB4w4s4g/zE84XNbMOQanb BginLYEhAacLQVAvM3XdvUEhwrhaMQdjdSEB1krResL3/mbxrtKwdHSMbHA3IS3XdvxFWTB7 P5JjUSPsW6hqgoidbn4w3OxaNHhs45H2b0Nx5QiKcSyepmCZuB52gCEHnEnrdaz8TFQMXOLq 94WbTmZeIjChW3FB61m1gTf0UEFjoZAfTAUB+pbwoCa4AykIeZnDC19vjsruVU9Gy5rLglwd bjsZNfXIJGOZNEvdF8FOBwM7DlXx7SYvTJcUNoNJjOKtQ0bYGVgGqYOB/y2mTjVuKeU0eOkN Uw== In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 11/20/23 15:40, Bjoern A. Zeeb wrote: > On Mon, 20 Nov 2023, Mitchell Horne wrote: > > Hi Mitchell, > >> On 11/16/23 18:21, Bjoern A. Zeeb wrote: >>> Hi, >>> >>> I seem to remember changes related to that a while ago but my cache >>> is miss for the actual change.  Are we suppoed to handle this case? >>> >>> It would be nice if "reset" would reset again the first time ... >>> >> >> Hi Bjoern, >> >> This is still my fault, I am sorry to say. If you recall, I proposed a >> fix after your initial report (back in February!), see > > now that you say I do.  I thought we had this all sorted.  Cache miss, > miss. > Maybe I had a local patch and hadn't seen it for a while because of that. > I likely dropped the ball on review and testing feedback. > > >> I posted what I believe to be the better fix just now, see >> https://reviews.freebsd.org/D42684. I will commit this ASAP along with >> some other tweaks to shutdown hooks which should (loaded word) >> eliminate this type of recursive panic during debugger reset. At >> least, that is the goal of the series :) >> >> I apologize for the delay on this, my ability to finish some of the >> work I've started has been spotty this year. > > Oh, no worries; I've been way worse this year.  Thank you for stepping up > working on this and the fixes.  It is much appreciated! > > Bjoern > You are welcome. FYI the fix has landed in main, 4e78a766f607. Mitchell From nobody Thu Nov 23 19:37:26 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SbpLP717Wz51MYT for ; Thu, 23 Nov 2023 19:38:05 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp6.goneo.de (smtp6.goneo.de [85.220.129.31]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SbpLN34yPz3ZcP for ; Thu, 23 Nov 2023 19:38:04 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=HgUjc5vF; spf=none (mx1.freebsd.org: domain of freebsd@walstatt-de.de has no SPF policy when checking 85.220.129.31) smtp.mailfrom=freebsd@walstatt-de.de; dmarc=none Received: from hub1.goneo.de (hub1.goneo.de [85.220.129.52]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by smtp6.goneo.de (Postfix) with ESMTPS id 8DEC3240CD1 for ; Thu, 23 Nov 2023 20:37:56 +0100 (CET) Received: from hub1.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPS id AB053240A46 for ; Thu, 23 Nov 2023 20:37:54 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1700768274; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Uszr54cKOwntukZ7S8GG1FQQx8D04OwRUH/18SyzIW0=; b=HgUjc5vFNQerTWuthAHwi08pM/mhLhXOphDgeo7OOYWpWtmxBth3gd/L+ixzN53Bz1DU+V N6uD3u+FLw0BepOe3lWKQ383zuxkvdM8E4vZvSGEhi22cY2ilFA8KC7carSE5hLLNINvg4 yW/b825NDr2I5TlNDBwdPHpbPUcn0n/a5ZzMd3JzQFwM8Qu5Pj8KtSrBtE60/Iw8QP+F8a 6VEGyY0iQXk4GUlWg9zGYa/NPGJ+B7XBz147fJFj078/ee6fOnEQG1l1eyCISwnn1W+TbC kuOEURoBuLqGvzAqDTrP2QSRdmKccCD9e7+NxNS2868ORJC/yRaY40RcCKnSRg== Received: from thor.intern.walstatt.dynvpn.de (dynamic-089-014-074-227.89.14.pool.telefonica.de [89.14.74.227]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPSA id 6C1872407DA for ; Thu, 23 Nov 2023 20:37:54 +0100 (CET) Date: Thu, 23 Nov 2023 20:37:26 +0100 From: FreeBSD User To: FreeBSD CURRENT Subject: uSB: uslcom: CP2102 weirdness when serial communication via cu/tip/cutecom Message-ID: <20231123203753.6ca93cc8@thor.intern.walstatt.dynvpn.de> Organization: walstatt-de.de List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: a716bf X-Rspamd-UID: da030d X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; DKIM_TRACE(0.00)[walstatt-de.de:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_NA(0.00)[no SPF record]; TO_DN_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:25394, ipnet:85.220.128.0/17, country:DE]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; HAS_ORG_HEADER(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[walstatt-de.de]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4SbpLN34yPz3ZcP X-Spamd-Bar: --- Hello, I'm facing some strange behaviour when connecting to a CP2102 USB-UART bridge built-in into a I2C-USB device (I2C is handled by an Amtel88, USB-UART is handled by an CP2102, see here: https://de.elv.com/pc-usb-i2c-interface-200958 for an sketch/overview. The device is recognised as uslcom0 on uhub6 uslcom0: on usbus0 for more details via usbconfig see below. Using FreeBSD CURRENT and 14-STABLE (both most recent) connecting to the device via cu -l /dev/cuaUX -s 115200 results in most cases in a stuck connection. The LED of the device is responding to every keystroke made in the terminal, but I never see any output (which should). In some rare cases disconneting and reconnecting the USB link and connecting via "cu" gives the opportunity for a couple of seconds to enter "?" in the terminal which provides some firmware data on the device - then the communications goes dark. This behaviour is erratic and non predictable. I tried several platforms (hardware), all USB 3, tried FreeBSD 14-STABLE and CURRENT. On a notebook (Lenovo T560) running 14-STABLE the very same situation, but trying the Lenovo with Xubuntu 23.10 gives me a working "cu" and I'm able reading environmental data from the device. Using the methusalem USB 2 port of a computer were available gives me a few seconds more on FreeBSD, before the serial connection goes mute. In cases were possible I tried the same hardware with FreeBSD and Linux Xubuntu, FreeBSD fails, Xubuntu prevails the task. At this point I was quite sure that FreeBSD's uslcom-driver might be the culprit. Out of couriosity I tried a FSBD-13.2RELENG image for AARCH64 (PINE64 I have at hand) and connected the same way, the same device acts as normal as I see under Linux Xubuntu. I'm able to take environmental data as long as I please without a problem so far. Can someone hint me how to debug such a situation? I'm unwilling to use Linux since our infrastructure is based on FreeBSD and ... well, no further explanation on that subject ;-) Thanks in advance, O. Hartmann -- O. Hartmann From nobody Thu Nov 23 19:40:12 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SbpPb0Mxlz51NfG for ; Thu, 23 Nov 2023 19:40:51 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp052.goneo.de (smtp052.goneo.de [85.220.129.60]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SbpPY70Frz3bj4 for ; Thu, 23 Nov 2023 19:40:49 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=EaKMKpvV; spf=none (mx1.freebsd.org: domain of freebsd@walstatt-de.de has no SPF policy when checking 85.220.129.60) smtp.mailfrom=freebsd@walstatt-de.de; dmarc=none Received: from hub2.goneo.de (hub2.goneo.de [IPv6:2001:1640:5::8:53]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by smtp5.goneo.de (Postfix) with ESMTPS id 5CF07240C8E for ; Thu, 23 Nov 2023 20:40:42 +0100 (CET) Received: from hub2.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPS id 849132403AA for ; Thu, 23 Nov 2023 20:40:40 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1700768440; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FX04g2yroxRcxdCnI629YwsWEyWaaTlc46E1TP837nw=; b=EaKMKpvV8SiJcbcYx+r702bOJJlH3H/An5GApm9YcjoSjl+ReyRPwKZV1PprIj9v+h2ZUP 0O00fn0G2T1Vij8dvczjTR2QsBupwYXerXdtAVwFabNer8m59wDXFH0Rwi36kI2iztGsXX mtjXVOKxFnOH0sBl6gUZz2envbcFU2pIdJ6ATLtwZj85wlvNZFPgaQ1IRJEowRoFMwEn7M ZevTVcaQ46Fvm7bBFlbFHuy15IibJ4XrCih+r79LjBQ7ypKw1yalsYTTYKPjS83dO0WrjR W6Ja0PfJYtD2J4Jy1ThpK5fgHBDWa7HYgyaLDKTyyADBVqBLuHppTIIxvZgPNw== Received: from thor.intern.walstatt.dynvpn.de (dynamic-089-014-074-227.89.14.pool.telefonica.de [89.14.74.227]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPSA id 37727240269 for ; Thu, 23 Nov 2023 20:40:40 +0100 (CET) Date: Thu, 23 Nov 2023 20:40:12 +0100 From: FreeBSD User To: FreeBSD CURRENT Subject: Re: uSB: uslcom: CP2102 weirdness when serial communication via cu/tip/cutecom Message-ID: <20231123204039.68c77fde@thor.intern.walstatt.dynvpn.de> In-Reply-To: <20231123203753.6ca93cc8@thor.intern.walstatt.dynvpn.de> References: <20231123203753.6ca93cc8@thor.intern.walstatt.dynvpn.de> Organization: walstatt-de.de List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: 4104a1 X-Rspamd-UID: 8ab116 X-Spamd-Result: default: False [-3.29 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.993]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; DKIM_TRACE(0.00)[walstatt-de.de:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_NA(0.00)[no SPF record]; TO_DN_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:25394, ipnet:85.220.128.0/17, country:DE]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; HAS_ORG_HEADER(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[walstatt-de.de]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4SbpPY70Frz3bj4 X-Spamd-Bar: --- Am Thu, 23 Nov 2023 20:37:26 +0100 FreeBSD User schrieb: > Hello, > > I'm facing some strange behaviour when connecting to a CP2102 USB-UART bridge built-in into > a I2C-USB device (I2C is handled by an Amtel88, USB-UART is handled by an CP2102, see here: > > https://de.elv.com/pc-usb-i2c-interface-200958 > > for an sketch/overview. The device is recognised as > > uslcom0 on uhub6 > uslcom0: on usbus0 > > for more details via usbconfig see below. > > Using FreeBSD CURRENT and 14-STABLE (both most recent) connecting to the device via > > cu -l /dev/cuaUX -s 115200 > > results in most cases in a stuck connection. The LED of the device is responding to every > keystroke made in the terminal, but I never see any output (which should). > In some rare cases disconneting and reconnecting the USB link and connecting via "cu" gives > the opportunity for a couple of seconds to enter "?" in the terminal which provides some > firmware data on the device - then the communications goes dark. This behaviour is erratic > and non predictable. > > I tried several platforms (hardware), all USB 3, tried FreeBSD 14-STABLE and CURRENT. On a > notebook (Lenovo T560) running 14-STABLE the very same situation, but trying the Lenovo with > Xubuntu 23.10 gives me a working "cu" and I'm able reading environmental data from the > device. > > Using the methusalem USB 2 port of a computer were available gives me a few seconds more on > FreeBSD, before the serial connection goes mute. > > In cases were possible I tried the same hardware with FreeBSD and Linux Xubuntu, FreeBSD > fails, Xubuntu prevails the task. At this point I was quite sure that FreeBSD's uslcom-driver > might be the culprit. > > Out of couriosity I tried a FSBD-13.2RELENG image for AARCH64 (PINE64 I have at hand) and > connected the same way, the same device acts as normal as I see under Linux Xubuntu. I'm able > to take environmental data as long as I please without a problem so far. > > Can someone hint me how to debug such a situation? I'm unwilling to use Linux since our > infrastructure is based on FreeBSD and ... well, no further explanation on that subject ;-) > > Thanks in advance, > > O. Hartmann Forgot this one: [...] usbconfig -d 0.7 dump_all_desc ugen0.7: at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (500mA) bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0110 bDeviceClass = 0x0000 bDeviceSubClass = 0x0000 bDeviceProtocol = 0x0000 bMaxPacketSize0 = 0x0040 idVendor = 0x10c4 idProduct = 0xea60 bcdDevice = 0x0100 iManufacturer = 0x0001 iProduct = 0x0002 iSerialNumber = 0x0003 bNumConfigurations = 0x0001 Configuration index 0 bLength = 0x0009 bDescriptorType = 0x0002 wTotalLength = 0x0020 bNumInterfaces = 0x0001 bConfigurationValue = 0x0001 iConfiguration = 0x0000 bmAttributes = 0x0080 bMaxPower = 0x00fa Interface 0 bLength = 0x0009 bDescriptorType = 0x0004 bInterfaceNumber = 0x0000 bAlternateSetting = 0x0000 bNumEndpoints = 0x0002 bInterfaceClass = 0x00ff bInterfaceSubClass = 0x0000 bInterfaceProtocol = 0x0000 iInterface = 0x0002 Endpoint 0 bLength = 0x0007 bDescriptorType = 0x0005 bEndpointAddress = 0x0081 bmAttributes = 0x0002 wMaxPacketSize = 0x0040 bInterval = 0x0000 bRefresh = 0x0000 bSynchAddress = 0x0000 Endpoint 1 bLength = 0x0007 bDescriptorType = 0x0005 bEndpointAddress = 0x0001 bmAttributes = 0x0002 wMaxPacketSize = 0x0040 bInterval = 0x0000 bRefresh = 0x0000 bSynchAddress = 0x0000 -- O. Hartmann From nobody Thu Nov 23 21:28:26 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sbrp34xpLz523G4 for ; Thu, 23 Nov 2023 21:28:43 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-21.consmr.mail.gq1.yahoo.com (sonic309-21.consmr.mail.gq1.yahoo.com [98.137.65.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sbrp254lnz4R0x for ; Thu, 23 Nov 2023 21:28:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=HrkXBeiG; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1700774920; bh=SsoqG7jkzXrAh1oo02XFf7fCClClCMdLCifuRrp09d8=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=HrkXBeiGMKDCOBTiNzNTG/E6yLKywDtGOxRcQSjfU/mjZHs3ivzHD9dFSklM0oGfihZBWXy6+VppZt07WVo/GqwsibIPgUvm2E+c5molxswSvQPg/sZ2dlKIpwI4UqemPfjxBIJ9ODasXJUrsfUdViHiN+fEMuQiNvxN+8NHW9rjmlMdLAAiae/BQxqLmLFSuqBtWjUpS24EKLrA/ybjG75pbW/kyOW2nH6GVgvG+pW5O0l1NUvRmx4vKMHsRpdF/s3DS1V1BDpsSE1LzuU/F4+L3uHkRsj9Oc9VX7k9Hn6l7oJu3QIQYYUyJelLho5ni6zMYIeuzcUPYftfdIACuQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1700774920; bh=oAh/377hmY1GTOf0HwSoJBzfUx0jQf2AqR5qO51awEj=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=NBwq5xwKVVz/gcfA4WOc+pm34O/p0VdmRgvZoxlT7Y9cSwT0SUBY5HK9MZpfWJRwU6h44ggpjvxp8W2NryVu19CvR1k+A3DO0Y2ieN6eJfphwAhde/Ptz3jkroCqJqOCxcpp75mU7r/x8Dn/L5poQLkD3/bxTT3HFtb54nn6j1H7lOnyl71hcrtZkDkgJgbVcrDiGwhf8BZ6yR0qVf9HjPRGbM+PcyMUSFbgar0ThJURmNyMp7bBPFB6Z9tBqnmX3FdvSK5l7LQd8GAGePdpUp55YXXa5WBLgKBg2YJkPB8XTYpV/BxJYmYG67wXr1kMKmTLCWrPuIGQ/Tn+tbreOw== X-YMail-OSG: oYoL3wQVM1nD.nmd8PWwsT6mGUl5YmtvwtM3H47QlAGcIaeoQJAUNOAy96b8yaF GPILkE7Mx5kDx3UgKUsA8Bw23GWcjPxmZOx5ZQMqAT2cVuagF9Fgd0PX.GZnU9.gwE757IQbYISt DFo_HpQV3OytkqBzMs6fJCqGt0FyfGp7maE6tx.zZTfHI7cIM19kEf4ZeOq.T718ZrJ1e3hb39tQ 2aE7YRuSv_I7svMmzEGlqJfNxnDV88o6bAzGcjhGBBo3AR2p1NQqluelZv9ektJLDQE_pSdHotUg wX7inermDR_MHnF.Fb0wKoWrexOvwCwU102MGL_aG.aIxxIIEZHiiZ5SX1B_Z7SmwQTb06R4SGt1 pJ_gbbIYseP8JMCuHnVhrO11M1ZyjWTub.A7EkcHSBEKNmR7AuLRdivU3QPfJDrVr9bCcG0kMsyW gGIbWnzKsYO6XbhCu147fLQjwja4RUoXM_oI4fwCNfGjTwmznVWRvksjtXH9tb4oPuE.zS8oiC_H 0s0ZIXIbnUw5vdKRZ89GdrsRAEr47hazv.Jka0oZY1tF_8a1OYTw_i34.DlgeDVsiipmVhbtf84O 9AdWyvUMiq18bVydg1V1J3NNPXudb2ikSzb3HJA_YSIIXpg0jM7UuwLZPS9eXCOF3dNlmu3ZI5jZ 6NHztnA7YFKUOgA98_ZJaaQ..onYHfYQJ5shiyrfvT6FgQ.xilUNMzMemYUJWIHmSxXt43XNk2N1 gRRB9yx4yXVqM7Yd1b5HOsyOcnoNPUsburAPbXbDcT7w0Y4qg7wh7Ujq6RSZNakSRrhMr2odb1RP Bs.sO9gxt1w3hk.K3u7lZfbccgzIorZrgDFPXGN3sC_nBRr3j5uylUytPnJDjHuDBFdVi6YbVmnw 7qE6OgY9DTxf4w4TzvyHV21nd3uZlJTLNoo1i2Jzuj8j0.AZJunXApEc4dloqFprVCMnwWbnb0CL AlE6cEzHtdeOc99fo8D_VZAmy8Ns8GGsCMvnbAblzVCYB0J1mmO.lHTL5hN7Zq0b0dbZWIq7zXFC zGeBp_mVGf5oP3VKKg4e4vxofs5kgPtFiWm3YLivEgAO_v6vHczY8LcE50xsXYrMG1kxKOdSoWLt EHgSyLbky8gz9e6uf12CZbdFQ_xvMnF82AxQS4lBWFhnLlleRA9J2iRUyh79DCWa7z73H2zr75Kj m9CO81yxFyWGNs9DVK2R6V6ZlHiEr385o_QfqUN6F7k59XHwDcNPQdT0vvf1hADtu_hAVHDho1ou qLQF1vtb_W5khu4s2Y.JkuhotpYF.ZCaR5O_NNT4HoB9XXt.tNZrSeFIwdEloFtTvKZvtwsnUAFo q54W_moHxqu.iiX2.oqLrSX3Jn4yoKG0w.83bxYvhiS_zwqt3GamqmY7p.YRc_f5M9qRE..8E7tW 1iz0rJywVQm1KGMHAnrjLXTgrfXXZl1qPT2k8tH6ktlEPrRr7kC_nyrGb3xHp4luBNEBc42wUZ9. lkM69pyQOqQ7Zky0XBLgwGE4YORiu3rqE4W5kIk0we_EyuY9ghZ4ifx6Azx8ZSlVCeYxNjVIRyy4 8Gc3o8Kzi.08Shg4u6UI7U1l1gIE68JkmzU3hsnwFTQw7AzMTAwTNveWUvRud30l0jxmSS5oXs58 kWbBmiqxn8_hORkPkMRgOgiePXPl9.77PlQfCPSE4WpowFHqtAOd4yQNw9orxl.leRjAJwImvygq Da1G5u4dunSZ4ExOzlrueSWBIZtqwiU2KmshoLZAVeBK55h0uLwWt50xh.OA7thWhA2oykNJgudv Vp7sn_AUYtkaprWyOKsQCiS19oLbhFwH0kPWZesdf2SVHYbDyJYeBoSqGqzciUKQ4g4V5zOty2jq 92ZRZrTebSr6HvH6awt6FbS_SPg9yETHbKhmdTYEAJ0yYBKwz0SydLz9l1aeTubnGL63ZUgL6H6h ZmZ.DJowt7t_8hiUqOwUvo.pzaz6r7nqaKTN_vZBE0Dq9Hi0ERXIeSaZkoEac4dzV3EINVqOLc3L pf58Xa957ZuyAd0MR8sjm_rLiU3eFW_aaK7Seb5fvrVc.hb3DCIzQYZ4a_BlvLEiOw_1ceH5eQiu iQ_jcvYghw8YTz3A7t2eKK4Qk736wNlXM7fcV3E4l6Jxzm6tIXv.oZfyGlPUhG.ZTF58X_4YmIkM gy3gIJ3nmDufVOCoqheJmziXckV7SJesgkownffuBPmncsF8NKQQBROMbjAVeMOhUyU51jM83SxY PISNKccdAthFi.PVZJOMLegr.LIZJlnGipDVNC.SmEoijM_TWB7xszi31LE6VQA-- X-Sonic-MF: X-Sonic-ID: e259fc81-7b0e-4ab5-91a8-00a367f08e1f Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Thu, 23 Nov 2023 21:28:40 +0000 Received: by hermes--production-gq1-6775bfb8fc-hj2hj (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID d67bf145afb11b16c3c1ca1b18a34e47; Thu, 23 Nov 2023 21:28:37 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\)) Subject: Re: "options MAXMEMDOM=2" vs. amd64 DBG kernel booting: 3000+ "kernel: Process (pid 1) got signal 5" notices during booting Date: Thu, 23 Nov 2023 13:28:26 -0800 References: <6F60DA66-1333-4CF0-B433-F0CE56F5D301@yahoo.com> To: Current FreeBSD In-Reply-To: <6F60DA66-1333-4CF0-B433-F0CE56F5D301@yahoo.com> Message-Id: <91C200DC-94B2-4831-B3DF-9C4627556ADB@yahoo.com> X-Mailer: Apple Mail (2.3774.200.91.1.1) X-Spamd-Result: default: False [-3.32 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.82)[-0.818]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.147:from]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.147:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Sbrp254lnz4R0x X-Spamd-Bar: --- On Nov 21, 2023, at 21:43, Mark Millard wrote: > While my kernel/world build procedures build both DBG and NODBG > kernels and worlds, I normally run the NODBG kernel and world, > using DBG only when I need to for problem investigation. >=20 > I recently had reason to use the DBG kernel and found it got the > oddity of 3000+ instances of "kernel: Process (pid 1) got signal 5" > during booting, as reported in /var/log/messages . An example is: >=20 > . . . > Nov 20 23:13:09 7950X3D-UFS shutdown[20174]: reboot by root:=20 > Nov 20 23:13:09 7950X3D-UFS syslogd: exiting on signal 15 > Nov 20 23:14:21 7950X3D-UFS syslogd: kernel boot file is = /boot/kernel/kernel > Nov 20 23:14:21 7950X3D-UFS kernel: got signal 5 > Nov 20 23:14:21 7950X3D-UFS kernel: Process (pid 1) got signal 5 > Nov 20 23:14:21 7950X3D-UFS syslogd: last message repeated 3133 times > Nov 20 23:14:21 7950X3D-UFS kernel: intsmb0: at device 20.0 on pci0 > . . . >=20 > This stopped when I commented out the: >=20 > options MAXMEMDOM=3D2 >=20 > that I've had historically and built, installed, and booted > the resulting DBG kernel. >=20 > I'll note that I never had the messages for the NODDBG kernel, > despite it also having that line. >=20 >=20 > For reference: >=20 > # uname -apKU > FreeBSD 7950X3D-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT #2 = main-n266130-d521abdff236-dirty: Tue Nov 21 21:03:11 PST 2023 = root@7950X3D-UFS:/usr/obj/BUILDs/main-amd64-dbg-clang/usr/main-src/amd64.a= md64/sys/GENERIC-DBG amd64 amd64 1500002 1500002 >=20 > # ~/fbsd-based-on-what-commit.sh -C /usr/main-src/ > d521abdff236 (HEAD -> main, freebsd/main, freebsd/HEAD) Update ASLR = stack sysctl description in security.7 and mitigations.7 > Author: Ed Maste > Commit: Ed Maste > CommitDate: 2023-10-24 22:29:25 +0000 > branch: main > merge-base: d521abdff2367a5c72a773a815fc3d99403274f5 > merge-base: CommitDate: 2023-10-24 22:29:25 +0000 > n266130 (--first-parent --count for merge-base) >=20 A few more related notes follow. First off, while I've historically built with NUMA support present in the kernel, I'd only ever configured a UEFI/system to indicate NUMA back in some experiments in my early days of ThreadRipper 1950X use (years ago now). The "options MAXMEMDOM=3D2" had been in place since back then, with no known contributions to any problems. (Until recently the 1950X was the only amd64 context. Now there is also a Ryzen 9 7950X3D.) Recently, I've been testing a potential patch to UFS associated with changing from: #define UFS_LINK_MAX 32767 to, say, #define UFS_LINK_MAX 65530 I had discovered that "bulk -a" was broken on amd64 for UFS, and was so via hitting UFS_LINK_MAX for how the associated directories trees are currently structured. I'm not the author of the UFS patch. The "options MAXMEMDOM=3D2" vs. not status changed the behavior of my UFS "bulk -a" runs, not just the messages reporting: "kernel: Process (pid 1) got signal 5". With MAXMEMDOM=3D2 I was getting occasional random port build failures with the non-debug kernel for a UFS context. (The original discovery was in a non-debug kernel/world context.) I'd not gotten to 2 hr into a "bulk -a" without getting such a random port build failure. By contrast, my normal non-debug kernel ZFS context did not get such random port build errors during "bulk -a". (I've done multiple from-scratch "bulk -a" builds from the same, non-updated /usr/ports tree [same by content].) Without MAXMEMDOM=3D2 I am not getting the random port build failures in any context that I've tried. So far as I can tell, this "Without MAXMEMDOM=3D2" type of context is simply working correctly. A from-scratch "bulk -a" worked for a non-debug kernel used with UFS, no evidence of any problems, the port build failures being ones that happen in other contexts as well. For reference: [main-amd64-bulk_a-default] [2023-11-21_22h14m34s] [committing:] Queued: = 34683 Built: 33843 Failed: 163 Skipped: 357 Ignored: 320 Fetched: = 0 Tobuild: 0 Time: 33:17:31 [33:17:32] Logs: = /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-11-21_2= 2h14m34s =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Fri Nov 24 07:10:12 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sc5j94y3lz51kSc for ; Fri, 24 Nov 2023 07:10:21 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Received: from mail.flex-it.com.ua (mail.flex-it.com.ua [193.239.74.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sc5j85fHxz4cqr for ; Fri, 24 Nov 2023 07:10:20 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of shuriku@shurik.kiev.ua designates 193.239.74.7 as permitted sender) smtp.mailfrom=shuriku@shurik.kiev.ua; dmarc=none Received: from 93.183.208.50.ipv4.datagroup.ua ([93.183.208.50] helo=[192.168.200.105]) by mail.flex-it.com.ua with esmtpsa (TLS1.3) tls TLS_AES_128_GCM_SHA256 (Exim 4.96.2 (FreeBSD)) (envelope-from ) id 1r6QKK-000NCV-1n for freebsd-current@FreeBSD.org; Fri, 24 Nov 2023 09:10:12 +0200 Message-ID: <17265de6-fcbb-4154-adbd-4c56552b747d@shurik.kiev.ua> Date: Fri, 24 Nov 2023 09:10:12 +0200 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: uk-UA To: freebsd-current@FreeBSD.org From: Oleksandr Kryvulia Subject: openzfs and block cloning question Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ACL-Warn: SPF failed. 93.183.208.50 is not allowed to send mail from shurik.kiev.ua. X-Spamd-Result: default: False [-1.17 / 15.00]; NEURAL_HAM_SHORT(-0.92)[-0.915]; NEURAL_SPAM_LONG(0.77)[0.774]; NEURAL_HAM_MEDIUM(-0.74)[-0.742]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; MLMMJ_DEST(0.00)[freebsd-current@FreeBSD.org]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:35297, ipnet:193.239.72.0/22, country:UA]; MIME_TRACE(0.00)[0:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[shurik.kiev.ua]; TO_DN_NONE(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Sc5j85fHxz4cqr X-Spamd-Bar: - Hi, Recently cperciva@ published in his twitter [1] that enabling block cloning feature tends to data lost on 14.  Is this statement true for the current? Since I am using current for daily work and block cloning enabled by default how can I verify that my data is not affected? Thank you. From nobody Fri Nov 24 08:17:41 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sc7CL5yNdz519M6 for ; Fri, 24 Nov 2023 08:18:06 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sc7CL3Tm6z3NWd for ; Fri, 24 Nov 2023 08:18:06 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1700813878; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=5C6WJNbDsKYgEASSyT1Mgf2H49Pj9KLVWma38d+09rY=; b=2oRSQ2cVB/jmn+YtfrhyxyUfpKIPhD0fiDvnONxPgaBiQXnWNIp/H4wsMXFkVYhj5GAmjG eHlY5YYsW4GK5QrDtJjNbKEhOP3GP2FjdZkSOebaLIMCKj0DFQvrire6fj3ADmt6sOqNFi u/iPy4EQRTebeSxNKydVKNbl1xIKW80tqdyuKyD99i1f6Kym1ctxqr5WA5oIJSC2YmYipc 515Kt6BZrvwbPuRvY4nZpvP4mvO5EWCfEyXW/E5P/4+mp4KuhGUcbHoXcTKOYwyKrYVSRm 5aky4ITRMtV4pBqUYUy1n6oOuMz4VaaddSvDmGZuDrvTI6vgLvlo+rLypzAbjg== Date: Fri, 24 Nov 2023 09:17:41 +0100 From: Alexander Leidinger To: Oleksandr Kryvulia Cc: freebsd-current@freebsd.org Subject: Re: openzfs and block cloning question In-Reply-To: <17265de6-fcbb-4154-adbd-4c56552b747d@shurik.kiev.ua> References: <17265de6-fcbb-4154-adbd-4c56552b747d@shurik.kiev.ua> Message-ID: <25dab6bbc32f85ac710535776cd9d376@Leidinger.net> X-Sender: Alexander@Leidinger.net Organization: No organization, this is a private message. Content-Type: multipart/signed; protocol="application/pgp-signature"; boundary="=_a36447dd47b0b84069d95728201a34cd"; micalg=pgp-sha256 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE] X-Rspamd-Queue-Id: 4Sc7CL3Tm6z3NWd This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --=_a36447dd47b0b84069d95728201a34cd Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8; format=flowed Am 2023-11-24 08:10, schrieb Oleksandr Kryvulia: > Hi, > Recently cperciva@ published in his twitter [1] that enabling block > cloning feature tends to data lost on 14.  Is this statement true for > the current? Since I am using current for daily work and block cloning > enabled by default how can I verify that my data is not affected? > Thank you. Block cloning may have an issue, or it does things which amplifies an old existing issue, or there are two issues... The full story is at https://github.com/openzfs/zfs/issues/15526 To be on the safe side, you may want to have vfs.zfs.dmu_offset_next_sync=0 (loader.conf / sysctl.conf) for the moment. Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_a36447dd47b0b84069d95728201a34cd Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc; size=833 Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmVgXDMACgkQEg2wmwP4 2IaYOw//dc0S3iqJM4K1GKLjchtaLVd0aA9xwMVBlqzLIo2Zcc56NNgSzCIe4jSv 2jIkYR1h2FrvtBgz94uTtTTFh6Ee8Nd8TwCPfifg0kK8d8+Em9H+xWLmZZL/iCSX bcmEothXXVLkR6euMDtydP2NmD5ZqfnzWQ3/9G/dOyGhPj8RiJQN+G4QvVdqt3d5 kUtR3V1eEYkibdNIHh6alGuGq4jbc+nHKZQAoFXAKMmbZzsVe4MWNyj/oImfX8ah TC3vqPItf6cNTjaGqX32mpLoMHPUlFKG3qL3Qki1y8nAfUacOPVUipXJcVThHeYp Q5tBZuiXkUqX5mKFoQNgFK7fXcEOy4VqMQH/KffIsW6lMLRV7SlQkpwmpgQfV8zp Ldsgy4Hcz8y9PNvCFn0PsO3OMLwTtxKD3mOHyXpjs1OqQ2p1AUInYfTFvCRr/Pdy /cOPgmt3Fw+54KxvFt9bmSouD7EVw9pnOva5Dn97L/yy0Y5fYhX2/13Y2GPfx3E9 pjBfLelusE9hiePzEY2stxdgY3avZ7QT2pjCSOGPCdy8yzEmnLmhz2GrrqT3IuU4 AIQtlD6/6McEO5lVOA0vv0TbNkXcfGoAG7f11Y1gSjMxpHAlUHJuf9An5eFgoDn7 pEUV/sbkoivoyUnibz8V7+H79yPBzR1qHz3tS/0n3TcM1692VUc= =+n7E -----END PGP SIGNATURE----- --=_a36447dd47b0b84069d95728201a34cd-- From nobody Fri Nov 24 10:20:21 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sc9xT4rjQz52CZ1 for ; Fri, 24 Nov 2023 10:21:17 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (mailgate.leidinger.net [IPv6:2a00:1828:2000:313::1:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sc9xS15Htz4MQr for ; Fri, 24 Nov 2023 10:21:16 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=leidinger.net header.s=outgoing-alex header.b=kGJYHosg; spf=pass (mx1.freebsd.org: domain of Alexander@Leidinger.net designates 2a00:1828:2000:313::1:5 as permitted sender) smtp.mailfrom=Alexander@Leidinger.net; dmarc=pass (policy=quarantine) header.from=leidinger.net List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1700821270; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=n1rnCcWk7iqX1m0f0UlDuZAb5gxH2BrTvwcaPS7sMpo=; b=kGJYHosgDckI4boFbW6RNH36tdpKevJR0+RmoAw5eUzOzUhmzPLJ7O1/b+OYdv4AH/R06p 0GUemB/bg3TVbWzmYPDL145BgnjK/ofRcozE3hhb1CpA+uEfWTphr6Ourne/GDPOz3LwT6 bDng83Nd9z8Ca3TVaY8PaoMGZB1AQu/+21Ojgt6uMs0MAzvqtNd5xUwbN/TWn93AuJR04v Rd4C9AgwW84X+Z9yM08Vz4XY4DMrz56bLksN3n7CvlGr7GWmtchYiivLxgeRM0ApVfG5GO gABlu7o9MlkYyWnOLFqY5uHttRdkY/FpZ227TwrfLlImxeEfk03/xx+YWvn9cA== Date: Fri, 24 Nov 2023 11:20:21 +0100 From: Alexander Leidinger To: Current Subject: What is rc.d/opensm? Message-ID: <2271e3010e73deed37988cb4475b675c@Leidinger.net> X-Sender: Alexander@Leidinger.net Organization: No organization, this is a private message. Content-Type: multipart/signed; protocol="application/pgp-signature"; boundary="=_7731b32b3d698067bee87015c09d1c40"; micalg=pgp-sha256 X-Spamd-Result: default: False [-5.09 / 15.00]; SIGNED_PGP(-2.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.99)[-0.991]; DMARC_POLICY_ALLOW(-0.50)[leidinger.net,quarantine]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; R_SPF_ALLOW(-0.20)[+mx:c]; R_DKIM_ALLOW(-0.20)[leidinger.net:s=outgoing-alex]; MLMMJ_DEST(0.00)[current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_ZERO(0.00)[0]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; ASN(0.00)[asn:34240, ipnet:2a00:1828::/32, country:DE]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[leidinger.net:+]; HAS_ATTACHMENT(0.00)[]; HAS_ORG_HEADER(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Sc9xS15Htz4MQr X-Spamd-Bar: ----- This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --=_7731b32b3d698067bee87015c09d1c40 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed Hi, for my work on service jails (https://reviews.freebsd.org/D40370) I try to find out what opensm is. On my amd64 system I don't have a man page nor the binary (and man.freebsd.org doesn't know either about opensm). Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_7731b32b3d698067bee87015c09d1c40 Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc; size=833 Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmVgePQACgkQEg2wmwP4 2IasRxAAj5JOtPAY3WDXzrD8G88zh7G9sSYYaw12AlwJkvkIVmCRtMW0qox+IDsg ljhuoGq8TmO7uF6BKocxtMYGAzdhUlvpSyMT/UfCQt9Jk4QehfPBcXJ7XBJC0jIO nMj6sKuK+h1Rn4rYydPFctz4IFyQcB7g8L7tH2Gjndw4HaoNaoV0k3C3EvYTyHAQ NoJ7hcUYCtwtqM7Y8pVwciqZVhYaCmQ9CLUxURrwPXFh8R4fVA6GeDkJcec70hmC v1CR2KWqn7F587Ic8gQYn6INstqWf69FsCnrDXbuMTig9LU8dcHI3lBiigEE/Eqy 0JEvF7SPRLLQhlkopUrDL1lbTBMnE6rKHNHogl5YrqgW6yAs77LkKXCikzcChkKP KrLGVrXvAwqoWNPcsqbPxno/befWti3kV8XuyunBUM82Mob2JjvYYwyPTBi3H5vv gDHxlj/Ll6Nv3DLhTqj9fUM9SFwnvP6xNQN3gwj2Dcla1IJC5yKOi2hcS/DBvAGJ U0h1nwr3NF6WZJCSECF0bEupwpoVja8XnRnUjgvvsj6s6VmQs+Gg1U3IeYSC+Z0R wlW9E66I6Y2rpGCEawXDrjtdi42sK63BCfqRWDxpO+NUARUpOJVQB0KtPRp2aHhV V6t47C1FQFqwnMhgw7AeXTolwIxn8eRuH4XN39yEmE4bgPqHZkk= =AD3b -----END PGP SIGNATURE----- --=_7731b32b3d698067bee87015c09d1c40-- From nobody Fri Nov 24 10:36:18 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ScBGz2tSJz52HwZ for ; Fri, 24 Nov 2023 10:36:27 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Received: from mail.flex-it.com.ua (mail.flex-it.com.ua [193.239.74.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ScBGy42Blz4QT9 for ; Fri, 24 Nov 2023 10:36:26 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of shuriku@shurik.kiev.ua designates 193.239.74.7 as permitted sender) smtp.mailfrom=shuriku@shurik.kiev.ua; dmarc=none Received: from [188.231.181.61] (helo=[10.2.1.104]) by mail.flex-it.com.ua with esmtpsa (TLS1.3) tls TLS_AES_128_GCM_SHA256 (Exim 4.96.2 (FreeBSD)) (envelope-from ) id 1r6TXr-000Jf7-1k for freebsd-current@freebsd.org; Fri, 24 Nov 2023 12:36:23 +0200 Content-Type: multipart/alternative; boundary="------------pn6MHX6sm0GIlODVvhmJVdTQ" Message-ID: <3e296f9b-53b9-4a2c-8705-2404a2876db7@shurik.kiev.ua> Date: Fri, 24 Nov 2023 12:36:18 +0200 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: What is rc.d/opensm? Content-Language: uk-UA To: freebsd-current@freebsd.org References: <2271e3010e73deed37988cb4475b675c@Leidinger.net> From: Oleksandr Kryvulia In-Reply-To: <2271e3010e73deed37988cb4475b675c@Leidinger.net> X-ACL-Warn: SPF failed. 188.231.181.61 is not allowed to send mail from shurik.kiev.ua. X-Spamd-Result: default: False [-0.35 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.997]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; NEURAL_SPAM_LONG(0.93)[0.930]; R_SPF_ALLOW(-0.20)[+mx:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; ASN(0.00)[asn:35297, ipnet:193.239.72.0/22, country:UA]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[shurik.kiev.ua]; TO_DN_NONE(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4ScBGy42Blz4QT9 X-Spamd-Bar: / This is a multi-part message in MIME format. --------------pn6MHX6sm0GIlODVvhmJVdTQ Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit 24.11.23 12:20, Alexander Leidinger: > Hi, > > for my work on service jails (https://reviews.freebsd.org/D40370) I > try to find out what opensm is. On my amd64 system I don't have a man > page nor the binary (and man.freebsd.org doesn't know either about > opensm). > > Bye, > Alexander. > See man /usr/src/contrib/ofed/opensm/man/opensm.8 and https://wiki.freebsd.org/InfiniBand --------------pn6MHX6sm0GIlODVvhmJVdTQ Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit 24.11.23 12:20, Alexander Leidinger:
Hi,

for my work on service jails (https://reviews.freebsd.org/D40370) I try to find out what opensm is. On my amd64 system I don't have a man page nor the binary (and man.freebsd.org doesn't know either about opensm).

Bye,
Alexander.


See man /usr/src/contrib/ofed/opensm/man/opensm.8 and https://wiki.freebsd.org/InfiniBand


--------------pn6MHX6sm0GIlODVvhmJVdTQ-- From nobody Fri Nov 24 10:37:50 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ScBJc6jK6z52Jf2 for ; Fri, 24 Nov 2023 10:37:52 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Received: from mail.flex-it.com.ua (mail.flex-it.com.ua [193.239.74.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ScBJc37cyz4Rcn for ; Fri, 24 Nov 2023 10:37:52 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of shuriku@shurik.kiev.ua designates 193.239.74.7 as permitted sender) smtp.mailfrom=shuriku@shurik.kiev.ua; dmarc=none Received: from [188.231.181.61] (helo=[10.2.1.104]) by mail.flex-it.com.ua with esmtpsa (TLS1.3) tls TLS_AES_128_GCM_SHA256 (Exim 4.96.2 (FreeBSD)) (envelope-from ) id 1r6TZG-000Jf7-31 for freebsd-current@freebsd.org; Fri, 24 Nov 2023 12:37:50 +0200 Message-ID: Date: Fri, 24 Nov 2023 12:37:50 +0200 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: openzfs and block cloning question Content-Language: uk-UA To: freebsd-current@freebsd.org References: <17265de6-fcbb-4154-adbd-4c56552b747d@shurik.kiev.ua> <25dab6bbc32f85ac710535776cd9d376@Leidinger.net> From: Oleksandr Kryvulia In-Reply-To: <25dab6bbc32f85ac710535776cd9d376@Leidinger.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ACL-Warn: SPF failed. 188.231.181.61 is not allowed to send mail from shurik.kiev.ua. X-Spamd-Result: default: False [-2.38 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; R_SPF_ALLOW(-0.20)[+mx:c]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-0.09)[-0.088]; XM_UA_NO_VERSION(0.01)[]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ASN(0.00)[asn:35297, ipnet:193.239.72.0/22, country:UA]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[shurik.kiev.ua]; TO_DN_NONE(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4ScBJc37cyz4Rcn X-Spamd-Bar: -- 24.11.23 10:17, Alexander Leidinger: > Am 2023-11-24 08:10, schrieb Oleksandr Kryvulia: >> Hi, >> Recently cperciva@ published in his twitter [1] that enabling block >> cloning feature tends to data lost on 14.  Is this statement true for >> the current? Since I am using current for daily work and block >> cloning enabled by default how can I verify that my data is not >> affected? >> Thank you. > > Block cloning may have an issue, or it does things which amplifies an > old existing issue, or there are two issues... > The full story is at >     https://github.com/openzfs/zfs/issues/15526 > > To be on the safe side, you may want to have > vfs.zfs.dmu_offset_next_sync=0 (loader.conf / sysctl.conf) for the > moment. Thanks for clarification! From nobody Sat Nov 25 11:41:03 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Scqgp4tp8z52FHM; Sat, 25 Nov 2023 11:41:42 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Scqgn5Cnzz4Vnh; Sat, 25 Nov 2023 11:41:41 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=hoiORWyH; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::534 as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-x534.google.com with SMTP id 4fb4d7f45d1cf-548f0b7ab11so3744841a12.1; Sat, 25 Nov 2023 03:41:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700912500; x=1701517300; darn=freebsd.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=azjIXGhUpx8mRf9YywHFybitCenT4RIf9yxVGY4Q2P0=; b=hoiORWyHZ5ZxPXQkm9pn65hjiDLuv7C/MJ+Vxz5xypdrXDMwnzlV8JWaI53IszeI+x sJDEygaYXlOkCGV/PCWqekyH627QP09CegAcO9Fc1Mxqp781bWofyG43JTwTUyoEAqDr iUnRBD0uOuopLKrfNdJonaBS8MtKD4p3NaBqvS7BjiFKDYAXZJTmMy4HOFCG+6jtbJIT qSOQGpn9Mo/Xk/yIV6i/d1853ykmlW0fHsQPbO+M8+fRvpKFXV8Bsm3NWnbMcb50kTcj 5L/IgqnD6wN1Ksx4HkARTgp+HCsPA7dHA1Q2xJVZUxbvY9H1NtHAOxNi6fRZ/iExKBQc goTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700912500; x=1701517300; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=azjIXGhUpx8mRf9YywHFybitCenT4RIf9yxVGY4Q2P0=; b=vW1Is14FHhJjV0zzIrmZvrhJ11v5iETF1ImEnukYrtwYoDKjJJ7UToOeRfHBLqhFT2 ANl4vj8X+RHgBcvoiwfbJtJR53NTOE1wSc12MpMZaIhZJnQujozrGqpi82Tt4glIe/ov qpk+K5VdlXjQXDqY86kIkfjg0K8oK/hO87Dk+AYv3ZPZBSJBKSY7JZWstRInkhgeLsWL T5q4uLuPsNULsFKjC4V4wZf+xPf1QCEcvogWl4VPnV3uqflR+qxhbRCOLqVkmJygHIbN Q9Q/ulFue3MBCXgJG263suEISC2K8X98l4h9IKSz9WPytRrh9CNI60Y+W8/Uxru6LaVA iuqw== X-Gm-Message-State: AOJu0YzO05HLZ5j3VJQfFJZVZoGmeHBUQse1MK6iXGWzzXWvR+4THN30 v0VuaTWrFjJldNB0EGVQDbe5cUMif6pn7zNx+SeVVh1eTvo= X-Google-Smtp-Source: AGHT+IGpCIUAyjFh3G2HUz4/okVYKzia3wH/NAaFC75uH5L3c2mUsSVPHkOyuVI0/FhJpw74VS3WFG/LcW4W1pbsWhA= X-Received: by 2002:a17:906:15d:b0:a0b:6d32:2e09 with SMTP id 29-20020a170906015d00b00a0b6d322e09mr1386684ejh.36.1700912499758; Sat, 25 Nov 2023 03:41:39 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Sat, 25 Nov 2023 12:41:03 +0100 Message-ID: Subject: Fwd: Should we boot the FreeBSD kernel in ELF format or in zImage format ? How? To: freebsd-hackers , freebsd-arm@freebsd.org, FreeBSD Current , FreeBSD Mailing List , freebsd-xen@freebsd.org, royger@freebsd.org Content-Type: multipart/alternative; boundary="00000000000057995f060af88d0b" X-Spamd-Result: default: False [-2.48 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.995]; NEURAL_HAM_LONG(-0.85)[-0.854]; NEURAL_HAM_MEDIUM(-0.63)[-0.626]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::534:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org,freebsd-arm@freebsd.org,freebsd-current@freebsd.org,freebsd-questions@freebsd.org,freebsd-xen@freebsd.org]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_FROM(0.00)[gmail.com]; RCPT_COUNT_FIVE(0.00)[6]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4Scqgn5Cnzz4Vnh X-Spamd-Bar: -- --00000000000057995f060af88d0b Content-Type: text/plain; charset="UTF-8" Hello to everyone. we have just virtualized Debian 12 on our arm (32 bit) Chromebook. As host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It works great. But our goal is different. We want to virtualize FreeBSD as domU. Can we have a working Xen PV network driver for a FreeBSD arm guest ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I would like to know if Julien's work was accepted upstream by FreeBSD, in which case FreeBSD as a Xen guest on arm should work if we enable the Xen PV drivers in the FreeBSD on arm kernel. If Julien's work was not accepted upstream by FreeBSD, we will have to find his patches and apply them ourselves to the FreeBSD on arm kernel. We found these slides : https://events.static.linuxfound.org/sites/events/files/slides/Porting%20FreeBSD%20on%20Xen%20on%20ARM%20.pdf Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what we want to find. It looks like when that slide presentation was written, there were some limitations on FreeBSD Xen guests. For example, for our debian bookworm guest, I am using vcpus = '2' to match the number of real cpus on our Chromebook, but slide 13 mentions support for only 1 VCPU with a FreeBSD guest, so I will need to change that vcpus = '1' in the FreeBSD guest config unless support for 2 or more vcpus was added later, which is possible because that slide presentation is 9 years old. Here is where I would expect to find the XENHVM FreeBSD on arm kernel config file: https://cgit.freebsd.org/src/tree/sys/arm/conf But it is not there unless I am not understanding something correctly. For now, unfortunately conclude that the support for Xen on arm that Julien Grall mentioned in that slide presentation 9 years ago was never added to the official FreeBSD source code. I am searching the web now to see if the patches that Julien Grall wrote are still posted somewhere online. If we cannot find them, we can ask here and on the xen-users mailing list. Julien regularly reads that list and responds to question about Xen on arm, so I think he will tell us how to find the patches if we cannot find them online. According to this page from the FreeBSD wiki: https://wiki.freebsd.org/Xen I think FreeBSD only supports Xen on x86, not arm. So this is going to be a bit of a challenge to get a Xen FreeBSD guest on arm working. We know Julien Grall has some patches that made it work in the past ! I found a slightly newer slide presentation by Julien here: https://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd It is about the same, but it mentions the GENERIC FreeBSD kernel supports Xen on arm64, but still says we need the XENHVM FreeBSD config for Xen on arm 32 bit, which I haven't found online yet. Please,take a look at this output of the linux kernel that can boot on Xen, and the FreeBSD kernel that cannot : % file zImage-6.1.59-stb-xen-cbe+ zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (little-endian) % file FREEBSD-XENVIRT FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), not stripped The FreeBSD kernel that won't boot is in ELF format but the Linux kernel that does boot is in zImage format. I spent time reading the docs on xenbits.xenproject.org, and according to those docs Xen on arm only knows how to boot a kernel in the zImage format, so the FreeBSD kernel is in a format that modern Xen incorrectly detects as an x86 kernel. I also watched Julien Grall's 30 minute video presentation of his work to boot FreeBSD/arm on Xen at FOSDEM 2014 here : https://archive.fosdem.org/2014/schedule/event/freebsd_xen_arm/ In that video, and in other places, Julien mentions that the boot ABI for FreeBSD/arm on Xen was not yet developed and he was getting occasional crashes and needed to investigate the problem. He mentioned the zImage ABI that Linux uses, but pointed out FreeBSD does not use that format, and back then it was an open question which format to use to boot FreeBSD/arm on Xen. Unfortunately, nine years later, the only supported format is still the zImage format that Linux uses. It looks like Julien's work back then was using an ELF binary to boot FreeBSD/arm on Xen instead of the supported zImage format that Linux uses and the modern Xen toolstack exits with an error when trying to boot the FreeBSD ELF formatted binary that Julien's patch creates. So the best solution would be to try to port the rules to build a FreeBSD kernel in the zImage format instead of the ELF format. I have been studying the Makefiles in Linux to see how Linux builds the Linux arm kernel in the zImage format, but it is not trivial to understand. -- Mario. --00000000000057995f060af88d0b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello to every= one.

we have just virtualized Debian 12 on our arm (32 bit) Chr= omebook. As host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It=20 works great. But our goal is different. We want to virtualize FreeBSD as domU. Can we have a working Xen PV network driver for a FreeBSD arm=20 guest ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I would like to know if Julien's work was accepted upstream by= =20 FreeBSD, in which case FreeBSD as a Xen guest on arm should work if we=20 enable the Xen PV drivers in the FreeBSD on arm kernel. If Julien's wor= k was not accepted upstream by FreeBSD, we will have to find his patches=20 and apply them ourselves to the FreeBSD on arm kernel.

We found these slides :

https://events.static.linuxfound.org/sites/events/file= s/slides/Porting%20FreeBSD%20on%20Xen%20on%20ARM%20.pdf

Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what = we want to find.

It looks like when that slide presentation was written, there were=20 some limitations on FreeBSD Xen guests. For example, for our debian=20 bookworm guest, I am using vcpus =3D '2' to match the number of rea= l cpus=20 on our Chromebook, but slide 13 mentions support for only 1 VCPU with a=20 FreeBSD guest, so I will need to change that vcpus =3D '1' in the F= reeBSD=20 guest config unless support for 2 or more vcpus was added later, which=20 is possible because that slide presentation is 9 years old.

Here is where I would expect to find the XENHVM FreeBSD on arm kernel co= nfig file:

https://cgit.freebsd.org/src/tree/sys/arm/= conf

But it is not there unless I am not understanding something=20 correctly. For now, unfortunately conclude that the support for Xen on=20 arm that Julien Grall mentioned in that slide presentation 9 years ago=20 was never added to the official FreeBSD source code. I am searching the=20 web now to see if the patches that Julien Grall wrote are still posted=20 somewhere online. If we cannot find them, we can ask here and on the=20 xen-users mailing list. Julien regularly reads that list and responds to question about Xen on arm, so I think he will tell us how to find the=20 patches if we cannot find them online.

According to this page from the FreeBSD wiki:

https://wiki.freebsd.org/Xen

I think FreeBSD only supports Xen on x86, not arm. So this is going=20 to be a bit of a challenge to get a Xen FreeBSD guest on arm working. We know Julien Grall has some patches that made it work in the past !

I found a slightly newer slide presentation by Julien here:

https://www.slide= share.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd

It is about the same, but it mentions the GENERIC FreeBSD kernel=20 supports Xen on arm64, but still says we need the XENHVM FreeBSD config=20 for Xen on arm 32 bit, which I haven't found online yet.

Please,take a look at this output of the linux kernel that can boot on X= en, and the FreeBSD kernel that cannot :


% file zImage-6.1.59-stb-xen-cbe+
zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (little=
-endian)

% file FREEBSD-XENVIRT         =20
FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dy=
namically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), not=
 stripped


The FreeBSD kernel that won't boot is in ELF format but t= he Linux kernel that does boot is in zImage format.

I spent time reading the docs on xenbits.xenproject.org, and=20 according to those docs Xen on arm only knows how to boot a kernel in=20 the zImage format, so the FreeBSD kernel is in a format that modern Xen=20 incorrectly detects as an x86 kernel.

I also watched Julien Grall's 30 minute video presentation of his wo= rk to boot FreeBSD/arm on Xen at FOSDEM 2014 here :

https://archive.fosdem.or= g/2014/schedule/event/freebsd_xen_arm/

In that video, and in other places, Julien mentions that the boot ABI for FreeBSD/arm on Xen was not yet developed and he was getting=20 occasional crashes and needed to investigate the problem. He mentioned=20 the zImage ABI that Linux uses, but pointed out FreeBSD does not use=20 that format, and back then it was an open question which format to use=20 to boot FreeBSD/arm on Xen. Unfortunately, nine years later, the only=20 supported format is still the zImage format that Linux uses.

It looks like Julien's work back then was using an ELF binary to boo= t FreeBSD/arm on Xen instead of the supported zImage format that Linux=20 uses and the modern Xen toolstack exits with an error when trying to=20 boot the FreeBSD ELF formatted binary that Julien's patch creates. So= =20 the best solution would be to try to port the rules to build a FreeBSD=20 kernel in the zImage format instead of the ELF format. I have been=20 studying the Makefiles in Linux to see how Linux builds the Linux arm=20 kernel in the zImage format, but it is not trivial to understand.

=
--
M= ario.
--00000000000057995f060af88d0b-- From nobody Sat Nov 25 13:45:24 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SctQj6YxWz52JpB for ; Sat, 25 Nov 2023 13:45:33 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from mx.catwhisker.org (mx.catwhisker.org [107.204.234.170]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SctQh5Zt6z3Lkq for ; Sat, 25 Nov 2023 13:45:32 +0000 (UTC) (envelope-from david@catwhisker.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of david@catwhisker.org designates 107.204.234.170 as permitted sender) smtp.mailfrom=david@catwhisker.org; dmarc=none Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.17.1/8.15.2) with ESMTP id 3APDjOOw087878 for ; Sat, 25 Nov 2023 13:45:24 GMT (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.17.1/8.17.1/Submit) id 3APDjOpm087877 for current@freebsd.org; Sat, 25 Nov 2023 05:45:24 -0800 (PST) (envelope-from david) Date: Sat, 25 Nov 2023 05:45:24 -0800 From: David Wolfskill To: current@freebsd.org Subject: After update to main-n266611-49a83b94395a: panic: bus_generic_rman_activate_resource Message-ID: Reply-To: current@freebsd.org Mail-Followup-To: current@freebsd.org List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="maT7EITLWhiES7W7" Content-Disposition: inline X-Spamd-Result: default: False [-0.36 / 15.00]; REPLYTO_EQ_TO_ADDR(5.00)[]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; NEURAL_HAM_SHORT(-0.97)[-0.967]; R_SPF_ALLOW(-0.20)[+ip4:107.204.234.170]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; MLMMJ_DEST(0.00)[current@freebsd.org]; ASN(0.00)[asn:7018, ipnet:107.192.0.0/12, country:US]; DMARC_NA(0.00)[catwhisker.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; HAS_REPLYTO(0.00)[current@freebsd.org]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FREEFALL_USER(0.00)[david]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[current@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4SctQh5Zt6z3Lkq X-Spamd-Bar: / --maT7EITLWhiES7W7 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable After I saw this with both my headless "build machine" and a laptop (and verified that there were no more recent commits to main), I figured it might be worth reporting. This was after a source update from: FreeBSD 15.0-CURRENT #473 main-n266588-2a35f3cdf63d-dirty: Fri Nov 24 13:11= :48 UTC 2023 root@freebeast.catwhisker.org:/common/S4/obj/usr/src/amd64= =2Eamd64/sys/GENERIC amd64 1500004 1500004 ("-dirty" because I had hand-applied 50335b1ae4e4: main-n266589-50335b1ae4e4 commit 50335b1ae4e48712f831e85ddfa7b00da0af382c Author: Emmanuel Vadot Date: Fri Nov 24 11:30:09 2023 +0100 sys/mutex.h: Reorder includes Fixes: 2a35f3cdf63d ("sys/mutex.h: Include sys/lock.h instead of sys/_= lock.h") after seeing a build failure yesterday after updating to main-n266588-2a35f3cdf63d.) Here's a backtrace: =2E.. Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 panic: bus_generic_rman_activate_resource: rman 0xffffffff81b0b0e8 doesn't = match for resource 0xfffff801092b9280 cpuid =3D 20 time =3D 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffffff82eaf= 8b0 vpanic() at vpanic+0x132/frame 0xffffffff82eaf9e0 panic() at panic+0x43/frame 0xffffffff82eafa40 bus_generic_rman_activate_resource() at bus_generic_rman_activate_resource+= 0x146/frame 0xffffffff82eafaa0 acpi_alloc_sysres() at acpi_alloc_sysres+0x83/frame 0xffffffff82eafae0 acpi_alloc_resource() at acpi_alloc_resource+0x108/frame 0xffffffff82eafba0 bus_alloc_resource() at bus_alloc_resource+0x9b/frame 0xffffffff82eafc00 acpi_timer_probe() at acpi_timer_probe+0xaa/frame 0xffffffff82eafc80 device_probe_child() at device_probe_child+0x16f/frame 0xffffffff82eafcd0 device_probe() at device_probe+0x8a/frame 0xffffffff82eafcf0 device_probe_and_attach() at device_probe_and_attach+0x32/frame 0xffffffff8= 2eafd20 bus_generic_attach() at bus_generic_attach+0x18/frame 0xffffffff82eafd40 acpi_probe_children() at acpi_probe_children+0x226/frame 0xffffffff82eafda0 acpi_attach() at acpi_attach+0x97b/frame 0xffffffff82eafe30 device_attach() at device_attach+0x3bc/frame 0xffffffff82eafe70 device_probe_and_attach() at device_probe_and_attach+0x70/frame 0xffffffff8= 2eafea0 bus_generic_attach() at bus_generic_attach+0x18/frame 0xffffffff82eafec0 device_attach() at device_attach+0x3bc/frame 0xffffffff82eaff00 device_probe_and_attach() at device_probe_and_attach+0x70/frame 0xffffffff8= 2eaff30 bus_generic_new_pass() at bus_generic_new_pass+0xed/frame 0xffffffff82eaff60 bus_set_pass() at bus_set_pass+0x36/frame 0xffffffff82eaff90 configure() at configure+0x9/frame 0xffffffff82eaffa0 mi_startup() at mi_startup+0x1c8/frame 0xffffffff82eafff0 KDB: enter: panic [ thread pid 0 tid 100000 ] Stopped at kdb_enter+0x32: movq $0,0xe3cf53(%rip) db> I can leave this machine in this state for a few hours, so if there's any diagnostic information to be gained, I'm happy to do that, but I'll need clues as to what to do. If I can get a dump, I'm happy to make it available (but I'll hold off on trying that for now, as I expect the attempt could disturb evidence). Information about the machine(s) & update history may be found at https://www.catwhisker.org/~david/FreeBSD/history/ in case that's of use. (This includes a copy of /var/run/dmesg.boot from the last successful verbose boot, which was from yesterday.) Peace, david --=20 David H. Wolfskill david@catwhisker.org The notion that anyone would perceive a need to "make neo-Nazis look bad" is about as absurd as perceiving a need to "hydrate" water. See https://www.catwhisker.org/~david/publickey.gpg for my public key. --maT7EITLWhiES7W7 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iNUEARYKAH0WIQSTLzOSbomIK53fjFliipiWhXYx5QUCZWH6dF8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0OTMy RjMzOTI2RTg5ODgyQjlEREY4QzU5NjI4QTk4OTY4NTc2MzFFNQAKCRBiipiWhXYx 5XurAQC/d01s7yiafNbff0ZZCI+NJAnD5PUEra6kQIhYQmMUFwEA2dWb2TnYttpg H83D5pX8dbDevj0WOHjGsm1/7BJmAQw= =Umis -----END PGP SIGNATURE----- --maT7EITLWhiES7W7-- From nobody Sat Nov 25 17:02:43 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ScypR69y9z51jwM for ; Sat, 25 Nov 2023 17:02:55 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ScypR1y6Fz3FvY for ; Sat, 25 Nov 2023 17:02:55 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-a02c48a0420so402965066b.2 for ; Sat, 25 Nov 2023 09:02:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1700931773; x=1701536573; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=lJcyYwjFsGDCNUEF2SrPc1fBl800LPaPtK2E4VfkqSk=; b=Em3noFN9pbttYLD3ebSyUU2FmX67+VwjZcNMCmbA+67sOfb4IkdZQINaPY9jAaLvvG CnME70BylLfHioC2TETwbQuCqtJKzQQc5x/W7EuZ1NXwjclKy/dt4Vg0Y5L8DJAj3e2w xeiCNWSOdR0qwRYnz54Z6rEuUNorj/QMXmwNudqsAV5NLwXdz8G0SipvjKTRKONgCAIo tu4CFB4dXTRogK5vNuQBKjA5ZUXO7nS2aBDlvna6q3TmPIS4Tnm5OVZuKNr8UI3mi6Cf SsCuXfcdqL1MgzEcxnGsIDWHDCw/IFE9t8EV7rG1GQS8WXsTBoekDFC7pc6OEzDRq9T3 qwBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700931773; x=1701536573; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=lJcyYwjFsGDCNUEF2SrPc1fBl800LPaPtK2E4VfkqSk=; b=RxT6Pt//u4PzbkAfM2AH5JPTBqj4GTyisUK1zyNFbrl8n9AkzRmPdHj87mZEYQl0JS krvkQJUJasTS8tps6tRoWank8Gla57wNj/IbI7X7W21Nl3q6YUVR/6V+rRxBQS/0H73a kwPULcN1O5J3Jou960NbU8WQt+vcaqBvjFgTRXkoZEdtIx59/JkjheZxvMQDKsaENpw4 pWVRYS7idV3w4ltsX9+t6EN3Uf6PMDiMhQAfn8RbAF0kfM42iky8gKFMqJbwU5l6i7hE mQ+XjaghTWk0qaS1kkqJMaWW5pX6ucY44K5EVEdN4hw3Y4Go8r4ewSvsLbEIbqSHnZ6W T3WQ== X-Gm-Message-State: AOJu0Yz6UX0t2ZfCCrcwZTQC10xnEziQhuGwfJCmlWcJt0DSY0X8K6hc RBvcxF1L8NJorS2msZlOcBF0ZYRMeNtnCCQF4FpO3g== X-Google-Smtp-Source: AGHT+IGt9WzvZLY0tUhXeNt9BnWnm2hSaeduHPjx5ve9C1ZRnPuOtAra26+KL+N3o0hRlFoli27UFdGuH6gX0G+ZPys= X-Received: by 2002:a17:906:693:b0:9fc:9b28:7ff7 with SMTP id u19-20020a170906069300b009fc9b287ff7mr5534824ejb.60.1700931773247; Sat, 25 Nov 2023 09:02:53 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Sat, 25 Nov 2023 10:02:43 -0700 Message-ID: Subject: Re: Should we boot the FreeBSD kernel in ELF format or in zImage format ? How? To: Mario Marietto Cc: freebsd-hackers , freebsd-arm@freebsd.org, FreeBSD Current , FreeBSD Mailing List , freebsd-xen@freebsd.org, royger@freebsd.org Content-Type: multipart/alternative; boundary="00000000000021c269060afd0a2c" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4ScypR1y6Fz3FvY --00000000000021c269060afd0a2c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Nov 25, 2023 at 4:41=E2=80=AFAM Mario Marietto wrote: > Hello to everyone. > > we have just virtualized Debian 12 on our arm (32 bit) Chromebook. As hos= t > / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It works > great. But our goal is different. We want to virtualize FreeBSD as domU. > Can we have a working Xen PV network driver for a FreeBSD arm guest ?. I > found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I > would like to know if Julien's work was accepted upstream by FreeBSD, in > which case FreeBSD as a Xen guest on arm should work if we enable the Xen > PV drivers in the FreeBSD on arm kernel. If Julien's work was not accepte= d > upstream by FreeBSD, we will have to find his patches and apply them > ourselves to the FreeBSD on arm kernel. > > We found these slides : > > > https://events.static.linuxfound.org/sites/events/files/slides/Porting%20= FreeBSD%20on%20Xen%20on%20ARM%20.pdf > > Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what w= e > want to find. > > It looks like when that slide presentation was written, there were some > limitations on FreeBSD Xen guests. For example, for our debian bookworm > guest, I am using vcpus =3D '2' to match the number of real cpus on our > Chromebook, but slide 13 mentions support for only 1 VCPU with a FreeBSD > guest, so I will need to change that vcpus =3D '1' in the FreeBSD guest > config unless support for 2 or more vcpus was added later, which is > possible because that slide presentation is 9 years old. > > Here is where I would expect to find the XENHVM FreeBSD on arm kernel > config file: > > https://cgit.freebsd.org/src/tree/sys/arm/conf > > But it is not there unless I am not understanding something correctly. Fo= r > now, unfortunately conclude that the support for Xen on arm that Julien > Grall mentioned in that slide presentation 9 years ago was never added to > the official FreeBSD source code. I am searching the web now to see if th= e > patches that Julien Grall wrote are still posted somewhere online. If we > cannot find them, we can ask here and on the xen-users mailing list. Juli= en > regularly reads that list and responds to question about Xen on arm, so I > think he will tell us how to find the patches if we cannot find them onli= ne. > > According to this page from the FreeBSD wiki: > > https://wiki.freebsd.org/Xen > > I think FreeBSD only supports Xen on x86, not arm. So this is going to be > a bit of a challenge to get a Xen FreeBSD guest on arm working. We know > Julien Grall has some patches that made it work in the past ! > > I found a slightly newer slide presentation by Julien here: > > https://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd > > It is about the same, but it mentions the GENERIC FreeBSD kernel supports > Xen on arm64, but still says we need the XENHVM FreeBSD config for Xen on > arm 32 bit, which I haven't found online yet. > > Please,take a look at this output of the linux kernel that can boot on > Xen, and the FreeBSD kernel that cannot : > > > % file zImage-6.1.59-stb-xen-cbe+ > zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (litt= le-endian) > > % file FREEBSD-XENVIRT > FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), = dynamically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), n= ot stripped > > > The FreeBSD kernel that won't boot is in ELF format but the Linux kernel > that does boot is in zImage format. > > I spent time reading the docs on xenbits.xenproject.org, and according to > those docs Xen on arm only knows how to boot a kernel in the zImage forma= t, > so the FreeBSD kernel is in a format that modern Xen incorrectly detects = as > an x86 kernel. > > I also watched Julien Grall's 30 minute video presentation of his work to > boot FreeBSD/arm on Xen at FOSDEM 2014 here : > > https://archive.fosdem.org/2014/schedule/event/freebsd_xen_arm/ > > In that video, and in other places, Julien mentions that the boot ABI for > FreeBSD/arm on Xen was not yet developed and he was getting occasional > crashes and needed to investigate the problem. He mentioned the zImage AB= I > that Linux uses, but pointed out FreeBSD does not use that format, and ba= ck > then it was an open question which format to use to boot FreeBSD/arm on > Xen. Unfortunately, nine years later, the only supported format is still > the zImage format that Linux uses. > > It looks like Julien's work back then was using an ELF binary to boot > FreeBSD/arm on Xen instead of the supported zImage format that Linux uses > and the modern Xen toolstack exits with an error when trying to boot the > FreeBSD ELF formatted binary that Julien's patch creates. So the best > solution would be to try to port the rules to build a FreeBSD kernel in t= he > zImage format instead of the ELF format. I have been studying the Makefil= es > in Linux to see how Linux builds the Linux arm kernel in the zImage forma= t, > but it is not trivial to understand > Look at kernel.bin in FreeBSD's kernel. It's enabled -DWITH_KERNEL_BIN. It should be easy to adapt the target to build that. I've done similar things with u-boot formats in the past, but that was 4 employers and 20 years ago now. This path is not well trod. I do know that arm64 virtualization with bhyve is hitting the tree. I'm not sure how easy/hard this will be to modernize. I'm interested to see how your explorations go. Warner --00000000000021c269060afd0a2c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sat, Nov 25, 2023 at 4:41=E2=80=AF= AM Mario Marietto <marietto200= 8@gmail.com> wrote:
Hel= lo to everyone.

we have just virtualized Debian 12 on our arm (= 32 bit) Chromebook. As host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It=20 works great. But our goal is different. We want to virtualize FreeBSD as domU. Can we have a working Xen PV network driver for a FreeBSD arm=20 guest ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I would like to know if Julien's work was accepted upstream by= =20 FreeBSD, in which case FreeBSD as a Xen guest on arm should work if we=20 enable the Xen PV drivers in the FreeBSD on arm kernel. If Julien's wor= k was not accepted upstream by FreeBSD, we will have to find his patches=20 and apply them ourselves to the FreeBSD on arm kernel.

We found these slides :

https://events.static.linuxfound.org/sites/events/file= s/slides/Porting%20FreeBSD%20on%20Xen%20on%20ARM%20.pdf

Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what = we want to find.

It looks like when that slide presentation was written, there were=20 some limitations on FreeBSD Xen guests. For example, for our debian=20 bookworm guest, I am using vcpus =3D '2' to match the number of rea= l cpus=20 on our Chromebook, but slide 13 mentions support for only 1 VCPU with a=20 FreeBSD guest, so I will need to change that vcpus =3D '1' in the F= reeBSD=20 guest config unless support for 2 or more vcpus was added later, which=20 is possible because that slide presentation is 9 years old.

Here is where I would expect to find the XENHVM FreeBSD on arm kernel co= nfig file:

https://cgit.freebsd.org/src/tree/sys/arm/= conf

But it is not there unless I am not understanding something=20 correctly. For now, unfortunately conclude that the support for Xen on=20 arm that Julien Grall mentioned in that slide presentation 9 years ago=20 was never added to the official FreeBSD source code. I am searching the=20 web now to see if the patches that Julien Grall wrote are still posted=20 somewhere online. If we cannot find them, we can ask here and on the=20 xen-users mailing list. Julien regularly reads that list and responds to question about Xen on arm, so I think he will tell us how to find the=20 patches if we cannot find them online.

According to this page from the FreeBSD wiki:

https://wiki.freebsd.org/Xen

I think FreeBSD only supports Xen on x86, not arm. So this is going=20 to be a bit of a challenge to get a Xen FreeBSD guest on arm working. We know Julien Grall has some patches that made it work in the past !

I found a slightly newer slide presentation by Julien here:

https://www.slide= share.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd

It is about the same, but it mentions the GENERIC FreeBSD kernel=20 supports Xen on arm64, but still says we need the XENHVM FreeBSD config=20 for Xen on arm 32 bit, which I haven't found online yet.

Please,take a look at this output of the linux kernel that can boot on X= en, and the FreeBSD kernel that cannot :


% file zImage-6.1.59-stb-xen-cbe+
zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (little=
-endian)

% file FREEBSD-XENVIRT         =20
FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dy=
namically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), not=
 stripped


The FreeBSD kernel that won't boot is in ELF format but t= he Linux kernel that does boot is in zImage format.

I spent time reading the docs on xenbits.xenproject.org, and=20 according to those docs Xen on arm only knows how to boot a kernel in=20 the zImage format, so the FreeBSD kernel is in a format that modern Xen=20 incorrectly detects as an x86 kernel.

I also watched Julien Grall's 30 minute video presentation of his wo= rk to boot FreeBSD/arm on Xen at FOSDEM 2014 here :

https://archive.fosdem.or= g/2014/schedule/event/freebsd_xen_arm/

In that video, and in other places, Julien mentions that the boot ABI for FreeBSD/arm on Xen was not yet developed and he was getting=20 occasional crashes and needed to investigate the problem. He mentioned=20 the zImage ABI that Linux uses, but pointed out FreeBSD does not use=20 that format, and back then it was an open question which format to use=20 to boot FreeBSD/arm on Xen. Unfortunately, nine years later, the only=20 supported format is still the zImage format that Linux uses.

It looks like Julien's work back then was using an ELF binary to boo= t FreeBSD/arm on Xen instead of the supported zImage format that Linux=20 uses and the modern Xen toolstack exits with an error when trying to=20 boot the FreeBSD ELF formatted binary that Julien's patch creates. So= =20 the best solution would be to try to port the rules to build a FreeBSD=20 kernel in the zImage format instead of the ELF format. I have been=20 studying the Makefiles in Linux to see how Linux builds the Linux arm=20 kernel in the zImage format, but it is not trivial to understand

<= /div>

Look at kernel.bin = in FreeBSD's kernel. It's enabled -DWITH_KERNEL_BIN. It should be e= asy to adapt the target to build that. I've done similar things with u-= boot formats in the past, but that was 4 employers and 20 years ago now.

This path is not well trod. I do know that arm64= virtualization with bhyve is hitting the tree. I'm not sure how easy/h= ard this will be to modernize. I'm interested to see how your explorati= ons go.

Warner
--00000000000021c269060afd0a2c-- From nobody Sat Nov 25 19:36:33 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sd2F80y3xz51l7y for ; Sat, 25 Nov 2023 19:37:48 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Received: from www121.sakura.ne.jp (www121.sakura.ne.jp [153.125.133.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sd2F73QSRz4RL7 for ; Sat, 25 Nov 2023 19:37:46 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Authentication-Results: mx1.freebsd.org; none Received: from kalamity.joker.local (123-1-22-158.area1b.commufa.jp [123.1.22.158]) (authenticated bits=0) by www121.sakura.ne.jp (8.16.1/8.16.1/[SAKURA-WEB]/20201212) with ESMTPA id 3APJaY9D027242; Sun, 26 Nov 2023 04:36:34 +0900 (JST) (envelope-from junchoon@dec.sakura.ne.jp) Date: Sun, 26 Nov 2023 04:36:33 +0900 From: Tomoaki AOKI To: current@freebsd.org Cc: David Wolfskill Subject: Re: After update to main-n266611-49a83b94395a: panic: bus_generic_rman_activate_resource Message-Id: <20231126043633.15c60223c7a518d103b6dbef@dec.sakura.ne.jp> In-Reply-To: References: Organization: Junchoon corps X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd14.0) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:7684, ipnet:153.125.128.0/18, country:JP] X-Rspamd-Queue-Id: 4Sd2F73QSRz4RL7 Hi. Not tested, but does upgrading to commit ed88eef140a1 [1] and later help? (I'm still at commit 5d4f897f88ed on main.) [1] https://cgit.freebsd.org/src/commit/?id=ed88eef140a1c3d57d546f409c216806dd3da809 Regards. On Sat, 25 Nov 2023 05:45:24 -0800 David Wolfskill wrote: > After I saw this with both my headless "build machine" and a laptop (and > verified that there were no more recent commits to main), I figured it > might be worth reporting. > > This was after a source update from: > FreeBSD 15.0-CURRENT #473 main-n266588-2a35f3cdf63d-dirty: Fri Nov 24 13:11:48 UTC 2023 root@freebeast.catwhisker.org:/common/S4/obj/usr/src/amd64.amd64/sys/GENERIC amd64 1500004 1500004 > > ("-dirty" because I had hand-applied 50335b1ae4e4: > > main-n266589-50335b1ae4e4 > commit 50335b1ae4e48712f831e85ddfa7b00da0af382c > Author: Emmanuel Vadot > Date: Fri Nov 24 11:30:09 2023 +0100 > > sys/mutex.h: Reorder includes > > Fixes: 2a35f3cdf63d ("sys/mutex.h: Include sys/lock.h instead of sys/_lock.h") > > after seeing a build failure yesterday after updating to > main-n266588-2a35f3cdf63d.) > > Here's a backtrace: > > ... > Event timer "HPET6" frequency 14318180 Hz quality 340 > Event timer "HPET7" frequency 14318180 Hz quality 340 > panic: bus_generic_rman_activate_resource: rman 0xffffffff81b0b0e8 doesn't match for resource 0xfffff801092b9280 > cpuid = 20 > time = 1 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffffff82eaf8b0 > vpanic() at vpanic+0x132/frame 0xffffffff82eaf9e0 > panic() at panic+0x43/frame 0xffffffff82eafa40 > bus_generic_rman_activate_resource() at bus_generic_rman_activate_resource+0x146/frame 0xffffffff82eafaa0 > acpi_alloc_sysres() at acpi_alloc_sysres+0x83/frame 0xffffffff82eafae0 > acpi_alloc_resource() at acpi_alloc_resource+0x108/frame 0xffffffff82eafba0 > bus_alloc_resource() at bus_alloc_resource+0x9b/frame 0xffffffff82eafc00 > acpi_timer_probe() at acpi_timer_probe+0xaa/frame 0xffffffff82eafc80 > device_probe_child() at device_probe_child+0x16f/frame 0xffffffff82eafcd0 > device_probe() at device_probe+0x8a/frame 0xffffffff82eafcf0 > device_probe_and_attach() at device_probe_and_attach+0x32/frame 0xffffffff82eafd20 > bus_generic_attach() at bus_generic_attach+0x18/frame 0xffffffff82eafd40 > acpi_probe_children() at acpi_probe_children+0x226/frame 0xffffffff82eafda0 > acpi_attach() at acpi_attach+0x97b/frame 0xffffffff82eafe30 > device_attach() at device_attach+0x3bc/frame 0xffffffff82eafe70 > device_probe_and_attach() at device_probe_and_attach+0x70/frame 0xffffffff82eafea0 > bus_generic_attach() at bus_generic_attach+0x18/frame 0xffffffff82eafec0 > device_attach() at device_attach+0x3bc/frame 0xffffffff82eaff00 > device_probe_and_attach() at device_probe_and_attach+0x70/frame 0xffffffff82eaff30 > bus_generic_new_pass() at bus_generic_new_pass+0xed/frame 0xffffffff82eaff60 > bus_set_pass() at bus_set_pass+0x36/frame 0xffffffff82eaff90 > configure() at configure+0x9/frame 0xffffffff82eaffa0 > mi_startup() at mi_startup+0x1c8/frame 0xffffffff82eafff0 > KDB: enter: panic > [ thread pid 0 tid 100000 ] > Stopped at kdb_enter+0x32: movq $0,0xe3cf53(%rip) > db> > > I can leave this machine in this state for a few hours, so if there's > any diagnostic information to be gained, I'm happy to do that, but I'll > need clues as to what to do. > > If I can get a dump, I'm happy to make it available (but I'll hold off > on trying that for now, as I expect the attempt could disturb evidence). > > Information about the machine(s) & update history may be found at > https://www.catwhisker.org/~david/FreeBSD/history/ in case that's > of use. (This includes a copy of /var/run/dmesg.boot from the last > successful verbose boot, which was from yesterday.) > > Peace, > david > -- > David H. Wolfskill david@catwhisker.org > The notion that anyone would perceive a need to "make neo-Nazis > look bad" is about as absurd as perceiving a need to "hydrate" water. > > See https://www.catwhisker.org/~david/publickey.gpg for my public key. -- Tomoaki AOKI From nobody Sat Nov 25 19:41:20 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sd2KL2hZgz51mm0 for ; Sat, 25 Nov 2023 19:41:26 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from mx.catwhisker.org (mx.catwhisker.org [107.204.234.170]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sd2KK75Mcz4TBH for ; Sat, 25 Nov 2023 19:41:25 +0000 (UTC) (envelope-from david@catwhisker.org) Authentication-Results: mx1.freebsd.org; none Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.17.1/8.15.2) with ESMTP id 3APJfKNj091676; Sat, 25 Nov 2023 19:41:20 GMT (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.17.1/8.17.1/Submit) id 3APJfKMa091675; Sat, 25 Nov 2023 11:41:20 -0800 (PST) (envelope-from david) Date: Sat, 25 Nov 2023 11:41:20 -0800 From: David Wolfskill To: Tomoaki AOKI Cc: current@freebsd.org Subject: Re: After update to main-n266611-49a83b94395a: panic: bus_generic_rman_activate_resource Message-ID: Mail-Followup-To: David Wolfskill , Tomoaki AOKI , current@freebsd.org References: <20231126043633.15c60223c7a518d103b6dbef@dec.sakura.ne.jp> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="lxb1//8w9DJ6paXq" Content-Disposition: inline In-Reply-To: <20231126043633.15c60223c7a518d103b6dbef@dec.sakura.ne.jp> X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:7018, ipnet:107.192.0.0/12, country:US] X-Rspamd-Queue-Id: 4Sd2KK75Mcz4TBH --lxb1//8w9DJ6paXq Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Nov 26, 2023 at 04:36:33AM +0900, Tomoaki AOKI wrote: > Hi. >=20 > Not tested, but does upgrading to commit ed88eef140a1 [1] and later > help? (I'm still at commit 5d4f897f88ed on main.) >=20 > [1] > https://cgit.freebsd.org/src/commit/?id=3Ded88eef140a1c3d57d546f409c21680= 6dd3da809 >=20 > Regards. > .... Thanks! That does look as if it addresses the problem, so I will try it (& reoprt to the list). Peace, david --=20 David H. Wolfskill david@catwhisker.org The notion that anyone would perceive a need to "make neo-Nazis look bad" is about as absurd as perceiving a need to "hydrate" water. See https://www.catwhisker.org/~david/publickey.gpg for my public key. --lxb1//8w9DJ6paXq Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iNUEARYKAH0WIQSTLzOSbomIK53fjFliipiWhXYx5QUCZWJN4F8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0OTMy RjMzOTI2RTg5ODgyQjlEREY4QzU5NjI4QTk4OTY4NTc2MzFFNQAKCRBiipiWhXYx 5Sw+AQD1oj15BLWZ4qI2BfkX8uSETawKlCJmOiYz4IZnQ9pGogD+IYqMRBSIMg6q /ZUnzwQbj6UWuTPiy+R3DAMshlYC2Qc= =BwKT -----END PGP SIGNATURE----- --lxb1//8w9DJ6paXq-- From nobody Sat Nov 25 20:16:49 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sd36G1WCfz5257c for ; Sat, 25 Nov 2023 20:16:54 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from mx.catwhisker.org (mx.catwhisker.org [107.204.234.170]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sd36F52chz4d3l for ; Sat, 25 Nov 2023 20:16:53 +0000 (UTC) (envelope-from david@catwhisker.org) Authentication-Results: mx1.freebsd.org; none Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.17.1/8.15.2) with ESMTP id 3APKGnbK091960; Sat, 25 Nov 2023 20:16:49 GMT (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.17.1/8.17.1/Submit) id 3APKGnEn091959; Sat, 25 Nov 2023 12:16:49 -0800 (PST) (envelope-from david) Date: Sat, 25 Nov 2023 12:16:49 -0800 From: David Wolfskill To: Tomoaki AOKI Cc: current@freebsd.org Subject: Re: [resolved] After update to main-n266611-49a83b94395a: panic: bus_generic_rman_activate_resource Message-ID: Reply-To: current@freebsd.org Mail-Followup-To: current@freebsd.org, Tomoaki AOKI References: <20231126043633.15c60223c7a518d103b6dbef@dec.sakura.ne.jp> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="UarTHo0UMT5Dblam" Content-Disposition: inline In-Reply-To: <20231126043633.15c60223c7a518d103b6dbef@dec.sakura.ne.jp> X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:7018, ipnet:107.192.0.0/12, country:US] X-Rspamd-Queue-Id: 4Sd36F52chz4d3l --UarTHo0UMT5Dblam Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Nov 26, 2023 at 04:36:33AM +0900, Tomoaki AOKI wrote: > Hi. >=20 > Not tested, but does upgrading to commit ed88eef140a1 [1] and later > help? (I'm still at commit 5d4f897f88ed on main.) >=20 > [1] > https://cgit.freebsd.org/src/commit/?id=3Ded88eef140a1c3d57d546f409c21680= 6dd3da809 >=20 > Regards. > .... Yes; thank you (and jhb@): after hand-applying the ed88eef140a1 commit & rebuilding, the laptop rebooted OK: g1-48(15.0-C)[1] uname -aUK FreeBSD g1-48.catwhisker.org 15.0-CURRENT FreeBSD 15.0-CURRENT #530 main-n2= 66611-49a83b94395a-dirty: Sat Nov 25 20:04:25 UTC 2023 root@g1-48.catwh= isker.org:/common/S4/obj/usr/src/amd64.amd64/sys/CANARY amd64 1500004 15000= 04 Peace, david --=20 David H. Wolfskill david@catwhisker.org The notion that anyone would perceive a need to "make neo-Nazis look bad" is about as absurd as perceiving a need to "hydrate" water. See https://www.catwhisker.org/~david/publickey.gpg for my public key. --UarTHo0UMT5Dblam Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iNUEARYKAH0WIQSTLzOSbomIK53fjFliipiWhXYx5QUCZWJWMV8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0OTMy RjMzOTI2RTg5ODgyQjlEREY4QzU5NjI4QTk4OTY4NTc2MzFFNQAKCRBiipiWhXYx 5b90AQC0LHQmUmN97eQPA6M/4UPeCExdkW0ndzHltm/KJkMPWgEA8f+yJNM02QZL jSKvfjdW/k2BpINDdRCDvnsLOyFkYQw= =eFR6 -----END PGP SIGNATURE----- --UarTHo0UMT5Dblam--