Date: Sat, 18 Nov 2023 19:19:04 -0800 From: Rick Macklem <rick.macklem@gmail.com> To: Mike Karels <mike@karels.net> Cc: FreeBSD CURRENT <freebsd-current@freebsd.org>, Alexander Motin <mav@freebsd.org>, Martin Matuska <mm@freebsd.org>, Garrett Wollman <wollman@bimajority.org> Subject: Re: NFS exports of ZFS snapshots broken Message-ID: <CAM5tNy6jqH3nRrtAhTaOpAR1RCXKxygSKHtDMKeQStDnkC1skg@mail.gmail.com> In-Reply-To: <AD48F344-DDD1-476F-B353-B0F5C05C9808@karels.net> References: <25943.60056.880614.452966@hergotha.csail.mit.edu> <DDC3BBBB-561A-4779-B076-83EC054859C2@karels.net> <CAM5tNy6gcThf0nrcs4iC8r6J8cMFOHdUNcUjfJVDSpTW_tqXKQ@mail.gmail.com> <ACAFC3CD-0024-4076-8C69-CAA3B1550B41@karels.net> <AB4690A9-A8F4-490B-82AA-358CD8118DF0@karels.net> <CAM5tNy5LGWF424i9v_w2u%2B41wtzmW9kn6RZWwo6Pi0Y4wj92UA@mail.gmail.com> <CAM5tNy6K6psrtem%2Bc3ta0pM-7fhy9LYci0DEeR0O-EF%2B-iGo6Q@mail.gmail.com> <91988E23-ED50-4379-AA5F-4B069E08D80F@karels.net> <CAM5tNy68tq24C_BA0xNYM-xUbCzpAve0jY-gsr42ixQLHR5gHg@mail.gmail.com> <AD48F344-DDD1-476F-B353-B0F5C05C9808@karels.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000071e6ce060a78d585 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Nov 18, 2023 at 4:43=E2=80=AFPM Mike Karels <mike@karels.net> wrote= : > > On 18 Nov 2023, at 17:23, Rick Macklem wrote: > > > On Sat, Nov 18, 2023 at 2:27=E2=80=AFPM Mike Karels <mike@karels.net> w= rote: > >> > >> On 18 Nov 2023, at 15:58, Rick Macklem wrote: > >> > >>> On Sat, Nov 18, 2023 at 8:09=E2=80=AFAM Rick Macklem <rick.macklem@gm= ail.com> wrote: > >>>> > >>>> On Fri, Nov 17, 2023 at 8:19=E2=80=AFPM Mike Karels <mike@karels.net= > wrote: > >>>>> > >>>>> On 17 Nov 2023, at 22:14, Mike Karels wrote: > >>>>> > >>>>>> On 17 Nov 2023, at 21:24, Rick Macklem wrote: > >>>>>> > >>>>>>> Most of the changes in stable/13 that are not in releng/13.2 > >>>>>>> are the "make it work in a jail" stuff. Unfortunately, they are > >>>>>>> a large # of changes (mostly trivial edits adding vnet macros), > >>>>>>> but it also includes export check changes. > >>>>>>> > >>>>>>> I have attached a trivial patch that I think disables the export > >>>>>>> checks for jails. If either of you can try it and see if it fixes > >>>>>>> the problem, that would be great. > >>>>>>> (Note that this is only for testing, although it probably does no= t > >>>>>>> matter unless you are running nfsd(8) in vnet jails.) > >>>>>> > >>>>>> Yes, I can see snapshots with the patch. This system is just a te= st > >>>>>> system that doesn't normally run ZFS or NFS, so no problem messing > >>>>>> with permissions. It's a bhyve VM, so I just added a small disk a= nd > >>>>>> enabled ZFS for testing. > >>>>> > >>>>> btw, you might try to get mm@ or maybe mav@ to help out from the ZF= S > >>>>> side. It must be doing something differently inside a snapshot tha= n > >>>>> outside, maybe with file handles or something like that. > >>>> Yes. I've added freebsd-current@ (although Garrett is not on it, he = is > >>>> cc'd) and these guys specifically... > >>>> > >>>> So, here's what appears to be the problem... > >>>> Commit 88175af (in main and stable/13, but not 13.2) added checks fo= r > >>>> nfsd(8) running in jails by filling in mnt_exjail with a reference t= o the cred > >>>> used when the file system is exported. > >>>> When mnt_exjail is found NULL, the current nfsd code assumes that th= ere > >>>> is no access allowed for the mount. > >>>> > >>>> My vague understanding is that when a ZFS snapshot is accessed, it i= s > >>>> "pseudo-mounted" by zfsctl_snapdir_lookup() and I am guessing that > >>>> mnt_exjail is NULL as a result. > >>>> Since I do not know the ZFS code and don't even have an easy way to > >>>> test this (thankfully Mike can test easily), I do not know what to d= o from > >>>> here? > >>>> > >>>> Is there a "struct mount" constructed for this pseudo mount > >>>> (or it actually appears to be the lookup of ".." that fails, so it > >>>> might be the parent of the snapshot subdir?)? > >>>> > >>>> One thought is that I can check to see if the mount pointer is in th= e > >>>> mountlist (I don't think the snapshot's mount is in the mountlist) a= nd > >>>> avoid the jail test for this case. This would assume that snapshots= are > >>>> always within the file system(s) exported via that jail (which inclu= des > >>>> the case of prison0, of course), so that they do not need a separate > >>>> jail check. > >>>> > >>>> If this doesn't work, there will need to be some sort of messing abo= ut > >>>> in ZFS to set mnt_exjail for these. > >>> Ok, so now onto the hard part... > >>> Thanks to Mike and others, I did create a snapshot under .zfs and I c= an > >>> see the problem. It is that mnt_exjail =3D=3D NULL. > >>> Now, is there a way that this "struct mount" can be recognized as "sp= ecial" > >>> for snapshots, so I can avoid the mnt_exjail =3D=3D NULL test? > >>> (I had hoped that "mp->mnt_list.tqe_prev" would be NULL, but that is = not > >>> the case.) > >> > >> Dumb question, is the mount point (mp presumably) different between th= e > >> snapshot and the main file system? > > Not a dump question and the answer is rather interesting... > > It is "sometimes" or "usually" according to my printf(). > > It seems that when you first "cd <snapshot-name"" you get a different m= p > > where mnt_exjail =3D=3D NULL.. Then when you look at directories within= the > > snapshot, you get the mp of the file system that .zfs exists in, which = does > > have mnt_exjail set non-NULL. > > > > There is this snippet of code in zfsctl_snapdir_lookup(): > > /* > > * Fix up the root vnode mounted on .zfs/snapshot/<snapname>. > > * > > * This is where we lie about our v_vfsp in order to > > * make .zfs/snapshot/<snapname> accessible over NFS > > * without requiring manual mounts of <snapname>. > > */ > > ASSERT3P(VTOZ(*vpp)->z_zfsvfs, !=3D, zfsvfs); > > VTOZ(*vpp)->z_zfsvfs->z_parent =3D zfsvfs; > > > > /* Clear the root flag (set via VFS_ROOT) as well. */ > > (*vpp)->v_vflag &=3D ~VV_ROOT; > > which seems to set the mp to that of the parent, but it > > seems this does not happen for the initial lookup of > > the <snapname>? > > > > I'll note that there is code before this in > > zfsctl_snapdir_lookup() for handling cases > > like "." and ".." that return without doing this. > > > > Now, why does this work without the mnt_exjail > > check (as in 13.2)? > > I am not quite sure, but there is this "cheat" in the > > NFS server (it has been there for years, maybe decades): > > /* > > * Allow a Lookup, Getattr, GetFH, Secinfo on an > > * non-exported directory if > > * nfs_rootfhset. Do I need to allow any other Ops? > > * (You can only have a non-exported vpnes if > > * nfs_rootfhset is true. See nfsd_fhtovp()) > > * Allow AUTH_SYS to be used for file systems > > * exported GSS only for certain Ops, to allow > > * clients to do mounts more easily. > > */ > > if (nfsv4_opflag[op].needscfh && vp) { > > if (!NFSVNO_EXPORTED(&vpnes) && > > op !=3D NFSV4OP_LOOKUP && > > op !=3D NFSV4OP_GETATTR && > > op !=3D NFSV4OP_GETFH && > > op !=3D NFSV4OP_ACCESS && > > op !=3D NFSV4OP_READLINK && > > op !=3D NFSV4OP_SECINFO && > > op !=3D NFSV4OP_SECINFONONAME) > > nd->nd_repstat =3D NFSERR_NOFILEHANDLE; > > This allows certain operations to be done on > > non-exported file systems and I think that is enough > > to allow this to work when mnt_exjail is not checked. > > (Note that NFSV4OP_LOOKUPP is not in the list, > > which might explain why it is the one that fails for > > Garrett. I don't think it can be added to this list > > safely, since that would allow a client to move above > > the exported file system into "uncharted territory".) > > > >> Just curious. Also, what is mnt_exjail > >> normally set to for file systems not in a jail? > > mnt_exjail is set to the credentials of the thread/process > > that exported the file system (usually mountd(8)). > > When not in a jail, cr_prison for these credentials > > points to prison0. > > > > Btw, I checked and the "other mp that has mnt_exjail =3D=3D NULL > > is in the mountlist, so the idea of checking "not in mountlist" > > is a dead end. > > > > I am looking for something "unique" about this other mp, > > but haven't found anything yet. > > Alternately, it might be necessary to add code to > > zfsctl_snapdir_lookup() to "cheat and change the mp" > > in more cases, such as "." and ".." lookups? > > It seems to me that if ZFS is creating an additional mount structure, > it should be responsible for setting it up correctly. That could > involve a vfs-level routine to do some of the cloning. In any case, > it seems to me that mnt_exjail should be set properly, e.g. by duping > the one in the original mount structure. Probably ZFS is the only > file system type that would need this added. I've created a patch that I think does this. It seemed to test ok for my ca= se. It's in D42672 on reviews.freebsd.org. I have also attached it here (this diff doesn't use -U999999, so it might b= e easier to apply). Hopefully this fixes the problem. Sorry for the breakage. rick > > Mike > > > rick > > ps: I added all the cc's back in because I want the > > ZFS folk to hopefully chime in. > > > >> > >> Mike > >> > >>> Do I need to search mountlist for it? > >>> > >>> rick > >>> ps: The hack patch attached should fix the problem, but can only be > >>> safely used if mountd/nfsd are not run in any jails. > >>> > >>>> > >>>> I will try and get a test setup going here, which leads me to.. > >>>> how do I create a ZFS snapshot? (I do have a simple ZFS pool running > >>>> on a test machine, but I've never done a snapshot.) > >>>> > >>>> Although this problem is not in 13.2, it will have shipped in 14.0. > >>>> > >>>> Any help with be appreciated, rick > >>>> > >>>>> > >>>>> Mike > >>>>>> > >>>>>>> rick > >>>>>>> > >>>>>>> On Fri, Nov 17, 2023 at 6:14=E2=80=AFPM Mike Karels <mike@karels.= net> wrote: > >>>>>>>> > >>>>>>>> CAUTION: This email originated from outside of the University of= Guelph. Do not click links or open attachments unless you recognize the se= nder and know the content is safe. If in doubt, forward suspicious emails t= o IThelp@uoguelph.ca. > >>>>>>>> > >>>>>>>> > >>>>>>>> Rick, have you been following this thread on freebsd-stable? I = have been able > >>>>>>>> to reproduce this using a 13-stable server from Oct 7 and a 15-c= urrent system > >>>>>>>> that is up to date using NFSv3. I did not reproduce with a 13.2= server. The > >>>>>>>> client was running 13.2. Any ideas? A full bisect seems fairly= painful, but > >>>>>>>> maybe you have an idea of points to try. Fortunately, these are= all test > >>>>>>>> systems that I can reboot at will. > >>>>>>>> > >>>>>>>> Mike > >>>>>>>> > >>>>>>>> Forwarded message: > >>>>>>>> > >>>>>>>>> From: Garrett Wollman <wollman@bimajority.org> > >>>>>>>>> To: Mike Karels <mike@karels.net> > >>>>>>>>> Cc: freebsd-stable@freebsd.org > >>>>>>>>> Subject: Re: NFS exports of ZFS snapshots broken > >>>>>>>>> Date: Fri, 17 Nov 2023 17:35:04 -0500 > >>>>>>>>> > >>>>>>>>> <<On Fri, 17 Nov 2023 15:57:42 -0600, Mike Karels <mike@karels.= net> said: > >>>>>>>>> > >>>>>>>>>> I have not run into this, so I tried it just now. I had no pr= oblem. > >>>>>>>>>> The server is 13.2, fully patched, the client is up-to-date -c= urrent, > >>>>>>>>>> and the mount is v4. > >>>>>>>>> > >>>>>>>>> On my 13.2 client and 13-stable server, I see: > >>>>>>>>> > >>>>>>>>> 25034 ls CALL open(0x237d32f9a000,0x120004<O_RDONLY|O_N= ONBLOCK|O_DIRECTORY|O_CLOEXEC>) > >>>>>>>>> 25034 ls NAMI "/mnt/tools/.zfs/snapshot/weekly-2023-45" > >>>>>>>>> 25034 ls RET open 4 > >>>>>>>>> 25034 ls CALL fcntl(0x4,F_ISUNIONSTACK,0x0) > >>>>>>>>> 25034 ls RET fcntl 0 > >>>>>>>>> 25034 ls CALL getdirentries(0x4,0x237d32faa000,0x1000,0= x237d32fa7028) > >>>>>>>>> 25034 ls RET getdirentries -1 errno 5 Input/output err= or > >>>>>>>>> 25034 ls CALL close(0x4) > >>>>>>>>> 25034 ls RET close 0 > >>>>>>>>> 25034 ls CALL exit(0) > >>>>>>>>> > >>>>>>>>> Certainly a libc bug here that getdirentries(2) returning [EIO] > >>>>>>>>> results in ls(1) returning EXIT_SUCCESS, but the [EIO] error is > >>>>>>>>> consistent across both FreeBSD and Linux clients. > >>>>>>>>> > >>>>>>>>> Looking at this from the RPC side: > >>>>>>>>> > >>>>>>>>> (PUTFH, GETATTR, LOOKUP(snapshotname), GETFH, GETATTR) > >>>>>>>>> [NFS4_OK for all ops] > >>>>>>>>> (PUTFH, GETATTR) > >>>>>>>>> [NFS4_OK, NFS4_OK] > >>>>>>>>> (PUTFH, ACCESS(0x3f), GETATTR) > >>>>>>>>> [NFS4_OK, NFS4_OK, rights =3D 0x03, NFS4_OK] > >>>>>>>>> (PUTFH, GETATTR, LOOKUPP, GETFH, GETATTR) > >>>>>>>>> [NFS4_OK, NFS4_OK, NFS4ERR_NOFILEHANDLE] > >>>>>>>>> > >>>>>>>>> and at this point the [EIO] is returned. > >>>>>>>>> > >>>>>>>>> It seems that clients always do a LOOKUPP before calling READDI= R, and > >>>>>>>>> this is failing when the subject file handle is the snapshot. = The > >>>>>>>>> client is perfectly able to *traverse into* the snapshot: if I = try to > >>>>>>>>> list a subdirectory I know exists in the snapshot, the client i= s able to > >>>>>>>>> LOOKUP(dirname) just fine, but LOOKUPP still fails with > >>>>>>>>> NFS4ERR_NOFILEHANDLE *on the subndirectory*. > >>>>>>>>> > >>>>>>>>> -GAWollman > >>>>>>>> --00000000000071e6ce060a78d585 Content-Type: application/octet-stream; name="zfssnap.patch" Content-Disposition: attachment; filename="zfssnap.patch" Content-Transfer-Encoding: base64 Content-ID: <f_lp4wse5y0> X-Attachment-Id: f_lp4wse5y0 LS0tIHN5cy9jb250cmliL29wZW56ZnMvaW5jbHVkZS9vcy9mcmVlYnNkL3NwbC9zeXMvdmZzLmgu emZzc25hcAkyMDIzLTExLTE4IDE3OjMyOjAwLjQ0OTYxODAwMCAtMDgwMAorKysgc3lzL2NvbnRy aWIvb3Blbnpmcy9pbmNsdWRlL29zL2ZyZWVic2Qvc3BsL3N5cy92ZnMuaAkyMDIzLTExLTE4IDE3 OjMxOjQ0LjUzOTcwNTAwMCAtMDgwMApAQCAtMTAxLDcgKzEwMSw3IEBAIHZvaWQgdmZzX3NldG1u dG9wdCh2ZnNfdCAqdmZzcCwgY29uc3QgY2hhciAqbmFtZSwgY29ucwogdm9pZCB2ZnNfY2xlYXJt bnRvcHQodmZzX3QgKnZmc3AsIGNvbnN0IGNoYXIgKm5hbWUpOwogaW50IHZmc19vcHRpb25pc3Nl dChjb25zdCB2ZnNfdCAqdmZzcCwgY29uc3QgY2hhciAqb3B0LCBjaGFyICoqYXJncCk7CiBpbnQg bW91bnRfc25hcHNob3Qoa3RocmVhZF90ICp0ZCwgdm5vZGVfdCAqKnZwcCwgY29uc3QgY2hhciAq ZnN0eXBlLAotICAgIGNoYXIgKmZzcGF0aCwgY2hhciAqZnNwZWMsIGludCBmc2ZsYWdzKTsKKyAg ICBjaGFyICpmc3BhdGgsIGNoYXIgKmZzcGVjLCBpbnQgZnNmbGFncywgdmZzX3QgKnZmc3ApOwog CiB0eXBlZGVmCXVpbnQ2NF90CXZmc19mZWF0dXJlX3Q7CiAKLS0tIHN5cy9jb250cmliL29wZW56 ZnMvbW9kdWxlL29zL2ZyZWVic2Qvc3BsL3NwbF92ZnMuYy56ZnNzbmFwCTIwMjMtMTEtMTggMTY6 NDQ6MzUuMjkyNjU1MDAwIC0wODAwCisrKyBzeXMvY29udHJpYi9vcGVuemZzL21vZHVsZS9vcy9m cmVlYnNkL3NwbC9zcGxfdmZzLmMJMjAyMy0xMS0xOCAxODozMDowNS40MzgyMTgwMDAgLTA4MDAK QEAgLTEyMCw3ICsxMjAsNyBAQCB2ZnNfb3B0aW9uaXNzZXQoY29uc3QgdmZzX3QgKnZmc3AsIGNv bnN0IGNoYXIgKm9wdCwgY2gKIAogaW50CiBtb3VudF9zbmFwc2hvdChrdGhyZWFkX3QgKnRkLCB2 bm9kZV90ICoqdnBwLCBjb25zdCBjaGFyICpmc3R5cGUsIGNoYXIgKmZzcGF0aCwKLSAgICBjaGFy ICpmc3BlYywgaW50IGZzZmxhZ3MpCisgICAgY2hhciAqZnNwZWMsIGludCBmc2ZsYWdzLCB2ZnNf dCAqcGFyZW50X3Zmc3ApCiB7CiAJc3RydWN0IHZmc2NvbmYgKnZmc3A7CiAJc3RydWN0IG1vdW50 ICptcDsKQEAgLTIxOSw2ICsyMTksMTIgQEAgbW91bnRfc25hcHNob3Qoa3RocmVhZF90ICp0ZCwg dm5vZGVfdCAqKnZwcCwgY29uc3QgY2hhCiAJCXZmc19mcmVlb3B0cyhtcC0+bW50X29wdCk7CiAJ bXAtPm1udF9vcHQgPSBtcC0+bW50X29wdG5ldzsKIAkodm9pZCkgVkZTX1NUQVRGUyhtcCwgJm1w LT5tbnRfc3RhdCk7CisKKwkvKgorCSAqIElmIHRoZSBwYXJlbnQgbW91bnQgaGFzIG1udF9leGph aWwgc2V0LCBzZXQgbW50X2V4amFpbCB0byB0aGUKKwkgKiBzYW1lIGNyZWRlbnRpYWxzLgorCSAq LworCXZmc19leGphaWxfY2xvbmUocGFyZW50X3Zmc3AsIG1wKTsKIAogCS8qCiAJICogUHJldmVu dCBleHRlcm5hbCBjb25zdW1lcnMgb2YgbW91bnQgb3B0aW9ucyBmcm9tIHJlYWRpbmcKLS0tIHN5 cy9jb250cmliL29wZW56ZnMvbW9kdWxlL29zL2ZyZWVic2QvemZzL3pmc19jdGxkaXIuYy56ZnNz bmFwCTIwMjMtMTEtMTggMTg6MDE6NTMuNjYxNjgzMDAwIC0wODAwCisrKyBzeXMvY29udHJpYi9v cGVuemZzL21vZHVsZS9vcy9mcmVlYnNkL3pmcy96ZnNfY3RsZGlyLmMJMjAyMy0xMS0xOCAxODow Mjo0OC41MDkzNTYwMDAgLTA4MDAKQEAgLTEwMjYsNyArMTAyNiw4IEBAIHpmc2N0bF9zbmFwZGly X2xvb2t1cChzdHJ1Y3Qgdm9wX2xvb2t1cF9hcmdzICphcCkKIAkgICAgIiVzLyIgWkZTX0NUTERJ Ul9OQU1FICIvc25hcHNob3QvJXMiLAogCSAgICBkdnAtPnZfdmZzcC0+bW50X3N0YXQuZl9tbnRv bm5hbWUsIG5hbWUpOwogCi0JZXJyID0gbW91bnRfc25hcHNob3QoY3VydGhyZWFkLCB2cHAsICJ6 ZnMiLCBtb3VudHBvaW50LCBmdWxsbmFtZSwgMCk7CisJZXJyID0gbW91bnRfc25hcHNob3QoY3Vy dGhyZWFkLCB2cHAsICJ6ZnMiLCBtb3VudHBvaW50LCBmdWxsbmFtZSwgMCwKKwkgICAgZHZwLT52 X3Zmc3ApOwogCWttZW1fZnJlZShtb3VudHBvaW50LCBtb3VudHBvaW50X2xlbik7CiAJaWYgKGVy ciA9PSAwKSB7CiAJCS8qCi0tLSBzeXMva2Vybi92ZnNfbW91bnQuYy56ZnNzbmFwCTIwMjMtMTEt MTggMTc6Mzc6MjIuOTc2NTQ0MDAwIC0wODAwCisrKyBzeXMva2Vybi92ZnNfbW91bnQuYwkyMDIz LTExLTE4IDE4OjUwOjI3LjA5ODI1NDAwMCAtMDgwMApAQCAtMzE0MSwzICszMTQxLDI5IEBAIHJl c3VtZV9hbGxfZnModm9pZCkKIAl9CiAJbXR4X3VubG9jaygmbW91bnRsaXN0X210eCk7CiB9CisK Ky8qCisgKiBDbG9uZSB0aGUgbW50X2V4amFpbCBmaWVsZCB0byBhIG5ldyBtb3VudCBwb2ludC4K KyAqLwordm9pZAordmZzX2V4amFpbF9jbG9uZShzdHJ1Y3QgbW91bnQgKmlubXAsIHN0cnVjdCBt b3VudCAqb3V0bXApCit7CisJc3RydWN0IHVjcmVkICpjcjsKKworCU1OVF9JTE9DSyhpbm1wKTsK KwljciA9IGlubXAtPm1udF9leGphaWw7CisJaWYgKGNyICE9IE5VTEwpIHsKKwkJY3Job2xkKGNy KTsKKwkJTU5UX0lVTkxPQ0soaW5tcCk7CisJCU1OVF9JTE9DSyhvdXRtcCk7CisJCWlmIChvdXRt cC0+bW50X2V4amFpbCA9PSBOVUxMKSB7CisJCQlvdXRtcC0+bW50X2V4amFpbCA9IGNyOworCQkJ YXRvbWljX2FkZF9pbnQoJmNyLT5jcl9wcmlzb24tPnByX2V4cG9ydGNudCwgMSk7CisJCQljciA9 IE5VTEw7CisJCX0KKwkJTU5UX0lVTkxPQ0sob3V0bXApOworCQlpZiAoY3IgIT0gTlVMTCkKKwkJ CWNyZnJlZShjcik7CisJfSBlbHNlCisJCU1OVF9JVU5MT0NLKGlubXApOworfQotLS0gc3lzL3N5 cy9tb3VudC5oLnpmc3NuYXAJMjAyMy0xMS0xOCAxNjo0MDoyOC4yMjg4NTkwMDAgLTA4MDAKKysr IHN5cy9zeXMvbW91bnQuaAkyMDIzLTExLTE4IDE4OjA2OjM0LjI1MzM1NDAwMCAtMDgwMApAQCAt MTAxNyw2ICsxMDE3LDcgQEAgaW50CXZmc19zZXRwdWJsaWNmcwkJCSAgICAvKiBzZXQgcHVibGlj bHkgZXhwb3J0ZWQgZnMgCiAJICAgIChzdHJ1Y3QgbW91bnQgKiwgc3RydWN0IG5ldGV4cG9ydCAq LCBzdHJ1Y3QgZXhwb3J0X2FyZ3MgKik7CiB2b2lkCXZmc19wZXJpb2RpYyhzdHJ1Y3QgbW91bnQg KiwgaW50KTsKIGludAl2ZnNfYnVzeShzdHJ1Y3QgbW91bnQgKiwgaW50KTsKK3ZvaWQJdmZzX2V4 amFpbF9jbG9uZShzdHJ1Y3QgbW91bnQgKiwgc3RydWN0IG1vdW50ICopOwogdm9pZAl2ZnNfZXhq YWlsX2RlbGV0ZShzdHJ1Y3QgcHJpc29uICopOwogaW50CXZmc19leHBvcnQJCQkgLyogcHJvY2Vz cyBtb3VudCBleHBvcnQgaW5mbyAqLwogCSAgICAoc3RydWN0IG1vdW50ICosIHN0cnVjdCBleHBv cnRfYXJncyAqLCBib29sKTsK --00000000000071e6ce060a78d585--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAM5tNy6jqH3nRrtAhTaOpAR1RCXKxygSKHtDMKeQStDnkC1skg>