Date: Sat, 23 Mar 2024 16:06:35 +0000 From: Shawn Webb <shawn.webb@hardenedbsd.org> To: alan somers <asomers@gmail.com> Cc: Alan Somers <asomers@freebsd.org>, freebsd-hackers@freebsd.org Subject: Re: Filesystem extended attributes and Capsicum Message-ID: <tu57r3yxglz5pyijfiaevwvae5cu7nf36fbzzn5hctvvx7xqug@ftgs55qg4cxs> In-Reply-To: <56n3xzefvyd3s3b6p3ipggf5hshawrts4nezkw2oud3haihdhj@besj4txt7yar> References: <bjeg3z2aa5owo7uur75olwuw6le2g3whzxwfgef4ozyy6w6kog@33b426ax6inf> <CAOtMX2gC3iZ6X6HotpgxgfAfPN-8C7rg3ZCP%2Bpr5e8KddasUsQ@mail.gmail.com> <ae4isfrqtyc7l3rwqd2rz6efmuztwqxw5h67xqaqts6ra6br3l@ly347trhgdoq> <CAOtMX2hiWRPQM5mbrhTifnycgroQ2Y_MeCHN8bZ9mfypQYR_GA@mail.gmail.com> <gmrellzwibhxqazp3vt22tqk2flrsyklrcwrufr4mgnhjnj3cn@rufmfxktmop5> <CAOtMX2hoXr6-6rJHRcmistroZLqyZZLeCByU_amPNPVikOnG_A@mail.gmail.com> <tnbavez4e42ovbvyjrzguf6xhydrgrn3olfcwhuouxwhzh57c6@jplqitihfove> <om7536duc4lo6zsu2rmwgtghya6uxouwvuxec6czxp3lp4y7ey@pov3ksihtthl> <56n3xzefvyd3s3b6p3ipggf5hshawrts4nezkw2oud3haihdhj@besj4txt7yar>
next in thread | previous in thread | raw e-mail | index | archive | help
--qpnblpzeya3764pj
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Sat, Mar 23, 2024 at 04:00:42PM +0000, Shawn Webb wrote:
> On Sat, Mar 23, 2024 at 02:54:22PM +0000, Shawn Webb wrote:
> > On Sat, Mar 23, 2024 at 02:25:26PM +0000, Shawn Webb wrote:
> > > On Sat, Mar 23, 2024 at 07:58:37AM -0600, alan somers wrote:
> > > > On Fri, Mar 22, 2024 at 9:52=E2=80=AFPM Shawn Webb <shawn.webb@hard=
enedbsd.org> wrote:
> > > > >
> > > > > On Fri, Mar 22, 2024 at 08:07:17PM -0600, Alan Somers wrote:
> > > > > > On Fri, Mar 22, 2024 at 6:56=E2=80=AFPM Shawn Webb <shawn.webb@=
hardenedbsd.org> wrote:
> > > > > > >
> > > > > > > On Fri, Mar 22, 2024 at 06:20:48PM -0600, Alan Somers wrote:
> > > > > > > > On Fri, Mar 22, 2024 at 5:38=E2=80=AFPM Shawn Webb <shawn.w=
ebb@hardenedbsd.org> wrote:
> > > > > > > > >
> > > > > > > > > Hey all,
> > > > > > > > >
> > > > > > > > > I'm writing an application in which I hope to enable Caps=
icum. I'm
> > > > > > > > > experiencing an issue whereby extattr_get_fd fails with a=
file
> > > > > > > > > descriptor that has all the extended attribute capabiliti=
es enabled
> > > > > > > > > (CAP_EXTATTR_DELETE, CAP_EXTATTR_GET, CAP_EXTATTR_LIST, a=
nd
> > > > > > > > > CAP_EXTATTR_SET).
> > > > > > > > >
> > > > > > > > > Looking at the kernel source (sys/kern/vfs_extattr.c) tel=
ls me that
> > > > > > > > > kern_extattr_get_fd only requires CAP_EXTATTR_GET.
> > > > > > > > >
> > > > > > > > > So I'm a bit puzzled as to why my call to extattr_get_fd(=
2) is
> > > > > > > > > failing. Am I doing something wrong or are filesystem ext=
ended
> > > > > > > > > attributes not supported in a Capabilities-enabled proces=
s?
> > > > > > > > >
> > > > > > > > > Here's how I'm creating the file descriptor (before calli=
ng
> > > > > > > > > cap_enter(2)):
> > > > > > > > >
> > > > > > > > > =3D=3D=3D=3D BEGIN CODE =3D=3D=3D=3D
> > > > > > > > > static int
> > > > > > > > > open_file(const char *path)
> > > > > > > > > {
> > > > > > > > > cap_rights_t rights;
> > > > > > > > > int fd;
> > > > > > > > >
> > > > > > > > > fd =3D open(path, O_PATH | O_CLOEXEC);
> > > > > > > > > if (fd =3D=3D -1) {
> > > > > > > > > return (-1);
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > memset(&rights, 0, sizeof(rights));
> > > > > > > > > cap_rights_init(&rights, CAP_EXTATTR_DELETE, CAP_=
EXTATTR_GET,
> > > > > > > > > CAP_EXTATTR_LIST, CAP_EXTATTR_SET);
> > > > > > > > > cap_rights_limit(fd, &rights);
> > > > > > > > >
> > > > > > > > > return (fd);
> > > > > > > > > }
> > > > > > > > > =3D=3D=3D=3D END CODE =3D=3D=3D=3D
> > > > > > > > >
> > > > > > > > > Eventually, after calling cap_enter(2), the following cod=
e is called:
> > > > > > > > >
> > > > > > > > > =3D=3D=3D=3D BEGIN CODE =3D=3D=3D=3D
> > > > > > > > > #define ATTRNAME_ENABLED "hbsd.pax.aslr"
> > > > > > > > > sz =3D extattr_get_fd(fd, ctx->hc_namespace, ATTR=
NAME_ENABLED, NULL, 0);
> > > > > > > > > if (sz <=3D 0) {
> > > > > > > > > if (errno =3D=3D ENOATTR) {
> > > > > > > > > /*
> > > > > > > > > * This is okay, it just means tha=
t nothing has been set.
> > > > > > > > > * No error condition here.
> > > > > > > > > */
> > > > > > > > > return (RES_SUCCESS);
> > > > > > > > > }
> > > > > > > > > return (RES_FAIL);
> > > > > > > > > }
> > > > > > > > > =3D=3D=3D=3D END CODE =3D=3D=3D=3D
> > > > > > > > >
> > > > > > > > > For reference, the program's code is here:
> > > > > > > > > https://git.hardenedbsd.org/shawn.webb/hbsdctrl/-/tree/ma=
in?ref_type=3Dheads
> > > > > > > > >
> > > > > > > > > The library code, which is what's responsible for calling=
the
> > > > > > > > > filesystem extended attribute related syscalls is here:
> > > > > > > > >
> > > > > > > > > https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/tre=
e/hardened/current/hbsdcontrol-v2/lib/libhbsdcontrol?ref_type=3Dheads
> > > > > > > > >
> > > > > > > > > From the rights(4) manual page, I'm instructed all I need=
are to apply
> > > > > > > > > those capabilities to that file descriptor:
> > > > > > > > >
> > > > > > > > > =3D=3D=3D=3D BEGIN PASTE =3D=3D=3D=3D
> > > > > > > > > CAP_EXTATTR_DELETE Permit extattr_delete_fd(2).
> > > > > > > > >
> > > > > > > > > CAP_EXTATTR_GET Permit extattr_get_fd(2).
> > > > > > > > >
> > > > > > > > > CAP_EXTATTR_LIST Permit extattr_list_fd(2).
> > > > > > > > >
> > > > > > > > > CAP_EXTATTR_SET Permit extattr_set_fd(2).
> > > > > > > > > =3D=3D=3D=3D END PASTE =3D=3D=3D=3D
> > > > > > > > >
> > > > > > > > > So I'm a bit unsure if I'm doing something wrong.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Shawn Webb
> > > > > > > > > Cofounder / Security Engineer
> > > > > > > > > HardenedBSD
> > > > > > > > >
> > > > > > > > > Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50
> > > > > > > > > https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/mas=
ter/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc
> > > > > > > >
> > > > > > > > What error code does it fail with? If it's ENOTCAPABLE, th=
en I
> > > > > > > > suggest using dtrace to find the reason why it fails. Do s=
omething
> > > > > > > > like this:
> > > > > > > >
> > > > > > > > dtrace -i 'fbt:kernel::return /arg1 =3D=3D 93 && pid =3D=3D=
$target/
> > > > > > > > {trace(".");}' -c ./my_application
> > > > > > > >
> > > > > > > > That will print the name of every non-inlined kernel functi=
on that
> > > > > > > > returns ENOTCAPABLE during your process. But it will also =
print the
> > > > > > > > names of any other kernel functions that return an integer =
value of
> > > > > > > > 93. From there, guess which function is the real source of=
the error.
> > > > > > > > Then you can do
> > > > > > >
> > > > > > > DTrace is unavailable on this particular system.
> > > > > > >
> > > > > > > It does indeed fail with ENOTCAPABLE. I have the kern.trap_en=
otcap sysctl
> > > > > > > set to 1 so that I can know at exactly what point we're faili=
ng, and
> > > > > > > it's indeed at extattr_get_fd.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > --
> > > > > > > Shawn Webb
> > > > > > > Cofounder / Security Engineer
> > > > > > > HardenedBSD
> > > > > > >
> > > > > > > Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50
> > > > > > > https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/=
Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc
> > > > > >
> > > > > > Without dtrace, you've got your work cut out for you. I suggest
> > > > > > simply adding all capabilities, verifying that extattr_get_fd w=
orks,
> > > > > > and then removing capabilities until it fails. Or, run your pr=
ogram
> > > > > > on vanilla FreeBSD with dtrace.
> > > > >
> > > > > HardenedBSD doesn't have any modifications that would affect Caps=
icum
> > > > > in this manner. Regardless, I reproduced the problem successfully=
on
> > > > > FreeBSD 14.0-RELEASE without any code changes. I tried running yo=
ur
> > > > > DTrace script on FreeBSD 14.0-RELEASE and got this output:
> > > > >
> > > > > =3D=3D=3D=3D BEGIN OUTPUT =3D=3D=3D=3D
> > > > > $ sudo dtrace -i 'fbt:kernel::return /arg1 =3D=3D 93 && pid =3D=
=3D $target/ {trace(".");}' -c "obj/hbsdctrl pax list /bin/ls"
> > > > > dtrace: description 'fbt:kernel::return ' matched 31396 probes
> > > > > aslr: sysdef
> > > > > mprotect: sysdef
> > > > > pageexec: sysdef
> > > > > segvguard: sysdef
> > > > > dtrace: pid 29270 has exited
> > > > > CPU ID FUNCTION:NAME
> > > > > 3 47780 foffset_unlock_uio:return .
> > > > > 3 50605 foffset_lock:return .
> > > > > 3 47778 foffset_lock_uio:return .
> > > > > =3D=3D=3D=3D END OUTPUT =3D=3D=3D=3D
> > > > >
> > > > > But I'm still unsure what I'm missing, if anything.
> > > >=20
> > > > That's red herring. Those functions return void, but dtrace doesn't
> > > > know it. So the "93" is just register garbage. I also notice that
> > > > kern_extattr_get_fd isn't listed. Are you sure that your program is
> > > > really failing with ENOTCAPABLE? You can also try running it with
> > > > ktrace. kdump will show you exactly what capabilities you limited =
the
> > > > file descriptor to. That can help you verify if you applied the
> > > > limits correctly.
> > >=20
> > > I'm pretty sure it's failing with ENOTCAPABLE, since the
> > > kern.trap_enotcap sysctl logic is being hit at the point of the
> > > extattr_get_fd syscall. I'll see what I can do
> > > with ktrace.
> >=20
> > ktrace output captured here:
> > https://hardenedbsd.org/~shawn/2024-03-23_kdump-01.txt
> >=20
> > The relevant portion is pasted here:
> >=20
> > =3D=3D=3D=3D BEGIN ktrace OUTPUT =3D=3D=3D=3D
> > 41878 hbsdctrl CALL openat(AT_FDCWD,0x6894a36d4f86,0x500000<O_RDONLY|=
O_CLOEXEC|O_PATH>)
> > 41878 hbsdctrl NAMI "/scratch/tmp/ls"
> > 41878 hbsdctrl RET openat 3
> > 41878 hbsdctrl CALL cap_rights_limit(0x3,0x6894a36d4bc0)
> > 41878 hbsdctrl STRU cap_rights_t CAP_SEEK,CAP_FSTATAT,CAP_EXTATTR_DEL=
ETE,CAP_EXTATTR_GET,CAP_EXTATTR_LIST,CAP_EXTATTR_SET,CAP_ACL_GET
> > 41878 hbsdctrl RET cap_rights_limit 0
> > 41878 hbsdctrl CALL cap_enter
> > 41878 hbsdctrl RET cap_enter 0
> > 41878 hbsdctrl CALL extattr_get_fd(0x3,0x2,0x33ff63afbaf,0,0)
> > 41878 hbsdctrl NAMI "freebsd:system:hbsd.pax.segvguard"
> > 41878 hbsdctrl CAP restricted VFS lookup
> > 41878 hbsdctrl RET extattr_get_fd -1 errno 94 Not permitted in capab=
ility mode
> > 41878 hbsdctrl PSIG SIGTRAP SIG_DFL code=3DTRAP_CAP
> > =3D=3D=3D=3D END ktrace OUTPUT =3D=3D=3D=3D
> >=20
> > As you can see, I'm adding a bunch more capabilities to the file
> > descriptor. The point of failure is still extattr_get_fd returning
> > ENOTCAPABLE.
> >=20
> > I'm beginning to suspect that while extattr_get_fd is documented to be
> > allowed with a file descriptor with the CAP_EXTATTR_GET capability,
> > this functionality is broken in FreeBSD.
> >=20
> > The thing that really confuses me is that the extattr_get_fd syscall
> > entry in syscalls.master has the CAPENABLED flag set:
> >=20
> > =3D=3D=3D=3D BEGIN syscalls.master =3D=3D=3D=3D
> > 372 AUE_EXTATTR_GET_FD STD|CAPENABLED {
> > ssize_t extattr_get_fd(
> > int fd,
> > int attrnamespace,
> > _In_z_ const char *attrname,
> > _Out_writes_bytes_(nbytes) void *data,
> > size_t nbytes
> > );
> > }
> > =3D=3D=3D=3D END syscalls.master =3D=3D=3D=3D
> >=20
> > I think it's now time for me to file a bug report in Bugzilla.
>=20
> Bug report submitted:
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277908
Last message, sorry to bother. ZFS is indeed the culprit here:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277908#c1
Thanks,
--=20
Shawn Webb
Cofounder / Security Engineer
HardenedBSD
Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50
https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A=
4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc
--qpnblpzeya3764pj
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAmX+/gsACgkQ/y5nonf4
4folIw//W8aSWovfB3l7W3jW93QXFinDE9qmXzMbxMzs2cTVoi8l7GwENxaJIuNb
jDC74U4UeH/XBg6Ws+qVjpgj9ncqguS/dcRl/vdmBTAUegkb3gJzjsQO14MnwlnJ
8PCuGVRXrogSZRsQGhuyKbh7SXiCct/077uxO+iggD9r0GPyETWa8wHHZKtgOUNL
esy4cIJD77C2wYQEm7dRpPhWuS5Do6bvpApNthZ9XtDFoiKlbmvLLfVft+5NDMXL
XpLGSoz0h+9O1xNi72OfMwST0P3cpmEsz1x7NrHqHwde2JOk1g5g/wqSEexTi5Mb
U5Bh6BNSnItroF6YDbXMrr0JrSkI/0XlRHwJuFgC/4i4nMVpjsXKPRdv8xgH7mle
/ZCyDdwiaNgmU0+hFi/z8Uqqom161p4w/2WWOtp6AeMwVD7yYkL7D7w1TjoSt7iy
WKBFy5kkH2HFPm6QSFN/K68zBaZ+cnl2FQBg70F6exN6/blkByo6E3y3uEGatksJ
J8E5u/bzEBDA/9s+rYiW1wJQpBQ7RVDE94yFms5fbWeW1Ze3g4cx/McRv1aLAunR
tFxnXWpAb5/45DY+RW0S0OEM9GxoPZgPqG/AuGAWfOeRE+TlxVQpdoZnDb6mT5uS
51Gr8zgw/WWF/VYEHPE/VPZqV86HzTodfg0qYUDSMGJ9ih4wNkw=
=sJBp
-----END PGP SIGNATURE-----
--qpnblpzeya3764pj--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?tu57r3yxglz5pyijfiaevwvae5cu7nf36fbzzn5hctvvx7xqug>