Date: Thu, 1 Dec 2022 17:32:26 -0800 From: Rick Macklem <rick.macklem@gmail.com> To: Chris <bsd-lists@bsdforge.com> Cc: Peter Eriksson <pen@lysator.liu.se>, FreeBSD CURRENT <freebsd-current@freebsd.org>, "Bjoern A. Zeeb" <bz@freebsd.org>, Alan Somers <asomers@freebsd.org> Subject: Re: RFC: nfsd in a vnet jail Message-ID: <CAM5tNy7Uc82OqhjVc3W84GAYcJd6P4D4U_criEYTwFmvEU61xg@mail.gmail.com> In-Reply-To: <2980bcbd22f884962d358808f9440d77@bsdforge.com> References: <CAM5tNy7CQaBTRWG0m0aN6T0xG2L2zSQJGa%2BatGaH%2BmW%2BwEpdyQ@mail.gmail.com> <CAOtMX2hxeeNMxxdpma8NJ7ms60eRfuCWoFi7FixdSe83=qibkA@mail.gmail.com> <82103A1E-9D39-47B0-9520-205583C8B680@lysator.liu.se> <CAM5tNy71UAOkCQb9upc_OxhM-y5rp9jMKbKTJr619JFCGsfRkg@mail.gmail.com> <2980bcbd22f884962d358808f9440d77@bsdforge.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000000debab05eece505c Content-Type: text/plain; charset="UTF-8" On Thu, Dec 1, 2022 at 8:23 AM Chris <bsd-lists@bsdforge.com> wrote: > On 2022-11-29 16:21, Rick Macklem wrote: > > On Sun, Nov 27, 2022 at 10:04 AM Peter Eriksson <pen@lysator.liu.se> > wrote: > > > >> Keep the global variables as defaults that apply to all nfsds and allow > >> (at least some subset) to be overridden inside the net jails if some > things > >> need to be changed from the defaults? > >> > >> This is pretty much a reply to one of the posts selected at random, > > but I thought that better than starting a new email thread. > > > > bz@ and asomers@ have both asked about running mountd within a vnet > prison > > (one via offlist email and the other on phabricator). > > > > I think it is worth discussing here... > > mountd (rightly or wrongly) does two distinctly different things: > > 1 - It pushes the exports into the kernel via nmount() so they > > can be hung off of the "struct mount" for a file system's > > mount point. > > --> This can only work for file system mount points and can > > only be done once for any given file system mount point. > > > > At this time, I have it done once globally outside of the prisons. > > The alternative I can see is doing it within each prison, but I > > think that would require that each prison have its own file > system(s). > > (ie. The prison's root would always be a file system mount point.) > > > > 2 - It handles RPC Mount protocol requests from NFSv3 clients. This one > > is NFSv3 specific, which is why I have done this NFSv4 only at > > this time. To do this, it must be able to register with rpcbind, > > and I have no idea if running rpcbind in a vnet jail is practical. > > > > Enforcing the use for separate file systems for each jail also makes > > things safer, since the exports are enforced by the kernel. Without > > this, a malicious NFSv4 client could "guess" a file handle for a file > > outside the jail and gain access to that file. Put another way, without > > a separate file system, there is no way to stop a malicious client from > > finding files above the Root file handle. (Normal clients will use > > PutRootFH and LookupParent and these won't be able to go above the top > > of the jail.) > > > > So, what do others think of enforcing the requirement that each jail > > have its own file systems for this? > > I don't care for any of it. It looks like additional overhead with the > addition of potential security risks. All for a very limited (and as yet > unknown) use case. > I am thinking that if/when this goes into main, it would be under a new kernel build option called something like NFSD_VIMAGE. I think that would avoid the overhead/security risks for those that do not need/want it. rick > > --chris > > > > rick > > > > > >> - Peter > >> > >> > >> On Fri, Nov 25, 2022, 4:24 PM Rick Macklem <rick.macklem@gmail.com> > wrote: > >> > >>> Hi, > >>> > >>> bz@ has encouraged me to fiddle with the nfsd > >>> so that it works in a vnet jail. > >>> I have now basically done so, specifically for > >>> NFSv4, since NFSv3 presents various issues. > >>> > >>> What I have not yet done is put global variables > >>> in the vnet. This needs to be done so that the nfsd > >>> can be run in multiple jail instances and/or in and > >>> outside of a jail. > >>> The problem is that there are 100s of global variables. > >>> > >>> I can see two approaches: > >>> 1 - Move them all into the vnet jail. This would imply > >>> that all the sysctls need to somehow be changed, > >>> which would seem to be a POLA violation. > >>> It also implies a lot of stuff in the vnet. > >>> 2 - Just move the global variables that will always > >>> differ from one nfsd to another (this would make > >>> the sysctls global and apply to all nfsds). > >>> This will keep the number of globals in the vnet > >>> smaller. > >>> > >>> I am currently leaning towards #2, put what do others > >>> think? > >>> > >>> rick > >>> ps: Personally, I don't know what use there is of > >>> running the nfsd inside a vnet jail, but bz@ has > >>> some use case. > >>> > >> > >> > --0000000000000debab05eece505c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"fon= t-family:monospace"><br></div></div><br><div class=3D"gmail_quote"><div dir= =3D"ltr" class=3D"gmail_attr">On Thu, Dec 1, 2022 at 8:23 AM Chris <<a h= ref=3D"mailto:bsd-lists@bsdforge.com">bsd-lists@bsdforge.com</a>> wrote:= <br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8= ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 2022-11-29 1= 6:21, Rick Macklem wrote:<br> > On Sun, Nov 27, 2022 at 10:04 AM Peter Eriksson <<a href=3D"mailto:= pen@lysator.liu.se" target=3D"_blank">pen@lysator.liu.se</a>> wrote:<br> > <br> >> Keep the global variables as defaults that apply to all nfsds and = allow<br> >> (at least some subset) to be overridden inside the net jails if so= me things<br> >> need to be changed from the defaults?<br> >> <br> >> This is pretty much a reply to one of the posts selected at random= ,<br> > but I thought that better than starting a new email thread.<br> > <br> > bz@ and asomers@ have both asked about running mountd within a vnet pr= ison<br> > (one via offlist email and the other on phabricator).<br> > <br> > I think it is worth discussing here...<br> > mountd (rightly or wrongly) does two distinctly different things:<br> > 1 - It pushes the exports into the kernel via nmount() so they<br> >=C2=A0 =C2=A0 =C2=A0can be hung off of the "struct mount" for= a file system's<br> >=C2=A0 =C2=A0 =C2=A0mount point.<br> >=C2=A0 =C2=A0 =C2=A0--> This can only work for file system mount poi= nts and can<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0only be done once for any given file = system mount point.<br> > <br> >=C2=A0 =C2=A0 =C2=A0At this time, I have it done once globally outside = of the prisons.<br> >=C2=A0 =C2=A0 =C2=A0The alternative I can see is doing it within each p= rison, but I<br> >=C2=A0 =C2=A0 =C2=A0think that would require that each prison have its = own file system(s).<br> >=C2=A0 =C2=A0 =C2=A0(ie. The prison's root would always be a file s= ystem mount point.)<br> > <br> > 2 - It handles RPC Mount protocol requests from NFSv3 clients.=C2=A0 T= his one<br> >=C2=A0 =C2=A0 =C2=A0is NFSv3 specific, which is why I have done this NF= Sv4 only at<br> >=C2=A0 =C2=A0 =C2=A0this time.=C2=A0 To do this, it must be able to reg= ister with rpcbind,<br> >=C2=A0 =C2=A0 =C2=A0and I have no idea if running rpcbind in a vnet jai= l is practical.<br> > <br> > Enforcing the use for separate file systems for each jail also makes<b= r> > things safer, since the exports are enforced by the kernel. Without<br= > > this, a malicious NFSv4 client could "guess" a file handle f= or a file<br> > outside the jail and gain access to that file. Put another way, withou= t<br> > a separate file system, there is no way to stop a malicious client fro= m<br> > finding files above the Root file handle. (Normal clients will use<br> > PutRootFH and LookupParent and these won't be able to go above the= top<br> > of the jail.)<br> > <br> > So, what do others think of enforcing the requirement that each jail<b= r> > have its own file systems for this?<br> <br> I don't care for any of it. It looks like additional overhead with the<= br> addition of potential security risks. All for a very limited (and as yet<br= > unknown) use case.<br></blockquote><div><span class=3D"gmail_default" style= =3D"font-family:monospace">I am thinking that if/when this goes into main, = it would be</span></div><div><span class=3D"gmail_default" style=3D"font-fa= mily:monospace">under a new kernel build option called something like</span= ></div><div><span class=3D"gmail_default" style=3D"font-family:monospace">N= FSD_VIMAGE. I think that would avoid the overhead/security</span></div><div= ><span class=3D"gmail_default" style=3D"font-family:monospace">risks for th= ose that do not need/want it.</span></div><div><span class=3D"gmail_default= " style=3D"font-family:monospace"><br></span></div><div><span class=3D"gmai= l_default" style=3D"font-family:monospace">rick</span><span class=3D"gmail_= default" style=3D"font-family:monospace"></span>=C2=A0</div><blockquote cla= ss=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex"> <br> --chris<br> > <br> > rick<br> > <br> > <br> >> - Peter<br> >> <br> >> <br> >> On Fri, Nov 25, 2022, 4:24 PM Rick Macklem <<a href=3D"mailto:r= ick.macklem@gmail.com" target=3D"_blank">rick.macklem@gmail.com</a>> wro= te:<br> >> <br> >>> Hi,<br> >>> <br> >>> bz@ has encouraged me to fiddle with the nfsd<br> >>> so that it works in a vnet jail.<br> >>> I have now basically done so, specifically for<br> >>> NFSv4, since NFSv3 presents various issues.<br> >>> <br> >>> What I have not yet done is put global variables<br> >>> in the vnet. This needs to be done so that the nfsd<br> >>> can be run in multiple jail instances and/or in and<br> >>> outside of a jail.<br> >>> The problem is that there are 100s of global variables.<br> >>> <br> >>> I can see two approaches:<br> >>> 1 - Move them all into the vnet jail. This would imply<br> >>>=C2=A0 =C2=A0 =C2=A0that all the sysctls need to somehow be cha= nged,<br> >>>=C2=A0 =C2=A0 =C2=A0which would seem to be a POLA violation.<br= > >>>=C2=A0 =C2=A0 =C2=A0It also implies a lot of stuff in the vnet.= <br> >>> 2 - Just move the global variables that will always<br> >>>=C2=A0 =C2=A0 =C2=A0differ from one nfsd to another (this would= make<br> >>>=C2=A0 =C2=A0 =C2=A0the sysctls global and apply to all nfsds).= <br> >>>=C2=A0 =C2=A0 =C2=A0This will keep the number of globals in the= vnet<br> >>>=C2=A0 =C2=A0 =C2=A0smaller.<br> >>> <br> >>> I am currently leaning towards #2, put what do others<br> >>> think?<br> >>> <br> >>> rick<br> >>> ps: Personally, I don't know what use there is of<br> >>>=C2=A0 =C2=A0 =C2=A0running the nfsd inside a vnet jail, but bz= @ has<br> >>>=C2=A0 =C2=A0 =C2=A0some use case.<br> >>> <br> >> <br> >> <br> </blockquote></div></div> --0000000000000debab05eece505c--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAM5tNy7Uc82OqhjVc3W84GAYcJd6P4D4U_criEYTwFmvEU61xg>