Date: Tue, 29 Nov 2022 16:21:28 -0800 From: Rick Macklem <rick.macklem@gmail.com> To: Peter Eriksson <pen@lysator.liu.se> Cc: FreeBSD CURRENT <freebsd-current@freebsd.org>, "Bjoern A. Zeeb" <bz@freebsd.org>, Alan Somers <asomers@freebsd.org> Subject: Re: RFC: nfsd in a vnet jail Message-ID: <CAM5tNy71UAOkCQb9upc_OxhM-y5rp9jMKbKTJr619JFCGsfRkg@mail.gmail.com> In-Reply-To: <82103A1E-9D39-47B0-9520-205583C8B680@lysator.liu.se> References: <CAM5tNy7CQaBTRWG0m0aN6T0xG2L2zSQJGa%2BatGaH%2BmW%2BwEpdyQ@mail.gmail.com> <CAOtMX2hxeeNMxxdpma8NJ7ms60eRfuCWoFi7FixdSe83=qibkA@mail.gmail.com> <82103A1E-9D39-47B0-9520-205583C8B680@lysator.liu.se>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000096ddc205eea51614
Content-Type: text/plain; charset="UTF-8"
On Sun, Nov 27, 2022 at 10:04 AM Peter Eriksson <pen@lysator.liu.se> wrote:
> Keep the global variables as defaults that apply to all nfsds and allow
> (at least some subset) to be overridden inside the net jails if some things
> need to be changed from the defaults?
>
> This is pretty much a reply to one of the posts selected at random,
but I thought that better than starting a new email thread.
bz@ and asomers@ have both asked about running mountd within a vnet prison
(one via offlist email and the other on phabricator).
I think it is worth discussing here...
mountd (rightly or wrongly) does two distinctly different things:
1 - It pushes the exports into the kernel via nmount() so they
can be hung off of the "struct mount" for a file system's
mount point.
--> This can only work for file system mount points and can
only be done once for any given file system mount point.
At this time, I have it done once globally outside of the prisons.
The alternative I can see is doing it within each prison, but I
think that would require that each prison have its own file system(s).
(ie. The prison's root would always be a file system mount point.)
2 - It handles RPC Mount protocol requests from NFSv3 clients. This one
is NFSv3 specific, which is why I have done this NFSv4 only at
this time. To do this, it must be able to register with rpcbind,
and I have no idea if running rpcbind in a vnet jail is practical.
Enforcing the use for separate file systems for each jail also makes
things safer, since the exports are enforced by the kernel. Without
this, a malicious NFSv4 client could "guess" a file handle for a file
outside the jail and gain access to that file. Put another way, without
a separate file system, there is no way to stop a malicious client from
finding files above the Root file handle. (Normal clients will use
PutRootFH and LookupParent and these won't be able to go above the top
of the jail.)
So, what do others think of enforcing the requirement that each jail
have its own file systems for this?
rick
> - Peter
>
>
> On Fri, Nov 25, 2022, 4:24 PM Rick Macklem <rick.macklem@gmail.com> wrote:
>
>> Hi,
>>
>> bz@ has encouraged me to fiddle with the nfsd
>> so that it works in a vnet jail.
>> I have now basically done so, specifically for
>> NFSv4, since NFSv3 presents various issues.
>>
>> What I have not yet done is put global variables
>> in the vnet. This needs to be done so that the nfsd
>> can be run in multiple jail instances and/or in and
>> outside of a jail.
>> The problem is that there are 100s of global variables.
>>
>> I can see two approaches:
>> 1 - Move them all into the vnet jail. This would imply
>> that all the sysctls need to somehow be changed,
>> which would seem to be a POLA violation.
>> It also implies a lot of stuff in the vnet.
>> 2 - Just move the global variables that will always
>> differ from one nfsd to another (this would make
>> the sysctls global and apply to all nfsds).
>> This will keep the number of globals in the vnet
>> smaller.
>>
>> I am currently leaning towards #2, put what do others
>> think?
>>
>> rick
>> ps: Personally, I don't know what use there is of
>> running the nfsd inside a vnet jail, but bz@ has
>> some use case.
>>
>
>
--00000000000096ddc205eea51614
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"fon=
t-family:monospace"><br></div></div><br><div class=3D"gmail_quote"><div dir=
=3D"ltr" class=3D"gmail_attr">On Sun, Nov 27, 2022 at 10:04 AM Peter Erikss=
on <<a href=3D"mailto:pen@lysator.liu.se">pen@lysator.liu.se</a>> wro=
te:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px =
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Keep th=
e global variables as defaults that apply to all nfsds and allow (at least =
some subset) to be overridden inside the net jails if some things need to b=
e changed from the defaults?<div><br></div></div></blockquote><div><span cl=
ass=3D"gmail_default" style=3D"font-family:monospace">This is pretty much a=
reply to one of the posts selected at random,</span></div><div><span class=
=3D"gmail_default" style=3D"font-family:monospace">but I thought that bette=
r than starting a new email thread.</span></div><div><span class=3D"gmail_d=
efault" style=3D"font-family:monospace"><br></span></div><div><span class=
=3D"gmail_default" style=3D"font-family:monospace">bz@ and asomers@ have bo=
th asked about running mountd within a vnet prison</span></div><div><span c=
lass=3D"gmail_default" style=3D"font-family:monospace">(one via offlist ema=
il and the other on phabricator).</span></div><div><span class=3D"gmail_def=
ault" style=3D"font-family:monospace"><br></span></div><div><span class=3D"=
gmail_default" style=3D"font-family:monospace">I think it is worth discussi=
ng here...</span></div><div><span class=3D"gmail_default" style=3D"font-fam=
ily:monospace">mountd (rightly or wrongly) does two distinctly different th=
ings:</span></div><div><span class=3D"gmail_default" style=3D"font-family:m=
onospace">1 - It pushes the exports into the kernel via nmount() so they</s=
pan></div><div><span class=3D"gmail_default" style=3D"font-family:monospace=
">=C2=A0 =C2=A0 can be hung off of the "struct mount" for a file =
system's</span></div><div><span class=3D"gmail_default" style=3D"font-f=
amily:monospace">=C2=A0 =C2=A0 mount point.</span></div><div><span class=3D=
"gmail_default" style=3D"font-family:monospace">=C2=A0 =C2=A0 --> This c=
an only work for file system mount points and can</span></div><div><span cl=
ass=3D"gmail_default" style=3D"font-family:monospace">=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 only be done once for any given file system mount point.</span></div=
><div><span class=3D"gmail_default" style=3D"font-family:monospace"><br></s=
pan></div><div><span class=3D"gmail_default" style=3D"font-family:monospace=
">=C2=A0 =C2=A0 At this time, I have it done once globally outside of the p=
risons.</span></div><div><span class=3D"gmail_default" style=3D"font-family=
:monospace">=C2=A0 =C2=A0 The alternative I can see is doing it within each=
prison, but I</span></div><div><span class=3D"gmail_default" style=3D"font=
-family:monospace">=C2=A0 =C2=A0 think that would require that each prison =
have its own file system(s).</span></div><div><span class=3D"gmail_default"=
style=3D"font-family:monospace">=C2=A0 =C2=A0 (ie. The prison's root w=
ould always be a file system mount point.)</span></div><div><span class=3D"=
gmail_default" style=3D"font-family:monospace"><br></span></div><div><span =
class=3D"gmail_default" style=3D"font-family:monospace">2 - It handles RPC =
Mount protocol requests from NFSv3 clients.=C2=A0 This one</span></div><div=
><span class=3D"gmail_default" style=3D"font-family:monospace">=C2=A0 =C2=
=A0 is NFSv3 specific, which is why I have done this NFSv4 only at</span></=
div><div><span class=3D"gmail_default" style=3D"font-family:monospace">=C2=
=A0 =C2=A0 this time.=C2=A0 To do this, it must be able to register with rp=
cbind,</span></div><div><span class=3D"gmail_default" style=3D"font-family:=
monospace">=C2=A0 =C2=A0 and I have no idea if running rpcbind in a vnet ja=
il is practical.</span></div><div><span class=3D"gmail_default" style=3D"fo=
nt-family:monospace"><br></span></div><div><span class=3D"gmail_default" st=
yle=3D"font-family:monospace">Enforcing the use for separate file systems f=
or each jail also makes</span></div><div><span class=3D"gmail_default" styl=
e=3D"font-family:monospace">things safer, since the exports are enforced by=
the kernel. Without</span></div><div><span class=3D"gmail_default" style=
=3D"font-family:monospace">this, a malicious NFSv4 client could "guess=
" a file handle for a file</span></div><div><span class=3D"gmail_defau=
lt" style=3D"font-family:monospace">outside the jail and gain access to tha=
t file. Put another way, without</span></div><div><span class=3D"gmail_defa=
ult" style=3D"font-family:monospace">a separate file system, there is no wa=
y to stop a malicious client from</span></div><div><span class=3D"gmail_def=
ault" style=3D"font-family:monospace">finding files above the Root file han=
dle. (Normal clients will use</span></div><div><span class=3D"gmail_default=
" style=3D"font-family:monospace">PutRootFH and LookupParent and these won&=
#39;t be able to go above the top</span></div><div><span class=3D"gmail_def=
ault" style=3D"font-family:monospace">of the jail.)</span></div><div><span =
class=3D"gmail_default" style=3D"font-family:monospace"><br></span></div><d=
iv><span class=3D"gmail_default" style=3D"font-family:monospace">So, what d=
o others think of enforcing the requirement that each jail</span></div><div=
><span class=3D"gmail_default" style=3D"font-family:monospace">have its own=
file systems for this?</span></div><div><span class=3D"gmail_default" styl=
e=3D"font-family:monospace"><br></span></div><div><span class=3D"gmail_defa=
ult" style=3D"font-family:monospace">rick</span></div><div><span class=3D"g=
mail_default" style=3D"font-family:monospace"></span>=C2=A0</div><blockquot=
e class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px s=
olid rgb(204,204,204);padding-left:1ex"><div><div></div><div>- Peter</div><=
div><br></div><div><br><div><div><div dir=3D"auto"><div><div class=3D"gmail=
_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, Nov 25, 2022, 4:24 PM=
Rick Macklem <<a href=3D"mailto:rick.macklem@gmail.com" target=3D"_blan=
k">rick.macklem@gmail.com</a>> wrote:<br></div><blockquote class=3D"gmai=
l_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,20=
4,204);padding-left:1ex"><div dir=3D"ltr"><div style=3D"font-family:monospa=
ce">Hi,</div><div style=3D"font-family:monospace"><br></div><div style=3D"f=
ont-family:monospace">bz@ has encouraged me to fiddle with the nfsd</div><d=
iv style=3D"font-family:monospace">so that it works in a vnet jail.</div><d=
iv style=3D"font-family:monospace">I have now basically done so, specifical=
ly for</div><div style=3D"font-family:monospace">NFSv4, since NFSv3 present=
s various issues.</div><div style=3D"font-family:monospace"><br></div><div =
style=3D"font-family:monospace">What I have not yet done is put global vari=
ables</div><div style=3D"font-family:monospace">in the vnet. This needs to =
be done so that the nfsd</div><div style=3D"font-family:monospace">can be r=
un in multiple jail instances and/or in and</div><div style=3D"font-family:=
monospace">outside of a jail.</div><div style=3D"font-family:monospace">The=
problem is that there are 100s of global variables.</div><div style=3D"fon=
t-family:monospace"><br></div><div style=3D"font-family:monospace">I can se=
e two approaches:</div><div style=3D"font-family:monospace">1 - Move them a=
ll into the vnet jail. This would imply</div><div style=3D"font-family:mono=
space">=C2=A0 =C2=A0 that all the sysctls need to somehow be changed,</div>=
<div style=3D"font-family:monospace">=C2=A0 =C2=A0 which would seem to be a=
POLA violation.</div><div style=3D"font-family:monospace">=C2=A0 =C2=A0 It=
also implies a lot of stuff in the vnet.</div><div style=3D"font-family:mo=
nospace">2 - Just move the global variables that will always</div><div styl=
e=3D"font-family:monospace">=C2=A0 =C2=A0 differ from one nfsd to another (=
this would make</div><div style=3D"font-family:monospace">=C2=A0 =C2=A0 the=
sysctls global and apply to all nfsds).</div><div style=3D"font-family:mon=
ospace">=C2=A0 =C2=A0 This will keep the number of globals in the vnet</div=
><div style=3D"font-family:monospace">=C2=A0 =C2=A0 smaller.</div><div styl=
e=3D"font-family:monospace"><br></div><div style=3D"font-family:monospace">=
I am currently leaning towards #2, put what do others</div><div style=3D"fo=
nt-family:monospace">think?</div><div style=3D"font-family:monospace"><br><=
/div><div style=3D"font-family:monospace">rick</div><div style=3D"font-fami=
ly:monospace">ps: Personally, I don't know what use there is of</div><d=
iv style=3D"font-family:monospace">=C2=A0 =C2=A0 running the nfsd inside a =
vnet jail, but bz@ has</div><div style=3D"font-family:monospace">=C2=A0 =C2=
=A0 some use case.</div></div></blockquote></div></div></div></div><blockqu=
ote type=3D"cite"><div><div dir=3D"auto"><div><div class=3D"gmail_quote"><b=
lockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-le=
ft:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"></div></bl=
ockquote></div></div></div>
</div></blockquote></div><br></div></div></blockquote></div></div>
--00000000000096ddc205eea51614--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAM5tNy71UAOkCQb9upc_OxhM-y5rp9jMKbKTJr619JFCGsfRkg>