Date: Mon, 28 Nov 2022 17:04:24 -0800 From: Rick Macklem <rick.macklem@gmail.com> To: Alan Somers <asomers@freebsd.org> Cc: FreeBSD CURRENT <freebsd-current@freebsd.org>, "Bjoern A. Zeeb" <bz@freebsd.org> Subject: Re: RFC: nfsd in a vnet jail Message-ID: <CAM5tNy7X81VVmYmWoAgrUoyjo96yqHb2-ZTys5D6ROQgvcoYBQ@mail.gmail.com> In-Reply-To: <CAOtMX2hxeeNMxxdpma8NJ7ms60eRfuCWoFi7FixdSe83=qibkA@mail.gmail.com> References: <CAM5tNy7CQaBTRWG0m0aN6T0xG2L2zSQJGa%2BatGaH%2BmW%2BwEpdyQ@mail.gmail.com> <CAOtMX2hxeeNMxxdpma8NJ7ms60eRfuCWoFi7FixdSe83=qibkA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000006e6e4805ee919271 Content-Type: text/plain; charset="UTF-8" On Fri, Nov 25, 2022 at 9:06 PM Alan Somers <asomers@freebsd.org> wrote: > > > On Fri, Nov 25, 2022, 4:24 PM Rick Macklem <rick.macklem@gmail.com> wrote: > >> Hi, >> >> bz@ has encouraged me to fiddle with the nfsd >> so that it works in a vnet jail. >> I have now basically done so, specifically for >> NFSv4, since NFSv3 presents various issues. >> >> What I have not yet done is put global variables >> in the vnet. This needs to be done so that the nfsd >> can be run in multiple jail instances and/or in and >> outside of a jail. >> The problem is that there are 100s of global variables. >> >> I can see two approaches: >> 1 - Move them all into the vnet jail. This would imply >> that all the sysctls need to somehow be changed, >> which would seem to be a POLA violation. >> It also implies a lot of stuff in the vnet. >> 2 - Just move the global variables that will always >> differ from one nfsd to another (this would make >> the sysctls global and apply to all nfsds). >> This will keep the number of globals in the vnet >> smaller. >> >> I am currently leaning towards #2, put what do others >> think? >> >> rick >> ps: Personally, I don't know what use there is of >> running the nfsd inside a vnet jail, but bz@ has >> some use case. >> > > This is super-awesome! Thank you so much! I've got a use case too. I > think it would be fine to leave most of the settings global, like > max_threads. But we should probably decide on a case by case basis . > The minthreads, maxthreads happen to be handled via nfsd command line options, so the sysctls are not needed and they can be set per-prison. Most of the sysctls are for weird cases or tuning of the DRC. Since the DRC is only used for NFSv4.0 mounts and not NFSv4.1 or NFSv4.2 ones, tuning the DRC should not usually be necessary. I have left them global for now. If anyone identifies one that needs to be set per-prison, I can move it into the vnet. If you want to see them all: # sysctl -a | fgrep vfs.nfsd I have put a first patch up on phabricator as D37519. Although I listed three people as reviewers, anyone is welcome to test/comment/review. If you can't easily get the patch from phabricator, just email me and I'll send it to you. I think it will apply cleanly to main and, maybe, stable/13. You only need to build a kernel from patched sources to test it. There is a change to rc.d/nfsd, which you only need in the prison's etc/rc.d/nfsd. A very basic setup document (also definitely a work in progress) can be found at... https://people.freebsd.org/~rmacklem/nfsd-vnet-prison-setup.txt Let me know if you test it or have other suggestions, rick ps: Thanks everyone for your comments. If I have specific questions related to them, I'll post. Otherwise I am digesting them. --0000000000006e6e4805ee919271 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"fon= t-family:monospace"><br></div></div><br><div class=3D"gmail_quote"><div dir= =3D"ltr" class=3D"gmail_attr">On Fri, Nov 25, 2022 at 9:06 PM Alan Somers &= lt;<a href=3D"mailto:asomers@freebsd.org">asomers@freebsd.org</a>> wrote= :<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.= 8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"au= to"><div><br><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail= _attr">On Fri, Nov 25, 2022, 4:24 PM Rick Macklem <<a href=3D"mailto:ric= k.macklem@gmail.com" target=3D"_blank">rick.macklem@gmail.com</a>> wrote= :<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.= 8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"lt= r"><div style=3D"font-family:monospace">Hi,</div><div style=3D"font-family:= monospace"><br></div><div style=3D"font-family:monospace">bz@ has encourage= d me to fiddle with the nfsd</div><div style=3D"font-family:monospace">so t= hat it works in a vnet jail.</div><div style=3D"font-family:monospace">I ha= ve now basically done so, specifically for</div><div style=3D"font-family:m= onospace">NFSv4, since NFSv3 presents various issues.</div><div style=3D"fo= nt-family:monospace"><br></div><div style=3D"font-family:monospace">What I = have not yet done is put global variables</div><div style=3D"font-family:mo= nospace">in the vnet. This needs to be done so that the nfsd</div><div styl= e=3D"font-family:monospace">can be run in multiple jail instances and/or in= and</div><div style=3D"font-family:monospace">outside of a jail.</div><div= style=3D"font-family:monospace">The problem is that there are 100s of glob= al variables.</div><div style=3D"font-family:monospace"><br></div><div styl= e=3D"font-family:monospace">I can see two approaches:</div><div style=3D"fo= nt-family:monospace">1 - Move them all into the vnet jail. This would imply= </div><div style=3D"font-family:monospace">=C2=A0 =C2=A0 that all the sysct= ls need to somehow be changed,</div><div style=3D"font-family:monospace">= =C2=A0 =C2=A0 which would seem to be a POLA violation.</div><div style=3D"f= ont-family:monospace">=C2=A0 =C2=A0 It also implies a lot of stuff in the v= net.</div><div style=3D"font-family:monospace">2 - Just move the global var= iables that will always</div><div style=3D"font-family:monospace">=C2=A0 = =C2=A0 differ from one nfsd to another (this would make</div><div style=3D"= font-family:monospace">=C2=A0 =C2=A0 the sysctls global and apply to all nf= sds).</div><div style=3D"font-family:monospace">=C2=A0 =C2=A0 This will kee= p the number of globals in the vnet</div><div style=3D"font-family:monospac= e">=C2=A0 =C2=A0 smaller.</div><div style=3D"font-family:monospace"><br></d= iv><div style=3D"font-family:monospace">I am currently leaning towards #2, = put what do others</div><div style=3D"font-family:monospace">think?</div><d= iv style=3D"font-family:monospace"><br></div><div style=3D"font-family:mono= space">rick</div><div style=3D"font-family:monospace">ps: Personally, I don= 't know what use there is of</div><div style=3D"font-family:monospace">= =C2=A0 =C2=A0 running the nfsd inside a vnet jail, but bz@ has</div><div st= yle=3D"font-family:monospace">=C2=A0 =C2=A0 some use case.</div><div style= =3D"font-family:monospace"></div></div></blockquote></div></div><div dir=3D= "auto"><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding= -left:1ex"><div dir=3D"ltr"><div style=3D"font-family:monospace"></div></di= v></blockquote></div></div><div dir=3D"auto"><br></div><div dir=3D"auto">Th= is is super-awesome! Thank you so much! I've got a use case too.=C2=A0 = I think it would be fine to leave most of the settings global,=C2=A0 like m= ax_threads. But we should probably decide on a case by case basis .</div></= div></blockquote><div><span class=3D"gmail_default" style=3D"font-family:mo= nospace">The minthreads, maxthreads happen to be handled via nfsd command l= ine options, so</span></div><div><span class=3D"gmail_default" style=3D"fon= t-family:monospace">the sysctls are not needed and they can be set per-pris= on.</span></div><div><span class=3D"gmail_default" style=3D"font-family:mon= ospace">Most of the sysctls are for weird cases or tuning of the DRC. Since= the DRC is</span></div><div><span class=3D"gmail_default" style=3D"font-fa= mily:monospace">only used for NFSv4.0 mounts and not NFSv4.1 or NFSv4.2 one= s, tuning the DRC</span></div><div><span class=3D"gmail_default" style=3D"f= ont-family:monospace">should not usually be necessary.</span></div><div><sp= an class=3D"gmail_default" style=3D"font-family:monospace"><br></span></div= ><div><span class=3D"gmail_default" style=3D"font-family:monospace">I have = left them global for now.</span></div><div><span class=3D"gmail_default" st= yle=3D"font-family:monospace"><br></span></div><div><span class=3D"gmail_de= fault" style=3D"font-family:monospace">If anyone identifies one that needs = to be set per-prison, I can move it into</span></div><div><span class=3D"gm= ail_default" style=3D"font-family:monospace">the vnet.</span></div><div><sp= an class=3D"gmail_default" style=3D"font-family:monospace">If you want to s= ee them all:</span></div><div><span class=3D"gmail_default" style=3D"font-f= amily:monospace"># sysctl -a | fgrep vfs.nfsd</span></div><div><span class= =3D"gmail_default" style=3D"font-family:monospace"><br></span></div><div><s= pan class=3D"gmail_default" style=3D"font-family:monospace">I have put a fi= rst patch up on phabricator as D37519. Although I listed three</span></div>= <div><span class=3D"gmail_default" style=3D"font-family:monospace">people a= s reviewers, anyone is welcome to test/comment/review.</span>=C2=A0</div><d= iv><div class=3D"gmail_default" style=3D"font-family:monospace">If you can&= #39;t easily get the patch from phabricator, just email me and I'll</di= v><div class=3D"gmail_default" style=3D"font-family:monospace">send it to y= ou. I think it will apply cleanly to main and, maybe, stable/13.</div><div = class=3D"gmail_default" style=3D"font-family:monospace">You only need to bu= ild a kernel from patched sources to test it. There is a</div><div class=3D= "gmail_default" style=3D"font-family:monospace">change to rc.d/nfsd, which = you only need in the prison's etc/rc.d/nfsd.</div><div class=3D"gmail_d= efault" style=3D"font-family:monospace"><br></div><div class=3D"gmail_defau= lt" style=3D"font-family:monospace">A very basic setup document (also defin= itely a work in progress) can be</div><div class=3D"gmail_default" style=3D= "font-family:monospace">found at...</div><div class=3D"gmail_default" style= =3D"font-family:monospace"><a href=3D"https://people.freebsd.org/~rmacklem/= nfsd-vnet-prison-setup.txt">https://people.freebsd.org/~rmacklem/nfsd-vnet-= prison-setup.txt</a></div><div class=3D"gmail_default" style=3D"font-family= :monospace"><br></div><div class=3D"gmail_default" style=3D"font-family:mon= ospace">Let me know if you test it or have other suggestions, rick</div><di= v class=3D"gmail_default" style=3D"font-family:monospace">ps: Thanks everyo= ne for your comments. If I have specific questions related</div><div class= =3D"gmail_default" style=3D"font-family:monospace">=C2=A0 =C2=A0 to them, I= 'll post. Otherwise I am digesting them.</div><div class=3D"gmail_defau= lt" style=3D"font-family:monospace"></div><br></div></div></div> --0000000000006e6e4805ee919271--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAM5tNy7X81VVmYmWoAgrUoyjo96yqHb2-ZTys5D6ROQgvcoYBQ>