Date: Mon, 28 Nov 2022 17:04:24 -0800 From: Rick Macklem <rick.macklem@gmail.com> To: Alan Somers <asomers@freebsd.org> Cc: FreeBSD CURRENT <freebsd-current@freebsd.org>, "Bjoern A. Zeeb" <bz@freebsd.org> Subject: Re: RFC: nfsd in a vnet jail Message-ID: <CAM5tNy7X81VVmYmWoAgrUoyjo96yqHb2-ZTys5D6ROQgvcoYBQ@mail.gmail.com> In-Reply-To: <CAOtMX2hxeeNMxxdpma8NJ7ms60eRfuCWoFi7FixdSe83=qibkA@mail.gmail.com> References: <CAM5tNy7CQaBTRWG0m0aN6T0xG2L2zSQJGa%2BatGaH%2BmW%2BwEpdyQ@mail.gmail.com> <CAOtMX2hxeeNMxxdpma8NJ7ms60eRfuCWoFi7FixdSe83=qibkA@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] On Fri, Nov 25, 2022 at 9:06 PM Alan Somers <asomers@freebsd.org> wrote: > > > On Fri, Nov 25, 2022, 4:24 PM Rick Macklem <rick.macklem@gmail.com> wrote: > >> Hi, >> >> bz@ has encouraged me to fiddle with the nfsd >> so that it works in a vnet jail. >> I have now basically done so, specifically for >> NFSv4, since NFSv3 presents various issues. >> >> What I have not yet done is put global variables >> in the vnet. This needs to be done so that the nfsd >> can be run in multiple jail instances and/or in and >> outside of a jail. >> The problem is that there are 100s of global variables. >> >> I can see two approaches: >> 1 - Move them all into the vnet jail. This would imply >> that all the sysctls need to somehow be changed, >> which would seem to be a POLA violation. >> It also implies a lot of stuff in the vnet. >> 2 - Just move the global variables that will always >> differ from one nfsd to another (this would make >> the sysctls global and apply to all nfsds). >> This will keep the number of globals in the vnet >> smaller. >> >> I am currently leaning towards #2, put what do others >> think? >> >> rick >> ps: Personally, I don't know what use there is of >> running the nfsd inside a vnet jail, but bz@ has >> some use case. >> > > This is super-awesome! Thank you so much! I've got a use case too. I > think it would be fine to leave most of the settings global, like > max_threads. But we should probably decide on a case by case basis . > The minthreads, maxthreads happen to be handled via nfsd command line options, so the sysctls are not needed and they can be set per-prison. Most of the sysctls are for weird cases or tuning of the DRC. Since the DRC is only used for NFSv4.0 mounts and not NFSv4.1 or NFSv4.2 ones, tuning the DRC should not usually be necessary. I have left them global for now. If anyone identifies one that needs to be set per-prison, I can move it into the vnet. If you want to see them all: # sysctl -a | fgrep vfs.nfsd I have put a first patch up on phabricator as D37519. Although I listed three people as reviewers, anyone is welcome to test/comment/review. If you can't easily get the patch from phabricator, just email me and I'll send it to you. I think it will apply cleanly to main and, maybe, stable/13. You only need to build a kernel from patched sources to test it. There is a change to rc.d/nfsd, which you only need in the prison's etc/rc.d/nfsd. A very basic setup document (also definitely a work in progress) can be found at... https://people.freebsd.org/~rmacklem/nfsd-vnet-prison-setup.txt Let me know if you test it or have other suggestions, rick ps: Thanks everyone for your comments. If I have specific questions related to them, I'll post. Otherwise I am digesting them. [-- Attachment #2 --] <div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:monospace"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 25, 2022 at 9:06 PM Alan Somers <<a href="mailto:asomers@freebsd.org">asomers@freebsd.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 25, 2022, 4:24 PM Rick Macklem <<a href="mailto:rick.macklem@gmail.com" target="_blank">rick.macklem@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:monospace">Hi,</div><div style="font-family:monospace"><br></div><div style="font-family:monospace">bz@ has encouraged me to fiddle with the nfsd</div><div style="font-family:monospace">so that it works in a vnet jail.</div><div style="font-family:monospace">I have now basically done so, specifically for</div><div style="font-family:monospace">NFSv4, since NFSv3 presents various issues.</div><div style="font-family:monospace"><br></div><div style="font-family:monospace">What I have not yet done is put global variables</div><div style="font-family:monospace">in the vnet. This needs to be done so that the nfsd</div><div style="font-family:monospace">can be run in multiple jail instances and/or in and</div><div style="font-family:monospace">outside of a jail.</div><div style="font-family:monospace">The problem is that there are 100s of global variables.</div><div style="font-family:monospace"><br></div><div style="font-family:monospace">I can see two approaches:</div><div style="font-family:monospace">1 - Move them all into the vnet jail. This would imply</div><div style="font-family:monospace"> that all the sysctls need to somehow be changed,</div><div style="font-family:monospace"> which would seem to be a POLA violation.</div><div style="font-family:monospace"> It also implies a lot of stuff in the vnet.</div><div style="font-family:monospace">2 - Just move the global variables that will always</div><div style="font-family:monospace"> differ from one nfsd to another (this would make</div><div style="font-family:monospace"> the sysctls global and apply to all nfsds).</div><div style="font-family:monospace"> This will keep the number of globals in the vnet</div><div style="font-family:monospace"> smaller.</div><div style="font-family:monospace"><br></div><div style="font-family:monospace">I am currently leaning towards #2, put what do others</div><div style="font-family:monospace">think?</div><div style="font-family:monospace"><br></div><div style="font-family:monospace">rick</div><div style="font-family:monospace">ps: Personally, I don't know what use there is of</div><div style="font-family:monospace"> running the nfsd inside a vnet jail, but bz@ has</div><div style="font-family:monospace"> some use case.</div><div style="font-family:monospace"></div></div></blockquote></div></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:monospace"></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">This is super-awesome! Thank you so much! I've got a use case too. I think it would be fine to leave most of the settings global, like max_threads. But we should probably decide on a case by case basis .</div></div></blockquote><div><span class="gmail_default" style="font-family:monospace">The minthreads, maxthreads happen to be handled via nfsd command line options, so</span></div><div><span class="gmail_default" style="font-family:monospace">the sysctls are not needed and they can be set per-prison.</span></div><div><span class="gmail_default" style="font-family:monospace">Most of the sysctls are for weird cases or tuning of the DRC. Since the DRC is</span></div><div><span class="gmail_default" style="font-family:monospace">only used for NFSv4.0 mounts and not NFSv4.1 or NFSv4.2 ones, tuning the DRC</span></div><div><span class="gmail_default" style="font-family:monospace">should not usually be necessary.</span></div><div><span class="gmail_default" style="font-family:monospace"><br></span></div><div><span class="gmail_default" style="font-family:monospace">I have left them global for now.</span></div><div><span class="gmail_default" style="font-family:monospace"><br></span></div><div><span class="gmail_default" style="font-family:monospace">If anyone identifies one that needs to be set per-prison, I can move it into</span></div><div><span class="gmail_default" style="font-family:monospace">the vnet.</span></div><div><span class="gmail_default" style="font-family:monospace">If you want to see them all:</span></div><div><span class="gmail_default" style="font-family:monospace"># sysctl -a | fgrep vfs.nfsd</span></div><div><span class="gmail_default" style="font-family:monospace"><br></span></div><div><span class="gmail_default" style="font-family:monospace">I have put a first patch up on phabricator as D37519. Although I listed three</span></div><div><span class="gmail_default" style="font-family:monospace">people as reviewers, anyone is welcome to test/comment/review.</span> </div><div><div class="gmail_default" style="font-family:monospace">If you can't easily get the patch from phabricator, just email me and I'll</div><div class="gmail_default" style="font-family:monospace">send it to you. I think it will apply cleanly to main and, maybe, stable/13.</div><div class="gmail_default" style="font-family:monospace">You only need to build a kernel from patched sources to test it. There is a</div><div class="gmail_default" style="font-family:monospace">change to rc.d/nfsd, which you only need in the prison's etc/rc.d/nfsd.</div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace">A very basic setup document (also definitely a work in progress) can be</div><div class="gmail_default" style="font-family:monospace">found at...</div><div class="gmail_default" style="font-family:monospace"><a href="https://people.freebsd.org/~rmacklem/nfsd-vnet-prison-setup.txt">https://people.freebsd.org/~rmacklem/nfsd-vnet-prison-setup.txt</a></div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace">Let me know if you test it or have other suggestions, rick</div><div class="gmail_default" style="font-family:monospace">ps: Thanks everyone for your comments. If I have specific questions related</div><div class="gmail_default" style="font-family:monospace"> to them, I'll post. Otherwise I am digesting them.</div><div class="gmail_default" style="font-family:monospace"></div><br></div></div></div>home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAM5tNy7X81VVmYmWoAgrUoyjo96yqHb2-ZTys5D6ROQgvcoYBQ>