Date: Fri, 26 Aug 2022 22:15:55 -0600 From: Warner Losh <imp@bsdimp.com> To: Peter Jeremy <peterj@freebsd.org>, Kyle Evans <kevans@freebsd.org> Cc: FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: ZFS bootfs vs vfs.root.mountfrom Message-ID: <CANCZdfq8s%2BmRc-roH0LcMiOgYXs2ri3S03P_uoU6C6ctXKxoUg@mail.gmail.com> In-Reply-To: <YwmXBSn8W4QfkRVR@server.rulingia.com> References: <YwmXBSn8W4QfkRVR@server.rulingia.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000023f0ef05e7314a51 Content-Type: text/plain; charset="UTF-8" On Fri, Aug 26, 2022 at 10:01 PM Peter Jeremy <peterj@freebsd.org> wrote: > When booting from ZFS was first introduced, the recommended approach > was to configure the root filesystem via vfs.root.mountfrom. See (e.g.) > https://lists.freebsd.org/pipermail/freebsd-fs/2011-September/012482.html > https://lists.freebsd.org/pipermail/svn-src-head/2011-October/030641.html > and people wanted to change that to use the zpool bootfs property - e.g. > > https://lists.freebsd.org/pipermail/freebsd-current/2009-October/012933.html > https://lists.freebsd.org/pipermail/freebsd-fs/2010-March/008010.html > resulting in it becoming optional in SVN r235330 in May 2012: > https://lists.freebsd.org/pipermail/svn-src-head/2012-May/036902.html Yes. This was a total hack at the time. The boot loader normally sets this variable w/o users intervening and this hack propagated into loader.conf files, which has always been a bad idea. The recommendation to use vfs.root.mountfrom remains in parts of the > wiki (e.g. https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror step > 2.6) and on mailing list posts (e.g. the thread starting at > https://lists.freebsd.org/pipermail/freebsd-fs/2020-July/028351.html). > The loader(8) and loader.conf(5) man pages state that the root > filesystem is set by vfs.root.mountfrom, with a default derived from > /etc/fstab or, for ZFS only, currdev (the boot filesystem). > Yes. That's how it is supposed to work. The boot loader sets this based on where it loaded the kernel. This variable should *NEVER* be set in a loader.conf file unless you have a 'very special snowflake' system that can't determine it automatically. ZFS root isn't special. > OTOH, https://wiki.freebsd.org/BootEnvironments states "/etc/fstab > must be purged of any / mount, and /boot/loader.conf must not be > setting vfs.root.mountfrom directly." I can't find any other > reference to this requirement. > /etc/fstab isn't used for zfs root. I believe that BootEnvironments is 100% right here. > As for the BootEnvironment management tools themselves: > * Neither beadm(8) nor bectl(8) mention vfs.root.mountfrom. > This is correct. > * beadm(8) refers to https://forums.freebsd.org/showthread.php?t=31662, > which recommends setting vfs.root.mountfrom > This is bad advice. > * beadm(8) will update vfs.root.mountfrom in /boot/loader.conf if it > exists but will not add it if it doesn't exist. > This is terrible. It's part of the continuing papering over of bugs that have long ago been fixed and should, at most, warn that's its set and that it really shouldn't be. > * bectl(8) ignores /boot/loader.conf > This is proper behavior. > The last point means that switching an existing system from beadm to > bectl is very likely to result in the system mounting the wrong root > filesystem: An existing system is likely to have vfs.root.mountfrom > set, since that used to be the only approach and is still widely > recommended, whilst bectl ignores it and will therefore leave the > old vfs.root.mountfrom value present. > Yes. bectl likely should, at most, warn that it's set in loader.conf as an antifootshooting measure, but only due to the all-too-long history. > IMO, if setting vfs.root.mountfrom in /boot/loader.conf is no longer > supported for ZFS, we need to do a better job of documenting that and > removing documentation to the contrary. I also believe that bectl > should at least warn if /boot/loader.conf sets vfs.root.mountfrom. > It really never had been the proper way. One should almost never do it today, except in the above 'special snowflake' systems that still crop up from time to time (we have one at work since we still have some systems that use gmirror roots, and the boot loader doesn't set it right for them since we can't for logistical reasons set /etc/fstab, but that's not ZFS related). We should actively militate against it with extreme prejudice, though, for routine uses. We should rip out all references in the docs related to ZFS. We should warn if we detect it today. I agree completely with what you are saying here. I would be extraordinarily surprised to find anybody with a contrary opinion. Now, who is going to do the work? :) Warner --00000000000023f0ef05e7314a51 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Fri, Aug 26, 2022 at 10:01 PM Pete= r Jeremy <<a href=3D"mailto:peterj@freebsd.org">peterj@freebsd.org</a>&g= t; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0p= x 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">When b= ooting from ZFS was first introduced, the recommended approach<br> was to configure the root filesystem via vfs.root.mountfrom.=C2=A0 See (e.g= .)<br> =C2=A0<a href=3D"https://lists.freebsd.org/pipermail/freebsd-fs/2011-Septem= ber/012482.html" rel=3D"noreferrer" target=3D"_blank">https://lists.freebsd= .org/pipermail/freebsd-fs/2011-September/012482.html</a><br> =C2=A0<a href=3D"https://lists.freebsd.org/pipermail/svn-src-head/2011-Octo= ber/030641.html" rel=3D"noreferrer" target=3D"_blank">https://lists.freebsd= .org/pipermail/svn-src-head/2011-October/030641.html</a><br> and people wanted to change that to use the zpool bootfs property - e.g.<br= > =C2=A0<a href=3D"https://lists.freebsd.org/pipermail/freebsd-current/2009-O= ctober/012933.html" rel=3D"noreferrer" target=3D"_blank">https://lists.free= bsd.org/pipermail/freebsd-current/2009-October/012933.html</a><br> =C2=A0<a href=3D"https://lists.freebsd.org/pipermail/freebsd-fs/2010-March/= 008010.html" rel=3D"noreferrer" target=3D"_blank">https://lists.freebsd.org= /pipermail/freebsd-fs/2010-March/008010.html</a><br> resulting in it becoming optional in SVN r235330 in May 2012:<br> =C2=A0<a href=3D"https://lists.freebsd.org/pipermail/svn-src-head/2012-May/= 036902.html" rel=3D"noreferrer" target=3D"_blank">https://lists.freebsd.org= /pipermail/svn-src-head/2012-May/036902.html</a></blockquote><div><br></div= ><div>Yes. This was a total hack at the time. The boot loader normally sets= this variable</div><div>w/o users intervening and this hack propagated=C2= =A0into loader.conf files, which has</div><div>always been a bad idea.</div= ><div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0= px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> The recommendation to use vfs.root.mountfrom remains in parts of the<br> wiki (e.g. <a href=3D"https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror"= rel=3D"noreferrer" target=3D"_blank">https://wiki.freebsd.org/RootOnZFS/GP= TZFSBoot/Mirror</a> step<br> 2.6) and on mailing list posts (e.g. the thread starting at<br> <a href=3D"https://lists.freebsd.org/pipermail/freebsd-fs/2020-July/028351.= html" rel=3D"noreferrer" target=3D"_blank">https://lists.freebsd.org/piperm= ail/freebsd-fs/2020-July/028351.html</a>).<br> The loader(8) and loader.conf(5) man pages state that the root<br> filesystem is set by vfs.root.mountfrom, with a default derived from<br> /etc/fstab or, for ZFS only, currdev (the boot filesystem).<br></blockquote= ><div><br></div><div>Yes. That's how it is supposed to work. The boot l= oader sets this based on</div><div>where it loaded the kernel. This variabl= e should *NEVER* be set in a</div><div>loader.conf file unless you have a &= #39;very special snowflake' system that</div><div>can't determine i= t automatically. ZFS root isn't special.</div><div>=C2=A0</div><blockqu= ote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px= solid rgb(204,204,204);padding-left:1ex"> OTOH, <a href=3D"https://wiki.freebsd.org/BootEnvironments" rel=3D"noreferr= er" target=3D"_blank">https://wiki.freebsd.org/BootEnvironments</a>; states = "/etc/fstab<br> must be purged of any / mount, and /boot/loader.conf must not be<br> setting vfs.root.mountfrom directly."=C2=A0 I can't find any other= <br> reference to this requirement.<br></blockquote><div><br></div><div>/etc/fst= ab isn't used for zfs root. I believe that BootEnvironments is 100%</di= v><div>right here.</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" = style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pa= dding-left:1ex"> As for the BootEnvironment management tools themselves:<br> * Neither beadm(8) nor bectl(8) mention vfs.root.mountfrom.<br></blockquote= ><div><br></div><div>This is correct.</div><div>=C2=A0</div><blockquote cla= ss=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex"> * beadm(8) refers to <a href=3D"https://forums.freebsd.org/showthread.php?t= =3D31662" rel=3D"noreferrer" target=3D"_blank">https://forums.freebsd.org/s= howthread.php?t=3D31662</a>,<br> =C2=A0 which recommends setting vfs.root.mountfrom<br></blockquote><div><br= ></div><div>This is bad advice.</div><div>=C2=A0</div><blockquote class=3D"= gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(20= 4,204,204);padding-left:1ex"> * beadm(8) will update vfs.root.mountfrom in /boot/loader.conf if it<br> =C2=A0 exists but will not add it if it doesn't exist.<br></blockquote>= <div><br></div><div>This is terrible. It's part of the continuing paper= ing over of bugs that have</div><div>long ago been fixed and should, at mos= t, warn that's its set and that it</div><div>really shouldn't be.</= div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> * bectl(8) ignores /boot/loader.conf<br></blockquote><div><br></div><div>Th= is is proper behavior.</div><div>=C2=A0</div><blockquote class=3D"gmail_quo= te" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204= );padding-left:1ex"> The last point means that switching an existing system from beadm to<br> bectl is very likely to result in the system mounting the wrong root<br> filesystem: An existing system is likely to have vfs.root.mountfrom<br> set, since that used to be the only approach and is still widely<br> recommended, whilst bectl ignores it and will therefore leave the<br> old vfs.root.mountfrom value present.<br></blockquote><div><br></div><div>Y= es. bectl=C2=A0likely should, at most, warn that it's set in loader.con= f as an</div><div>antifootshooting=C2=A0measure, but only due to the all-to= o-long history.</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" sty= le=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddi= ng-left:1ex"> IMO, if setting vfs.root.mountfrom in /boot/loader.conf is no longer<br> supported for ZFS, we need to do a better job of documenting that and<br> removing documentation to the contrary.=C2=A0 I also believe that bectl<br> should at least warn if /boot/loader.conf sets vfs.root.mountfrom.<br></blo= ckquote><div><br></div><div>It really never had been the proper way. One sh= ould almost never do</div><div>it today, except in the above 'special s= nowflake' systems that=C2=A0still crop up</div><div>from time to time (= we have one at work since we still have=C2=A0some systems</div><div>that us= e=C2=A0gmirror roots,=C2=A0and the boot loader doesn't set it right for= them since</div><div>we can't for logistical reasons set /etc/fstab,= =C2=A0but that's not ZFS related). We</div><div>should actively militat= e against it with extreme prejudice,=C2=A0though,=C2=A0for routine</div><di= v>uses. We should rip out all references in the docs related to ZFS. We sho= uld</div><div>warn if we detect it today. I agree completely with what you = are saying here.</div><div>I would be extraordinarily surprised to find any= body with a contrary opinion.</div><div><br></div><div>Now, who is going to= do the work? :)</div><div><br></div><div>Warner</div></div></div> --00000000000023f0ef05e7314a51--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfq8s%2BmRc-roH0LcMiOgYXs2ri3S03P_uoU6C6ctXKxoUg>