Date: Mon, 6 Nov 2023 21:45:57 -0600 From: Chris Watson <bsdunix44@gmail.com> To: DtxdF <DtxdF@disroot.org> Cc: freebsd-jail@freebsd.org Subject: Re: I can get zfs snapshot/rollback in a jail to work 99% but it isn't quite 100% working. What am I missing? Message-ID: <CAHnbxSSra=JQ0iqne0NBomrexSxu8ar7ZViq7tfKVDsO7vczAg@mail.gmail.com> In-Reply-To: <A696936B-D83F-4B7B-B037-F51CF47C44A9@disroot.org> References: <CAHnbxSRuwGwdB0PULA4LEKTWTJw6oA6VeX_7PL4TMgAv_YS=vw@mail.gmail.com> <A696936B-D83F-4B7B-B037-F51CF47C44A9@disroot.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000958012060987cfc3
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
So as I mentioned I=E2=80=99ve able to mail the dataset. It gets mounted up=
on
starting the jail. It shows up in =E2=80=9Czfs list=E2=80=9D. And when I do=
zfs snapshot on
the dataset it appears to create the snapshot as it shows up in a =E2=80=9C=
zfs list
-t snapshot=E2=80=9D but the snapdir isn=E2=80=99t visible even after setti=
ng snapdir to
visible, and when I rollback using the snapshot it doesn=E2=80=99t actually
rollback. I=E2=80=99m so close to this working, something just isn=E2=80=99=
t right and I
can=E2=80=99t figure out what. I really wish this was written up in the han=
dbook.
=E2=80=9CHow to jail a dataset from the host=E2=80=9D and =E2=80=9Chow to s=
napshot and rollback a
jailed dataset=E2=80=9D. :) if I figure this I=E2=80=99ll definitely be wri=
ting this up.
Chris
On Mon, Nov 6, 2023 at 2:35 PM DtxdF <DtxdF@disroot.org> wrote:
> Hi Chris,
>
> Maybe your dataset is not mounted inside the jail. I thought that simply
> enabling `/etc/rc.d/zfs` was fine, but no, it just doesn't work. I don't
> know if this behavior is a bug or something else, but at the moment I don=
't
> have time to investigate.
>
> I have a similar setup for a jail with a delegated dataset. I use AppJail=
,
> but the steps are similar to other tools:
>
> ```
> # zfs create -o jailed=3Don -o mountpoint=3D/jailed zroot/jailed
> # appjail quick jtest \
> mount_devfs \
> device=3D'include $devfsrules_hide_all' \
> device=3D'include $devfsrules_unhide_basic' \
> device=3D'include $devfsrules_unhide_login' \
> device=3D'path zfs unhide' \
> template=3Dtemplate.conf \
> overwrite=3Dforce \
> start
> ```
>
> In AppJail, a template configuration file is similar to `jail.conf(5)`:
>
> ```
> # cat template.conf
> exec.start: "/bin/sh /etc/rc"
> exec.stop: "/bin/sh /etc/rc.shutdown jail"
> allow.mount
> allow.mount.zfs
> enforce_statfs: 1
> exec.poststart: "zfs jail ${name} zroot/jailed"
> exec.poststart+: "appjail cmd jexec ${name} zfs mount zroot/jailed"
> exec.prestop: "appjail cmd jexec ${name} zfs umount zroot/jailed"
> exec.prestop+: "zfs unjail ${name} zroot/jailed"
> ```
>
> As you can see, the dataset is mounted after running `zfs-jail(8)`. The
> steps are similar when the jail is stopped, but the dataset is unmounted
> and `zfs-unjail(8)` is executed.
>
> Inside the jail I can see the mounted datasets:
>
> ```
> # appjail cmd jexec jtest zfs list -r
> NAME USED AVAIL REFER MOUNTPOINT
> zroot 34.1G 249G 96K /zroot
> zroot/jailed 96K 249G 96K /jailed
> # appjail cmd jexec jtest mount -t zfs
> zroot/appjail/jails/jtest/jail on / (zfs, local, noatime, nfsv4acls)
> zroot/jailed on /jailed (zfs, local, noatime, nfsv4acls)
> # appjail cmd jexec jtest ls /jailed
> index.txt
> # appjail cmd jexec jtest cat /jailed/index.txt
> Hi!
> ```
>
> And I can use `zfs-rollback(8)` just fine:
>
> ```
> # appjail cmd jexec jtest zfs snapshot zroot/jailed@guard
> # appjail cmd jexec jtest zfs list -t snapshot zroot/jailed
> NAME USED AVAIL REFER MOUNTPOINT
> zroot/jailed@guard 0B - 96K -
> # appjail cmd jexec jtest dd if=3D/dev/random of=3D/jailed/index.txt bs=
=3D16
> count=3D1
> 1+0 records in
> 1+0 records out
> 16 bytes transferred in 0.000102 secs (157318 bytes/sec)
> # appjail cmd jexec jtest hd /jailed/index.txt
> 00000000 a1 26 2a 9c f5 96 7b 81 90 8d ba 36 d6 f9 4d 93
> |.&*...{....6..M.|
> 00000010
> # appjail cmd jexec jtest zfs list -t snapshot zroot/jailed
> NAME USED AVAIL REFER MOUNTPOINT
> zroot/jailed@guard 56K - 96K -
> # appjail cmd jexec jtest zfs rollback zroot/jailed@guard
> # appjail cmd jexec jtest hd /jailed/index.txt
> 00000000 48 69 21 0a |Hi!.|
> 00000004
> ```
>
> I hope this can help you.
>
>
> ~ DtxdF
>
>
> El 6 de noviembre de 2023 6:07:06 p. m. UTC, Chris Watson <
> bsdunix44@gmail.com> escribi=C3=B3:
>
>> I've been trying to get a zfs dataset delegated into a jail (to run PG
>> on), and allow snapshots and rollback to take place inside the jail. I c=
an
>> get the dataset mounted into the jail, I can get zfs to take the snapsho=
t,
>> list the snapshot, but when I rollback or try to ls -la the directory to
>> see the '.zfs' dir it isn't there and the zfs rollback completes but it
>> doesn't actually rollback. I'm so close to getting this to work! I'm jus=
t
>> missing *something* in the sauce. When I do the zfs rollback zfs looks l=
ike
>> it completes the rollback and goes back to a shell prompt but the files =
I
>> remove before the rollback are not in the /var/db/postgres/data16 direct=
ory
>> nor is ".zfs" shown in ls -la. So something is wonky on my end. I'm so
>> close, it's halfway there, it looks like it takes a snapshot, the snapsh=
ot
>> shows up in a zfs list -t snapshot, but it's also not really there. I'm
>> doing something just slightly wrong here. I just cant figure out what I
>> have wrong.
>>
>> Below are the configs:
>> # The jail's config
>> https://bsd.to/P176
>> # zfs list from inside the jail
>> https://bsd.to/mPde
>> # zfs list -t snapshot from inside the jail
>> https://bsd.to/R8dw
>> # ls -la /var/db/postgres/data16 output from inside the jail
>> https://bsd.to/1di2
>> # rc.conf of the jail
>> https://bsd.to/JcnH
>>
>> The jail is running 13.2-P4.
>> Using bastillebsd 0.10.20231013 for creation/management.
>>
>> Thanks!
>> Chris
>>
>
--000000000000958012060987cfc3
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"auto">So as I mentioned I=E2=80=99ve able to mail the dataset. =
It gets mounted upon starting the jail. It shows up in =E2=80=9Czfs list=E2=
=80=9D. And when I do zfs snapshot on the dataset it appears to create the =
snapshot as it shows up in a =E2=80=9Czfs list -t snapshot=E2=80=9D but the=
snapdir isn=E2=80=99t visible even after setting snapdir to visible, and w=
hen I rollback using the snapshot it doesn=E2=80=99t actually rollback. I=
=E2=80=99m so close to this working, something just isn=E2=80=99t right and=
I can=E2=80=99t figure out what. I really wish this was written up in the =
handbook. =E2=80=9CHow to jail a dataset from the host=E2=80=9D and =E2=80=
=9Chow to snapshot and rollback a jailed dataset=E2=80=9D. :) if I figure t=
his I=E2=80=99ll definitely be writing this up.=C2=A0</div><div dir=3D"auto=
"><br></div><div dir=3D"auto">Chris=C2=A0</div><div><br><div class=3D"gmail=
_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Nov 6, 2023 at 2:35 P=
M DtxdF <<a href=3D"mailto:DtxdF@disroot.org">DtxdF@disroot.org</a>> =
wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0=
px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;bor=
der-left-color:rgb(204,204,204)"><div><div dir=3D"auto">Hi Chris,<br><br>Ma=
ybe your dataset is not mounted inside the jail. I thought that simply enab=
ling `/etc/rc.d/zfs` was fine, but no, it just doesn't work. I don'=
t know if this behavior is a bug or something else, but at the moment I don=
't have time to investigate.<br><br>I have a similar setup for a jail w=
ith a delegated dataset. I use AppJail, but the steps are similar to other =
tools:<br><br>```<br># zfs create -o jailed=3Don -o mountpoint=3D/jailed zr=
oot/jailed<br># appjail quick jtest \<br> mount_devfs \<br> device=3D'i=
nclude $devfsrules_hide_all' \<br> device=3D'include $devfsrules_un=
hide_basic' \<br> device=3D'include $devfsrules_unhide_login' \=
<br> device=3D'path zfs unhide' \<br> template=3Dtemplate.conf \<br=
> overwrite=3Dforce \<br> start<br>```<br><br>In AppJail, a template config=
uration file is similar to `jail.conf(5)`:<br><br>```<br># cat template.con=
f<br>exec.start: "/bin/sh /etc/rc"<br>exec.stop: "/bin/sh /e=
tc/rc.shutdown jail"<br>allow.mount<br>allow.mount.zfs<br>enforce_stat=
fs: 1<br>exec.poststart: "zfs jail ${name} zroot/jailed"<br>exec.=
poststart+: "appjail cmd jexec ${name} zfs mount zroot/jailed"<br=
>exec.prestop: "appjail cmd jexec ${name} zfs umount zroot/jailed"=
;<br>exec.prestop+: "zfs unjail ${name} zroot/jailed"<br>```<br><=
br>As you can see, the dataset is mounted after running `zfs-jail(8)`. The =
steps are similar when the jail is stopped, but the dataset is unmounted an=
d `zfs-unjail(8)` is executed.<br><br>Inside the jail I can see the mounted=
datasets:<br><br>```<br># appjail cmd jexec jtest zfs list -r<br>NAME=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 USED=C2=A0 AVAIL=
=C2=A0=C2=A0=C2=A0=C2=A0 REFER=C2=A0 MOUNTPOINT<br>zroot=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 34.1G=C2=A0=C2=A0 249G=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 96K=C2=A0 /zroot<br>zroot/jailed=C2=A0=C2=A0=C2=A0 96K=C2=
=A0=C2=A0 249G=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 96K=C2=A0 /jailed<br># a=
ppjail cmd jexec jtest mount -t zfs<br>zroot/appjail/jails/jtest/jail on / =
(zfs, local, noatime, nfsv4acls)<br>zroot/jailed on /jailed (zfs, local, no=
atime, nfsv4acls)<br># appjail cmd jexec jtest ls /jailed<br>index.txt<br>#=
appjail cmd jexec jtest cat /jailed/index.txt<br>Hi!<br>```<br><br>And I c=
an use `zfs-rollback(8)` just fine:<br><br>```<br># appjail cmd jexec jtest=
zfs snapshot zroot/jailed@guard<br># appjail cmd jexec jtest zfs list -t s=
napshot zroot/jailed<br>NAME=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 USED=C2=A0 AVAIL=C2=A0=
=C2=A0=C2=A0=C2=A0 REFER=C2=A0 MOUNTPOINT<br>zroot/jailed@guard=C2=A0=C2=A0=
=C2=A0=C2=A0 0B=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 96K=C2=A0 -<br># appjail cmd jexec jtest dd if=3D/dev/random of=
=3D/jailed/index.txt bs=3D16 count=3D1<br>1+0 records in<br>1+0 records out=
<br>16 bytes transferred in 0.000102 secs (157318 bytes/sec)<br># appjail c=
md jexec jtest hd /jailed/index.txt<br>00000000=C2=A0 a1 26 2a 9c f5 96 7b =
81=C2=A0 90 8d ba 36 d6 f9 4d 93=C2=A0 |.&*...{....6..M.|<br>00000010<b=
r># appjail cmd jexec jtest zfs list -t snapshot zroot/jailed<br>NAME=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 USED=C2=A0 AVAIL=C2=A0=C2=A0=C2=A0=C2=A0 REFER=C2=A0 MOUNTP=
OINT<br>zroot/jailed@guard=C2=A0=C2=A0=C2=A0 56K=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 96K=C2=A0 -<br># appjail cmd jexe=
c jtest zfs rollback zroot/jailed@guard<br># appjail cmd jexec jtest hd /ja=
iled/index.txt<br>00000000=C2=A0 48 69 21 0a=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |Hi!.|<br>00000004<br>```<=
br><br>I hope this can help you.</div></div><div><div dir=3D"auto"><br><br>=
~ DtxdF</div><br><br><div class=3D"gmail_quote"><div dir=3D"auto">El 6 de n=
oviembre de 2023 6:07:06 p. m. UTC, Chris Watson <<a href=3D"mailto:bsdu=
nix44@gmail.com" target=3D"_blank">bsdunix44@gmail.com</a>> escribi=C3=
=B3:</div><blockquote class=3D"gmail_quote" style=3D"margin:0pt 0pt 0pt 0.8=
ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-le=
ft-color:rgb(204,204,204)">
<div dir=3D"ltr">I've been trying to get a zfs dataset delegated into a=
jail (to run PG on), and allow snapshots and rollback to take place inside=
the jail. I can get the dataset mounted into the jail, I can get zfs to ta=
ke the snapshot, list the snapshot, but when I rollback or try to ls -la th=
e directory to see the '.zfs' dir it isn't there and the zfs ro=
llback completes but it doesn't actually rollback. I'm so close to =
getting this to work! I'm just missing *something* in the sauce. When I=
do the zfs rollback zfs looks like it completes the rollback and goes back=
to a shell prompt but the files I remove before the rollback are not in th=
e /var/db/postgres/data16 directory nor is ".zfs" shown in ls -la=
. So something is wonky on my end. I'm so close, it's halfway there=
, it looks like it takes a snapshot, the snapshot shows up in a zfs list -t=
snapshot, but it's also not really there. I'm doing something just=
slightly wrong here. I just cant figure out what I have wrong.<div><br></d=
iv><div>Below are the configs:</div><div># The jail's config</div><div>=
<a href=3D"https://bsd.to/P176" target=3D"_blank">https://bsd.to/P176</a></=
div><div># zfs list from inside the jail</div><div><a href=3D"https://bsd.t=
o/mPde" target=3D"_blank">https://bsd.to/mPde</a><br></div><div># zfs list =
-t snapshot from inside the jail</div><div><a href=3D"https://bsd.to/R8dw" =
target=3D"_blank">https://bsd.to/R8dw</a><br></div><div># ls -la /var/db/po=
stgres/data16 output from inside the jail</div><div><a href=3D"https://bsd.=
to/1di2" target=3D"_blank">https://bsd.to/1di2</a><br></div><div># rc.conf =
of the jail</div><div><a href=3D"https://bsd.to/JcnH" target=3D"_blank">htt=
ps://bsd.to/JcnH</a><br></div><div><br></div><div>The jail is running 13.2-=
P4.</div><div>Using bastillebsd 0.10.20231013 for creation/management.</div=
><div><br></div><div>Thanks!</div><div>Chris</div>
</div>
</blockquote></div></div></blockquote></div></div>
--000000000000958012060987cfc3--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHnbxSSra=JQ0iqne0NBomrexSxu8ar7ZViq7tfKVDsO7vczAg>