From nobody Tue Nov 7 03:45:57 2023 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SPYzQ10hwz50KvS for ; Tue, 7 Nov 2023 03:46:10 +0000 (UTC) (envelope-from bsdunix44@gmail.com) Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SPYzP3vwhz3bsZ for ; Tue, 7 Nov 2023 03:46:09 +0000 (UTC) (envelope-from bsdunix44@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1cc3388621cso48028355ad.1 for ; Mon, 06 Nov 2023 19:46:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699328768; x=1699933568; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=6M2QsscCDgKRhVHrB5B3Q0cjyrwLJ2YR324CSKuY9n4=; b=DB/6yQYVzWz9h29HOHiwcfLAthgbV1czbyw4uzFkrsweJHxIcMgtco5lubtYjXe6gO VWZQvpqLx+4qavDqVbgSumX80q+RURjjjNuqItzx+AGBxcYLS6xgPbFGIwcvdFwdPFNy U5CwFe5or/R8CuAmONfYGiB0ny7cKpai5OOId1eLyfRnMUMwsEpwWARrTwV5nr3+Xo5U ImllwzAwBJ3+yv9ZNrIUwCqb1BajyantI8Jckgjb0ptUKG04Ve6zeYimFJG5W6vVXk/9 rRVYcyA35gGn9JYiTeo1wkAlS+r0Z1eO05pp8WvmBYUYrnBRPWuRJTt2F7ddCvkwqX5U d23A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699328768; x=1699933568; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=6M2QsscCDgKRhVHrB5B3Q0cjyrwLJ2YR324CSKuY9n4=; b=KSiMHg3E2seYmB8I/YvM9opRG4VMIUjWW3WRbgguevYRNpznSIvUZY4vywd+Uv6WhP npkjQHLWqR8jyPRj00wz6LmbDC9m5IoCB6bwJGL38D69UHUHxYdhpT8rIQ4plomIDMaz tfPuhtrSaybQwOPzxIWV4OKYegTcMN3PLmqpp0c5HdUnibotyCoEp5KoOa3KTeyF4+6V q3eb2EBYln1pFtNkN2hRanM68G/k8S9tiXH04ibX99m7tKcl2wtbM9KotOha1wX/wIXy 7C3iqPxOezu7VIlM2sI5hP2qGEpfugHJDzArxg90kmN+BNXB6SIiBHwcigNhU9E4Nz/P 33Dg== X-Gm-Message-State: AOJu0YzHZ9/lPGbGLnrZ9t5RltTsvg76YW9cpvrIJFd3ii4bkguzIeKG lOXzUEAOWZyVZnHQS6CSI52FCYi+oTLsJ+I2UFM= X-Google-Smtp-Source: AGHT+IFigdnQHNtN8A0iACxX9eQdn0CE0ENrn2uZM9ofUcu4zlgts1vXCApdZ/wT6GT1hVp+g7TdU6sn+aNAwRAhMIU= X-Received: by 2002:a17:90b:38d0:b0:27d:433e:e69c with SMTP id nn16-20020a17090b38d000b0027d433ee69cmr2226520pjb.18.1699328768178; Mon, 06 Nov 2023 19:46:08 -0800 (PST) List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Chris Watson Date: Mon, 6 Nov 2023 21:45:57 -0600 Message-ID: Subject: Re: I can get zfs snapshot/rollback in a jail to work 99% but it isn't quite 100% working. What am I missing? To: DtxdF Cc: freebsd-jail@freebsd.org Content-Type: multipart/alternative; boundary="000000000000958012060987cfc3" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4SPYzP3vwhz3bsZ --000000000000958012060987cfc3 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable So as I mentioned I=E2=80=99ve able to mail the dataset. It gets mounted up= on starting the jail. It shows up in =E2=80=9Czfs list=E2=80=9D. And when I do= zfs snapshot on the dataset it appears to create the snapshot as it shows up in a =E2=80=9C= zfs list -t snapshot=E2=80=9D but the snapdir isn=E2=80=99t visible even after setti= ng snapdir to visible, and when I rollback using the snapshot it doesn=E2=80=99t actually rollback. I=E2=80=99m so close to this working, something just isn=E2=80=99= t right and I can=E2=80=99t figure out what. I really wish this was written up in the han= dbook. =E2=80=9CHow to jail a dataset from the host=E2=80=9D and =E2=80=9Chow to s= napshot and rollback a jailed dataset=E2=80=9D. :) if I figure this I=E2=80=99ll definitely be wri= ting this up. Chris On Mon, Nov 6, 2023 at 2:35 PM DtxdF wrote: > Hi Chris, > > Maybe your dataset is not mounted inside the jail. I thought that simply > enabling `/etc/rc.d/zfs` was fine, but no, it just doesn't work. I don't > know if this behavior is a bug or something else, but at the moment I don= 't > have time to investigate. > > I have a similar setup for a jail with a delegated dataset. I use AppJail= , > but the steps are similar to other tools: > > ``` > # zfs create -o jailed=3Don -o mountpoint=3D/jailed zroot/jailed > # appjail quick jtest \ > mount_devfs \ > device=3D'include $devfsrules_hide_all' \ > device=3D'include $devfsrules_unhide_basic' \ > device=3D'include $devfsrules_unhide_login' \ > device=3D'path zfs unhide' \ > template=3Dtemplate.conf \ > overwrite=3Dforce \ > start > ``` > > In AppJail, a template configuration file is similar to `jail.conf(5)`: > > ``` > # cat template.conf > exec.start: "/bin/sh /etc/rc" > exec.stop: "/bin/sh /etc/rc.shutdown jail" > allow.mount > allow.mount.zfs > enforce_statfs: 1 > exec.poststart: "zfs jail ${name} zroot/jailed" > exec.poststart+: "appjail cmd jexec ${name} zfs mount zroot/jailed" > exec.prestop: "appjail cmd jexec ${name} zfs umount zroot/jailed" > exec.prestop+: "zfs unjail ${name} zroot/jailed" > ``` > > As you can see, the dataset is mounted after running `zfs-jail(8)`. The > steps are similar when the jail is stopped, but the dataset is unmounted > and `zfs-unjail(8)` is executed. > > Inside the jail I can see the mounted datasets: > > ``` > # appjail cmd jexec jtest zfs list -r > NAME USED AVAIL REFER MOUNTPOINT > zroot 34.1G 249G 96K /zroot > zroot/jailed 96K 249G 96K /jailed > # appjail cmd jexec jtest mount -t zfs > zroot/appjail/jails/jtest/jail on / (zfs, local, noatime, nfsv4acls) > zroot/jailed on /jailed (zfs, local, noatime, nfsv4acls) > # appjail cmd jexec jtest ls /jailed > index.txt > # appjail cmd jexec jtest cat /jailed/index.txt > Hi! > ``` > > And I can use `zfs-rollback(8)` just fine: > > ``` > # appjail cmd jexec jtest zfs snapshot zroot/jailed@guard > # appjail cmd jexec jtest zfs list -t snapshot zroot/jailed > NAME USED AVAIL REFER MOUNTPOINT > zroot/jailed@guard 0B - 96K - > # appjail cmd jexec jtest dd if=3D/dev/random of=3D/jailed/index.txt bs= =3D16 > count=3D1 > 1+0 records in > 1+0 records out > 16 bytes transferred in 0.000102 secs (157318 bytes/sec) > # appjail cmd jexec jtest hd /jailed/index.txt > 00000000 a1 26 2a 9c f5 96 7b 81 90 8d ba 36 d6 f9 4d 93 > |.&*...{....6..M.| > 00000010 > # appjail cmd jexec jtest zfs list -t snapshot zroot/jailed > NAME USED AVAIL REFER MOUNTPOINT > zroot/jailed@guard 56K - 96K - > # appjail cmd jexec jtest zfs rollback zroot/jailed@guard > # appjail cmd jexec jtest hd /jailed/index.txt > 00000000 48 69 21 0a |Hi!.| > 00000004 > ``` > > I hope this can help you. > > > ~ DtxdF > > > El 6 de noviembre de 2023 6:07:06 p. m. UTC, Chris Watson < > bsdunix44@gmail.com> escribi=C3=B3: > >> I've been trying to get a zfs dataset delegated into a jail (to run PG >> on), and allow snapshots and rollback to take place inside the jail. I c= an >> get the dataset mounted into the jail, I can get zfs to take the snapsho= t, >> list the snapshot, but when I rollback or try to ls -la the directory to >> see the '.zfs' dir it isn't there and the zfs rollback completes but it >> doesn't actually rollback. I'm so close to getting this to work! I'm jus= t >> missing *something* in the sauce. When I do the zfs rollback zfs looks l= ike >> it completes the rollback and goes back to a shell prompt but the files = I >> remove before the rollback are not in the /var/db/postgres/data16 direct= ory >> nor is ".zfs" shown in ls -la. So something is wonky on my end. I'm so >> close, it's halfway there, it looks like it takes a snapshot, the snapsh= ot >> shows up in a zfs list -t snapshot, but it's also not really there. I'm >> doing something just slightly wrong here. I just cant figure out what I >> have wrong. >> >> Below are the configs: >> # The jail's config >> https://bsd.to/P176 >> # zfs list from inside the jail >> https://bsd.to/mPde >> # zfs list -t snapshot from inside the jail >> https://bsd.to/R8dw >> # ls -la /var/db/postgres/data16 output from inside the jail >> https://bsd.to/1di2 >> # rc.conf of the jail >> https://bsd.to/JcnH >> >> The jail is running 13.2-P4. >> Using bastillebsd 0.10.20231013 for creation/management. >> >> Thanks! >> Chris >> > --000000000000958012060987cfc3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
So as I mentioned I=E2=80=99ve able to mail the dataset. = It gets mounted upon starting the jail. It shows up in =E2=80=9Czfs list=E2= =80=9D. And when I do zfs snapshot on the dataset it appears to create the = snapshot as it shows up in a =E2=80=9Czfs list -t snapshot=E2=80=9D but the= snapdir isn=E2=80=99t visible even after setting snapdir to visible, and w= hen I rollback using the snapshot it doesn=E2=80=99t actually rollback. I= =E2=80=99m so close to this working, something just isn=E2=80=99t right and= I can=E2=80=99t figure out what. I really wish this was written up in the = handbook. =E2=80=9CHow to jail a dataset from the host=E2=80=9D and =E2=80= =9Chow to snapshot and rollback a jailed dataset=E2=80=9D. :) if I figure t= his I=E2=80=99ll definitely be writing this up.=C2=A0

Chris=C2=A0

On Mon, Nov 6, 2023 at 2:35 P= M DtxdF <DtxdF@disroot.org> = wrote:
Hi Chris,

Ma= ybe your dataset is not mounted inside the jail. I thought that simply enab= ling `/etc/rc.d/zfs` was fine, but no, it just doesn't work. I don'= t know if this behavior is a bug or something else, but at the moment I don= 't have time to investigate.

I have a similar setup for a jail w= ith a delegated dataset. I use AppJail, but the steps are similar to other = tools:

```
# zfs create -o jailed=3Don -o mountpoint=3D/jailed zr= oot/jailed
# appjail quick jtest \
mount_devfs \
device=3D'i= nclude $devfsrules_hide_all' \
device=3D'include $devfsrules_un= hide_basic' \
device=3D'include $devfsrules_unhide_login' \=
device=3D'path zfs unhide' \
template=3Dtemplate.conf \ overwrite=3Dforce \
start
```

In AppJail, a template config= uration file is similar to `jail.conf(5)`:

```
# cat template.con= f
exec.start: "/bin/sh /etc/rc"
exec.stop: "/bin/sh /e= tc/rc.shutdown jail"
allow.mount
allow.mount.zfs
enforce_stat= fs: 1
exec.poststart: "zfs jail ${name} zroot/jailed"
exec.= poststart+: "appjail cmd jexec ${name} zfs mount zroot/jailed"exec.prestop: "appjail cmd jexec ${name} zfs umount zroot/jailed"= ;
exec.prestop+: "zfs unjail ${name} zroot/jailed"
```
<= br>As you can see, the dataset is mounted after running `zfs-jail(8)`. The = steps are similar when the jail is stopped, but the dataset is unmounted an= d `zfs-unjail(8)` is executed.

Inside the jail I can see the mounted= datasets:

```
# appjail cmd jexec jtest zfs list -r
NAME=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 USED=C2=A0 AVAIL= =C2=A0=C2=A0=C2=A0=C2=A0 REFER=C2=A0 MOUNTPOINT
zroot=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 34.1G=C2=A0=C2=A0 249G=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 96K=C2=A0 /zroot
zroot/jailed=C2=A0=C2=A0=C2=A0 96K=C2= =A0=C2=A0 249G=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 96K=C2=A0 /jailed
# a= ppjail cmd jexec jtest mount -t zfs
zroot/appjail/jails/jtest/jail on / = (zfs, local, noatime, nfsv4acls)
zroot/jailed on /jailed (zfs, local, no= atime, nfsv4acls)
# appjail cmd jexec jtest ls /jailed
index.txt
#= appjail cmd jexec jtest cat /jailed/index.txt
Hi!
```

And I c= an use `zfs-rollback(8)` just fine:

```
# appjail cmd jexec jtest= zfs snapshot zroot/jailed@guard
# appjail cmd jexec jtest zfs list -t s= napshot zroot/jailed
NAME=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 USED=C2=A0 AVAIL=C2=A0= =C2=A0=C2=A0=C2=A0 REFER=C2=A0 MOUNTPOINT
zroot/jailed@guard=C2=A0=C2=A0= =C2=A0=C2=A0 0B=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 96K=C2=A0 -
# appjail cmd jexec jtest dd if=3D/dev/random of= =3D/jailed/index.txt bs=3D16 count=3D1
1+0 records in
1+0 records out=
16 bytes transferred in 0.000102 secs (157318 bytes/sec)
# appjail c= md jexec jtest hd /jailed/index.txt
00000000=C2=A0 a1 26 2a 9c f5 96 7b = 81=C2=A0 90 8d ba 36 d6 f9 4d 93=C2=A0 |.&*...{....6..M.|
00000010# appjail cmd jexec jtest zfs list -t snapshot zroot/jailed
NAME=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 USED=C2=A0 AVAIL=C2=A0=C2=A0=C2=A0=C2=A0 REFER=C2=A0 MOUNTP= OINT
zroot/jailed@guard=C2=A0=C2=A0=C2=A0 56K=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 96K=C2=A0 -
# appjail cmd jexe= c jtest zfs rollback zroot/jailed@guard
# appjail cmd jexec jtest hd /ja= iled/index.txt
00000000=C2=A0 48 69 21 0a=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |Hi!.|
00000004
```<= br>
I hope this can help you.


= ~ DtxdF


El 6 de n= oviembre de 2023 6:07:06 p. m. UTC, Chris Watson <bsdunix44@gmail.com> escribi=C3= =B3:
I've been trying to get a zfs dataset delegated into a= jail (to run PG on), and allow snapshots and rollback to take place inside= the jail. I can get the dataset mounted into the jail, I can get zfs to ta= ke the snapshot, list the snapshot, but when I rollback or try to ls -la th= e directory to see the '.zfs' dir it isn't there and the zfs ro= llback completes but it doesn't actually rollback. I'm so close to = getting this to work! I'm just missing *something* in the sauce. When I= do the zfs rollback zfs looks like it completes the rollback and goes back= to a shell prompt but the files I remove before the rollback are not in th= e /var/db/postgres/data16 directory nor is ".zfs" shown in ls -la= . So something is wonky on my end. I'm so close, it's halfway there= , it looks like it takes a snapshot, the snapshot shows up in a zfs list -t= snapshot, but it's also not really there. I'm doing something just= slightly wrong here. I just cant figure out what I have wrong.

Below are the configs:
# The jail's config
= https://bsd.to/P176
# zfs list from inside the jail
# zfs list = -t snapshot from inside the jail
# ls -la /var/db/po= stgres/data16 output from inside the jail
# rc.conf = of the jail

The jail is running 13.2-= P4.
Using bastillebsd 0.10.20231013 for creation/management.

Thanks!
Chris
--000000000000958012060987cfc3--