Date: Mon, 28 Feb 2022 17:11:27 +0100 From: Michael Gmelin <grembo@freebsd.org> To: FreeBSD User <freebsd@walstatt-de.de> Cc: FreeBSD virtualization <freebsd-virtualization@freebsd.org>, FreeBSD CURRENT <freebsd-current@freebsd.org> Subject: Re: bastille : poudriere not working in jail: jail: jail:_set: Operation not permitted! Message-ID: <20220228171127.2469a57d.grembo@freebsd.org> In-Reply-To: <20220228161545.251fe0d8@hermann> References: <20220228161545.251fe0d8@hermann>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 28 Feb 2022 16:15:45 +0100 FreeBSD User <freebsd@walstatt-de.de> wrote: > Hello folks, > > we run at least two poudriere build systems on recent CURRENT boxes > and one of these poudriere build systems is working within a jail - > setup via FreeBSD's /etc/jail.conf and by misusing the port ezjail > for copying/deploying our self-compiled jail binary. The poudriere > jail uses ZFS and is, to make it short, working like a charme. > > Now we try to setup another poudriere, but this time the base is > XigmaNAS 12.3.0.4/9009, which is based upon 12.X-RELENG, utilizing > "bastille". Bastille is up to date (in terms od the XigmaNAS plugin). > > Following the setup we used on the native CURRENT "jailed poudriere" > builder and also following this reference (for those who want to > check on this) > > https://www.mimar.rs/blog/host-your-own-services-with-freebsd-jails-part-3-poudriere > > which seems quite recent and with the exception, that we use "vnet" > on all of our systems for jails and so does XigmaNAS. > > Starting a building process via poudriere ends up with > > > # poudriere bulk -p head -z default -j 123-amd64 -f > /usr/local/etc/poudriere.d/zeit4-default.pkglist [00:00:00] Creating > the reference jail... done [00:00:01] Mounting system devices for > 123-amd64-head-default [00:00:01] Warning: Using packages from > previously failed, or uncommitted, build: > /mnt/poudriere/data/packages/123-amd64-head-default/.building > [00:00:01] Mounting ports from: /mnt/poudriere/ports/head [00:00:01] > Mounting packages from: > /mnt/poudriere/data/packages/123-amd64-head-default [00:00:01] > Mounting distfiles from: /mnt/poudriere/ports/distfiles [00:00:01] > Copying /var/db/ports from: > /usr/local/etc/poudriere.d/head-amd64-head-default-options [00:00:02] > Appending to make.conf: /usr/local/etc/poudriere.d/make.conf > /etc/resolv.conf -> > /mnt/poudriere/data/.m/123-amd64-head-default/ref/etc/resolv.conf > [00:00:02] Starting jail 123-amd64-head-default jail: jail_set: > Operation not permitted [00:00:02] Cleaning up [00:00:02] Unmounting > file systems > > poudriere jail -l: > > # poudriere jail -l > JAILNAME VERSION ARCH METHOD TIMESTAMP PATH > 123-amd64 12.3-RELEASE amd64 > url=https://download.freebsd.org/releases/a ... 3-RELEASE/ 2022-02-24 > 14:14:25 /mnt/poudriere/jails/123-amd64 130-amd64 13.0-RELEASE amd64 > url=https://download.freebsd.org/releases/a ... 0-RELEASE/ 2022-02-24 > 14:11:32 /mnt/poudriere/jails/130-amd64 > > The jail.conf for this specific jail is as follows: > > [...] > pulverfass-001 { > devfs_ruleset = 13; > enforce_statfs = 1; > exec.clean; > exec.consolelog = > /mnt/extensions/bastille/logs/pulverfass-001_console.log; exec.start > = '/bin/sh /etc/rc'; exec.stop = '/bin/sh /etc/rc.shutdown'; > host.hostname = XXXXXXXXX; > mount.devfs; > mount.fstab = /mnt/extensions/bastille/jails/pulverfass-001/fstab; > path = /mnt/extensions/bastille/jails/pulverfass-001/root; > securelevel = 0; > > vnet; > vnet.interface = e0b_bastille4; > exec.prestart += "jib addm bastille4 igb0"; > exec.prestart += "ifconfig e0a_bastille4 description \"vnet host > interface for Bastille jail pulverfass-001\""; exec.poststop += "jib > destroy bastille4"; > > allow.mount; > allow.mount.fdescfs; > allow.mount.devfs; > allow.mount.tmpfs; > allow.mount.nullfs; > allow.mount.procfs; > allow.mount.linsysfs; > allow.mount.linprocfs; > allow.mount.zfs; > > allow.chflags; > allow.raw_sockets; > allow.socket_af; > allow.sysvipc; > > linux = new; > > exec.created += "/sbin/zfs jail ${name} BUNKER00/poudriere"; > exec.start += "/sbin/zfs mount -a"; > exec.poststop += "/sbin/zfs unjail BUNKER00/poudriere"; > > } > [...] > > Tracking the execution of the build process by issuing > > poudriere -x bulk ... > > and examin the resulting trace doesn' tgive me any hint, the error > reported above immediately occurs when the jail is about to be > started: > > + set -u +x > + jail -c persist 'name=123-amd64-head-default' > 'path=/mnt/poudriere/data/.m/ \ 123-amd64-head-default/ref' > 'host.hostname=basehost.local.domain' \ 'ip4.addr=127.0.0.1' > 'ip6.addr=::1' allow.chflags allow.sysvipc jail: jail_set: Operation > not permitted > + exit_handler > [...] > > Searching the net revealed some issues with setting IP4 and IP6 in > poudriere, but those findings are dated back to 2017 and 2014 and I > guess this is solved right now. > > The difference between our manually jail.conf driven setup and the > XigmaNAS/bastille based one is, bastille uses jib/netgraph based > seutups of the vnet and the ip4/ip6 is setup from rc.conf, while we > use epair in the other world and the ip is setup from withing the > jail definition in jail.conf. > > I'm out of ideas here and after two days of trial and error and > trying to understand what's going on lost ... Any hints or tipps? > > Thanks in advance, > > O. Hartmann Hi Oliver, I don't see `children.max` set in any of the configuration you shared above. Cheers Michael -- Michael Gmelin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20220228171127.2469a57d.grembo>