From nobody Mon Feb 28 15:15:45 2022 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C4F9B19E655D; Mon, 28 Feb 2022 15:15:57 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp6.goneo.de (smtp6.goneo.de [85.220.129.31]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4K6kV42PVHz4pPf; Mon, 28 Feb 2022 15:15:56 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from hub2.goneo.de (hub2.goneo.de [IPv6:2001:1640:5::8:53]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp6.goneo.de (Postfix) with ESMTPS id 6A9B610A32E8; Mon, 28 Feb 2022 16:15:48 +0100 (CET) Received: from hub2.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPS id 8885D10A3308; Mon, 28 Feb 2022 16:15:46 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1646061346; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ARH83yi8Caw1S4UOmTrf0EbQak4CgsLv9xgtVffwm1g=; b=HmLqD38VYnQUNHjNNjsOYST7Bt0iqO9Qfc8uIDchT4vcH3KmP9yZ9eBFdWs6kEy1de5O1I 8nZl2T+DIDBssDVrJPxsoSTsVRiN2BscnHlx52XyBjczW43/LKDlRf1qe/NPm89T27ynjx FpzqMvVmtiJXOGx2/aPT733m1p9jDE0oiuGJ8DPBDeBxnJ83SV/r10MvqUWAl2XGNt4H5V r1R7Q8ICVwgR6dUBIMnJIRA9lQCnzjkQJSUAOAXWxAWfFv7lBzVNa0X+9YXWXXfyuZpwOY pURVCj2igkHkwbuiy4JUmxwxilTfTTFOwjHwTybjgEDNgR2WP249XtSqxszBsQ== Received: from hermann (dynamic-2a01-0c22-ad8a-3000-954e-91b4-d2cb-c2d0.c22.pool.telefonica.de [IPv6:2a01:c22:ad8a:3000:954e:91b4:d2cb:c2d0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPSA id 3D7C410A3306; Mon, 28 Feb 2022 16:15:46 +0100 (CET) Date: Mon, 28 Feb 2022 16:15:45 +0100 From: FreeBSD User To: FreeBSD virtualization , FreeBSD CURRENT Subject: bastille : poudriere not working in jail: jail: jail:_set: Operation not permitted! Message-ID: <20220228161545.251fe0d8@hermann> List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: 57d998 X-Rspamd-UID: b72891 X-Rspamd-Queue-Id: 4K6kV42PVHz4pPf X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=HmLqD38V; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@walstatt-de.de has no SPF policy when checking 85.220.129.31) smtp.mailfrom=freebsd@walstatt-de.de X-Spamd-Result: default: False [-2.78 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[walstatt-de.de]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[walstatt-de.de:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.88)[-0.885]; SUBJECT_ENDS_EXCLAIM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current,freebsd-virtualization]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:25394, ipnet:85.220.128.0/17, country:DE]; RCVD_TLS_ALL(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[85.220.129.31:from] X-ThisMailContainsUnwantedMimeParts: N Hello folks, we run at least two poudriere build systems on recent CURRENT boxes and one of these poudriere build systems is working within a jail - setup via FreeBSD's /etc/jail.conf and by misusing the port ezjail for copying/deploying our self-compiled jail binary. The poudriere jail uses ZFS and is, to make it short, working like a charme. Now we try to setup another poudriere, but this time the base is XigmaNAS 12.3.0.4/9009, which is based upon 12.X-RELENG, utilizing "bastille". Bastille is up to date (in terms od the XigmaNAS plugin). Following the setup we used on the native CURRENT "jailed poudriere" builder and also following this reference (for those who want to check on this) https://www.mimar.rs/blog/host-your-own-services-with-freebsd-jails-part-3-poudriere which seems quite recent and with the exception, that we use "vnet" on all of our systems for jails and so does XigmaNAS. Starting a building process via poudriere ends up with # poudriere bulk -p head -z default -j 123-amd64 -f /usr/local/etc/poudriere.d/zeit4-default.pkglist [00:00:00] Creating the reference jail... done [00:00:01] Mounting system devices for 123-amd64-head-default [00:00:01] Warning: Using packages from previously failed, or uncommitted, build: /mnt/poudriere/data/packages/123-amd64-head-default/.building [00:00:01] Mounting ports from: /mnt/poudriere/ports/head [00:00:01] Mounting packages from: /mnt/poudriere/data/packages/123-amd64-head-default [00:00:01] Mounting distfiles from: /mnt/poudriere/ports/distfiles [00:00:01] Copying /var/db/ports from: /usr/local/etc/poudriere.d/head-amd64-head-default-options [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf /etc/resolv.conf -> /mnt/poudriere/data/.m/123-amd64-head-default/ref/etc/resolv.conf [00:00:02] Starting jail 123-amd64-head-default jail: jail_set: Operation not permitted [00:00:02] Cleaning up [00:00:02] Unmounting file systems poudriere jail -l: # poudriere jail -l JAILNAME VERSION ARCH METHOD TIMESTAMP PATH 123-amd64 12.3-RELEASE amd64 url=https://download.freebsd.org/releases/a ... 3-RELEASE/ 2022-02-24 14:14:25 /mnt/poudriere/jails/123-amd64 130-amd64 13.0-RELEASE amd64 url=https://download.freebsd.org/releases/a ... 0-RELEASE/ 2022-02-24 14:11:32 /mnt/poudriere/jails/130-amd64 The jail.conf for this specific jail is as follows: [...] pulverfass-001 { devfs_ruleset = 13; enforce_statfs = 1; exec.clean; exec.consolelog = /mnt/extensions/bastille/logs/pulverfass-001_console.log; exec.start = '/bin/sh /etc/rc'; exec.stop = '/bin/sh /etc/rc.shutdown'; host.hostname = XXXXXXXXX; mount.devfs; mount.fstab = /mnt/extensions/bastille/jails/pulverfass-001/fstab; path = /mnt/extensions/bastille/jails/pulverfass-001/root; securelevel = 0; vnet; vnet.interface = e0b_bastille4; exec.prestart += "jib addm bastille4 igb0"; exec.prestart += "ifconfig e0a_bastille4 description \"vnet host interface for Bastille jail pulverfass-001\""; exec.poststop += "jib destroy bastille4"; allow.mount; allow.mount.fdescfs; allow.mount.devfs; allow.mount.tmpfs; allow.mount.nullfs; allow.mount.procfs; allow.mount.linsysfs; allow.mount.linprocfs; allow.mount.zfs; allow.chflags; allow.raw_sockets; allow.socket_af; allow.sysvipc; linux = new; exec.created += "/sbin/zfs jail ${name} BUNKER00/poudriere"; exec.start += "/sbin/zfs mount -a"; exec.poststop += "/sbin/zfs unjail BUNKER00/poudriere"; } [...] Tracking the execution of the build process by issuing poudriere -x bulk ... and examin the resulting trace doesn' tgive me any hint, the error reported above immediately occurs when the jail is about to be started: + set -u +x + jail -c persist 'name=123-amd64-head-default' 'path=/mnt/poudriere/data/.m/ \ 123-amd64-head-default/ref' 'host.hostname=basehost.local.domain' \ 'ip4.addr=127.0.0.1' 'ip6.addr=::1' allow.chflags allow.sysvipc jail: jail_set: Operation not permitted + exit_handler [...] Searching the net revealed some issues with setting IP4 and IP6 in poudriere, but those findings are dated back to 2017 and 2014 and I guess this is solved right now. The difference between our manually jail.conf driven setup and the XigmaNAS/bastille based one is, bastille uses jib/netgraph based seutups of the vnet and the ip4/ip6 is setup from rc.conf, while we use epair in the other world and the ip is setup from withing the jail definition in jail.conf. I'm out of ideas here and after two days of trial and error and trying to understand what's going on lost ... Any hints or tipps? Thanks in advance, O. Hartmann