From owner-freebsd-current@freebsd.org Fri Aug 30 12:18:03 2019 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 55415CD3C5 for ; Fri, 30 Aug 2019 12:18:03 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46KdmB0j2vz444j for ; Fri, 30 Aug 2019 12:18:01 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from outgoing.leidinger.net (p5B16567B.dip0.t-ipconnect.de [91.22.86.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by mailgate.Leidinger.net (Postfix) with ESMTPSA id 969AAAED6; Fri, 30 Aug 2019 14:17:53 +0200 (CEST) Received: from webmail.leidinger.net (webmail.Leidinger.net [IPv6:fd73:10c7:2053:1::3:102]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by outgoing.leidinger.net (Postfix) with ESMTPS id 35E5893CE; Fri, 30 Aug 2019 14:17:21 +0200 (CEST) Received: (from www@localhost) by webmail.leidinger.net (8.15.2/8.14.4/Submit) id x7UCHKNb075909; Fri, 30 Aug 2019 14:17:20 +0200 (CEST) (envelope-from Alexander@leidinger.net) X-Authentication-Warning: webmail.leidinger.net: www set sender to Alexander@leidinger.net using -f Received: from [::ffff:31.3.144.27] ([::ffff:31.3.144.27]) by webmail.leidinger.net (Horde Framework) with HTTPS; Fri, 30 Aug 2019 14:17:20 +0200 Date: Fri, 30 Aug 2019 14:17:20 +0200 Message-ID: <20190830141720.Horde.yKnzT-Dd2OP1_BMC2-svTaC@webmail.leidinger.net> From: Alexander Leidinger To: "O. Hartmann" Cc: freebsd-current@freebsd.org, Michael Zhilin Subject: Re: jails, ZFS, deprecated jail variables and poudriere problems References: <20190827101149.1efcb946@freyja> <20190828135700.Horde.m2EjpS9j67KYn2E1oSeoK8f@webmail.leidinger.net> <20190829142632.6b93892b@freyja> In-Reply-To: <20190829142632.6b93892b@freyja> User-Agent: Horde Application Framework 5 Accept-Language: de,en Content-Type: multipart/signed; boundary="=_6KdtFXi7O3zedMNg8yM4xn9"; protocol="application/pgp-signature"; micalg=pgp-sha1 MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1567167474; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=I7n82Amev7UMmeh00P1bFUGZv13SovdwfvMwPh+03Do=; b=Q7ZlAe1XuekCe4YLKqwddOyck38erEIr90kz4Mazqvmn481cpcLzwIdvWPcRG5QgHDJ1uo AtZ0TVy9snGTSrbnm3hDPGI6oDudXqSZ1AhYU79X9GC334riljO92xDm1tTlpo/WuNGxsK iuBf3nG9/gkPkmOKBd3C15p36EJ08xQNnJwsnjr4VZ7pV6M8Fy/nzKh8+6YIk4xhUx2pGp xHPNjyNA3rNEfgX2DyczcqI3dAPVjjhb2mTte7SJhVEOsqjZqyahDXaioaApTiwNvdogXE /6NSsgpyaWm7EppUxmRh/8yhCT0XlIWtlbKjp2Ijp1JqtiiK8T16jpq0yQIRjA== ARC-Seal: i=1; s=outgoing-alex; d=leidinger.net; t=1567167474; a=rsa-sha256; cv=none; b=vdmxb2o3wm61/Xt7TTDIRedYGalcdsWQLShHmlvkcDLP+bfTaQ3OUkAdNP/eYRwY8xN2Wi gbqL3nDACJD9CucLAoce0ULvhQ6kNwb2lDrwRGGTrLjv1B6DLE1Wma5QVvQa89FVld0H2K jWvRlGdJggdYLo7JsaNStqQLhhmlyIcBBvxan5530JV0VFTUMHmtqhuN3REhdZeGLWZOoN 6ha/UNADVOOQ0LPADs0Jd7O1X5F0P+45fRPInHDxMHwycw05w1v8JYfrzP2HCaz8znmfEZ 3aXrkPrdZImMFifuwVlzOEDakSFI0zqnGu8uvcCP/X644WGFBtqjOI0z0NCnRw== ARC-Authentication-Results: i=1; mailgate.Leidinger.net; auth=pass smtp.auth=netchild@leidinger.net smtp.mailfrom=Alexander@leidinger.net X-Rspamd-Queue-Id: 46KdmB0j2vz444j X-Spamd-Bar: --------- X-Spamd-Result: default: False [-9.77 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[leidinger.net:s=outgoing-alex]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; IP_SCORE(-3.68)[ip: (-9.66), ipnet: 89.238.64.0/18(-4.83), asn: 34240(-3.89), country: DE(-0.01)]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; XAW_SERVICE_ACCT(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[leidinger.net:+]; DMARC_POLICY_ALLOW(-0.50)[leidinger.net,quarantine]; RCVD_COUNT_THREE(0.00)[4]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.991,0]; RECEIVED_SPAMHAUS_PBL(0.00)[123.86.22.91.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE]; ARC_ALLOW(-1.00)[i=1]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 12:18:03 -0000 This message is in MIME format and has been PGP signed. --=_6KdtFXi7O3zedMNg8yM4xn9 Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Quoting "O. Hartmann" (from Thu, 29 Aug 2019=20=20 14:26:38=20+0200): > On Wed, 28 Aug 2019 13:57:00 +0200 > Alexander Leidinger wrote: > >> Quoting "O. Hartmann" (from Tue, 27 Aug 2019 >> 10:11:54 +0200): >> >> > We have a single ZFS pool (raidz), call it pool00 and this pool00=20= =20 >>=20conatins a >> > ZFS dataset pool00/poudriere which we want to exclusively attach=20=20 >>=20to a jail. >> > pool00/poudriere contains a complete clone of a former, now decomissio= ned >> > machine and is usable by the host bearing the jails. The jail, named >> > poudriere, >> > has these config parameters set in /etc/jail.conf as recommended: >> > >> > enforce_statfs=3D "0"; > > now set to > enforce_statfs=3D "1"; From a security point of view this is off course better, but "0"=20=20 should=20have been OK for your problem case. [...] >> > Here I find the first confusing observation. I can't interact with >> > the dataset >> > and its content within the jail. I've set the "jailed" property of >> > pool00/poudriere via "zfs set jailed=3Don pool00/poudriere" and I=20= =20 >>=20also have to >> > attach the jailed dataset manually via "zfs jail poudriere >> > pool00/poudriere" to >> > the (running) jail. But within the jail, listing ZFS's mountpoints rev= eal: >> > >> > NAME USED AVAIL REFER MOUNTPOINT >> > pool00 124G 8.62T 34.9K /pool00 >> > pool00/poudriere 34.9K 8.62T 34.9K /pool/poudriere >> > >> > but nothing below /pool/poudriere is visible to the jail. Being confus= ed I > > Since we use ezjail-admin for jail a rudimentary jail administration (jus= t > creating and/or deleting the jail, maintenance is done manually), jails a= re > rooting at > > pool00 /pool00 > pool00/ezjail/ /pool/jails > pool00/ezjail/pulverfass /pool/jails/pulverfass > > "pulverfass" is the jail supposed to do the poudriere's job. > > Since I got confused about the orientation of the "directory tree" - the = root > is toplevel instead of downlevel - I corrected the ZFS dataset holding th= e > poudriere stuff that way: > > pool00/ezjail/poudriere /pool/poudriere > > The jail "pulverfass" now is supposed to mount the dataset at > > /pool/jails/pulverfass/pool/poudriere > >> >> Please be more verbose what you mean by "interact" and "is visible". >> >> Do zfs commands on the dataset work? > > After I corrected my mistake by not respecting the mountpoint according t= o > statfs, with the changes explained above I'm able to mount /pool/poudrier= e > within the jail "pulverfass", but I still have problems with the way=20= =20 >=20how I have > to mount this dataset. When zfs-mounted (zfs mount -a), I'm able to use t= he > dataset with poudriere as expected! But after rebooting the host and=20= =20 >=20after all > jails has been restarted as well, I have to first make the dataset > /pool/poudriere available to the jail via the command "zfs jail pulverfas= s > ool00/ezjail/pulverfass" - which seems not to be done automatically by th= e > startup process - and then from within the jail "pulverfass" I can mount = the > dataset as desribed above. This seems to be a big step ahead for me. Great. >> Note, I don't remember if you can manage the root of the jail, but at >> least subsequent jails should be possible to manage. I don't have a >> jail where the root is managed in the jail, just additional ones. >> Those need to have set a mountpoint after the initial jailing and then >> maybe even be mounted for the first time. >> >> Please also check /etc/defaults/devfs.rules if the jail rule contains >> an unhide entry for zfs. > > Within /etc/jail.conf > > devfs_ruleset=3D "4"; > > is configured as a common rulesset for all jails (in the common portion o= f > /etc/jail.conf). > There is no custom devfs.rules in /etc/, so /etc/defaults/devfs.rules sho= uld > apply and as far as I can see, there is an "unhide" applied to zfs: > > [... /etc/defaults/devfs.rulse ...] > > # Devices usually found in a jail. > # > [devfsrules_jail=3D4] > add include $devfsrules_hide_all > add include $devfsrules_unhide_basic > add include $devfsrules_unhide_login > add path fuse unhide > add path zfs unhide > [...] > > So, I guess everything is all right from this perspective, isn't it? Yes. And as you was able to do the zfs mount, you even had the proof. > Is there a way to automatically provide the ZFS dataset of choice to the > propper jail or do i have to either > > issue manually "zfs jail jailid/jailname pool/dataset" or put such a=20= =20 >=20command as > script-command in the jail's definition portion as > > exec.prestart+=3D "zfs jail ${name} pool00/ezjail/poudriere"; > ? I never tried that with plain jails. ezjail has the=20=20 jail_xxx_zfs_datasets=20variable in the config directory. Unfortunately=20= =20 the=20attachment of the datasets to the jail is not early enough to be=20= =20 picked=20up by the start scripts (so probably with exec.poststart, if=20=20 you=20use that you can also use it to launch "jexec ... service zfs=20=20 start"=20inside the jail). iocage works much better in this regard. Bye, Alexander. --=20 http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_6KdtFXi7O3zedMNg8yM4xn9 Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAABAgAGBQJdaRPQAAoJEBINsJsD+NiGHs4P/ice82BNIeRlfwbQxzXjdVKL gH8HyrPPH9glfe20Pq/ByawoZhU+lEbayQUI0zzxVsC7VqCYYYI/DAWWEVZPVCkI XiKTdVt6+fjrtd2cnFcf6c1SWncxIKmO98Qe8fXBmPPp4Q+OcT9XhWEtlM/3SGxa vipXQvTVNEVaiuexKH7pa9QXEsyQ30YzQ58VTX12UNOue3LzoUsOu3QRgJGBdCzh 9Fd+G9+KOHurTfzLm9I1PAFmxQ7flwKT9pgh40q64I0/3M8hKDSfX/pqS/A+bz9O 9bdHnmbPlHQTrZ5YNbC/aWyg49pmGZWKbd6z1EGl+NYdQRjv61Xnuf7+d0WaOJ/6 Ulp5OIQ1VjLX6NxuMSvk6APWn4n5HwboO316WRTpIyDO+qf2KBztEi7Z1grvHIm5 qcI3D/YaDSjfOH90qYKp7pFwhP66ZHxcg2/1gI9HMBrunt/xcNpsibxsRf03fMX6 ECxKTfXgR1yxEQNuczLtllBMeWER6K0d6UWqoaXtRFYlGsgAHkUb4OneqavRiFhp TGRgb22BdVKlwCMdt0TtY+zn8YEUxIqUBNe7LhySM6j9rhRnh0mzQKEW85HxY75c AVHAd0pJJv1hB0DDfYmEj4GLWqwa5Yzk0m1krSlr/SmPUi+EG4Xaq3hAsG+dQ10V qu2cceo050kGtJUzG8wK =w+JP -----END PGP SIGNATURE----- --=_6KdtFXi7O3zedMNg8yM4xn9--