From owner-freebsd-jail@freebsd.org Mon Feb 22 01:30:07 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE47EAAFD62 for ; Mon, 22 Feb 2016 01:30:06 +0000 (UTC) (envelope-from ari@ish.com.au) Received: from mail14.tpgi.com.au (smtp-out14.tpgi.com.au [220.244.226.124]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.tpg.com.au", Issuer "RapidSSL SHA256 CA - G3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 806421EF2 for ; Mon, 22 Feb 2016 01:30:05 +0000 (UTC) (envelope-from ari@ish.com.au) X-TPG-Junk-Status: Message not scanned X-TPG-Antivirus: Passed X-TPG-Abuse: host=[202.161.115.54]; ip=202.161.115.54; date=Mon, 22 Feb 2016 12:13:42 +1100 Received: from fish.ish.com.au (202-161-115-54.static.tpgi.com.au [202.161.115.54] (may be forged)) by mail14.tpgi.com.au (envelope-from ari@ish.com.au) (8.14.3/8.14.3) with ESMTP id u1M1De7m029834 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 22 Feb 2016 12:13:42 +1100 Received: from ip-136.ish.com.au ([203.29.62.136]:59374) by fish.ish.com.au with esmtpsa (TLSv1.2:DHE-RSA-AES128-SHA:128) (Exim 4.82_1-5b7a7c0-XX) (envelope-from ) id 1aXf49-0002eK-1X for freebsd-jail@freebsd.org; Mon, 22 Feb 2016 12:13:33 +1100 X-CTCH-RefID: str=0001.0A090203.56CA60BD.0032:SCFSTAT29393324, ss=1, re=-4.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 To: freebsd-jail From: Aristedes Maniatis Subject: Jail management X-Enigmail-Draft-Status: N1110 Message-ID: Date: Mon, 22 Feb 2016 12:13:32 +1100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="JEC4XxAf0MKIGgeSV80L2lcWQUnLTbIwr" X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 01:30:07 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --JEC4XxAf0MKIGgeSV80L2lcWQUnLTbIwr Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I've been using FreeBSD jails (with ezjail) for many years and they work = very well. However I'm now reaching a critical mass (30+ jails) where I w= ant to be able to manage them in bulk more easily. In this environment, each jail runs just a single application, installed = from a package built using poudriere from a custom port. That package dep= ends on Java, so lots of other packages also get pulled in. That applicat= ion gets new versions roughly once every 4 weeks. The problems I have rig= ht now are: * FreeBSD's packaging system doesn't understand the concept of installing= a particular package version, so all my scripts will by default upgrade = the application to the current version even if I don't want to. I can't e= asily install a new jail at an old version. * It is hard to reproduce the environment exactly, matching the applicati= on to the same version of Java that was available at the time of deployme= nt. Again I'm fighting against the pkg system which always wants the late= st version. * For failover I want each jail reproduced exactly on another host, or at= least a snapshot which could be sent to another host within a few second= s. The jails are quite small (< 500Mb). Most of that is just the openjdk = pkg. As I understand, ezjail doesn't support multiple base jails. If it did, t= hen I could simply install the application (and packages) to the base jai= l and have versions of the base. Then by shutting down a jail, switching = the base to the new version and starting up, everything would upgrade eas= ily. Even better would be some concept of hierarchy with customer_jail si= tting on top of base_version_1.0 which in turn sits on top of base_jail. Would I need to abandon ezjail and be able to build all the above myself = with a combination of nullfs (basejail) and unionfs (intermediate version= ed jail)? Does unionfs now work with ZFS? Alternatively I could simply use zfs clones to deploy a new version of th= e application by destroying the whole jail and replacing it with a new on= e. I'd need to then script (I use saltstack) deploying the 2-3 config fil= es which are different in each jail. Thoughts? What seems like a more robust long term approach to jail manage= ment? Thanks Ari --=20 --------------------------> Aristedes Maniatis ish http://www.ish.com.au Level 1, 30 Wilson Street Newtown 2042 Australia phone +61 2 9550 5001 fax +61 2 9550 4001 GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A --JEC4XxAf0MKIGgeSV80L2lcWQUnLTbIwr Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAlbKYL0ACgkQ72p9Lj5JECrLswCcCTh6KOLwP+1fRPFqUizxLbQ/ mcwAnjeEmBO+PgjgthrpLqRSf5KfFZS/ =+mJd -----END PGP SIGNATURE----- --JEC4XxAf0MKIGgeSV80L2lcWQUnLTbIwr-- From owner-freebsd-jail@freebsd.org Mon Feb 22 01:38:13 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9686AB0049 for ; Mon, 22 Feb 2016 01:38:13 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7AB8B13B5 for ; Mon, 22 Feb 2016 01:38:13 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x235.google.com with SMTP id g62so150788776wme.1 for ; Sun, 21 Feb 2016 17:38:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=Lk/VYytTxyvwqsOy1FAJrvSyWmId7LKWyqsBt86w77g=; b=Z+k+JlDzp0O2MwPNrRNZF35T8leQKSYHPITZ+0QnOr7hFsDYNFzeb7+Sr9g0Zn7zJh ESKgBZaJYsxUSyZ4YNbAzOh9KaBcl/ZYfCfsQvnEa2QN0xoNH1ZUwbfgUcCnUZ840Sqa Hu2vXMPDe+tcI6QhUOVfaj08aGKg8PUtLN12lbRqD+fC33OlORm3cuOrD1Mj0yS6+z5D zNWveGSCLjW/uIqXDwpy+J2RSRNZBatJcIkucma8t3UKB98Oi/X6sdXyngWZzj8Ycgq6 2jh6TtN607VruTpCSSZ9qanqLvz0CzZVraUbJeuDoFzKPYj2rwULzfhGXjKPq9K99tSD eALg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=Lk/VYytTxyvwqsOy1FAJrvSyWmId7LKWyqsBt86w77g=; b=k5mWiyejWBDFENIGB/tELBEbmIdhEapQmOWFp36ShkA/crOd5YW0lcGDyMASkWdm/A GLHbvBZryRaeq1SWAWIfqjw8jwQ7ePDSKfzrKAJ0z0YTGlE35pMHn4x9xSsP49cAzgaj UsQBePbAwAZUmwGFehrNoeNa4q6ULPoNWbENWPzVKVlt+fcyxEogMqEURaVY68uJnOHM h+42nGf4h5B5YWUOewqX4JvLQgdjKh1Wo+GugJh1XwrRScakJp/G/jll9eRlJUl+3SbP YRIyRx6PFq0j92LVvITEh4tf+nDB8Tamv4ZCjtqTQbf/0nrNN1E6HhW5nMmpz1fgSjJ1 Ye6A== X-Gm-Message-State: AG10YOSqa6+kTskG9fP4WqA/M0LIWgMhDCAblbhvv6bHfJmO7qclckX0owuVsBomVv4NNvO8 X-Received: by 10.194.92.226 with SMTP id cp2mr24196430wjb.180.1456105091763; Sun, 21 Feb 2016 17:38:11 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id jo6sm22308381wjb.48.2016.02.21.17.38.10 for (version=TLSv1/SSLv3 cipher=OTHER); Sun, 21 Feb 2016 17:38:10 -0800 (PST) Subject: Re: Jail management To: freebsd-jail@freebsd.org References: From: Steven Hartland Message-ID: <56CA6685.4030705@multiplay.co.uk> Date: Mon, 22 Feb 2016 01:38:13 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 01:38:14 -0000 Checkout qjail from your description I think it will do what you want. On 22/02/2016 01:13, Aristedes Maniatis wrote: > I've been using FreeBSD jails (with ezjail) for many years and they work very well. However I'm now reaching a critical mass (30+ jails) where I want to be able to manage them in bulk more easily. > > In this environment, each jail runs just a single application, installed from a package built using poudriere from a custom port. That package depends on Java, so lots of other packages also get pulled in. That application gets new versions roughly once every 4 weeks. The problems I have right now are: > > * FreeBSD's packaging system doesn't understand the concept of installing a particular package version, so all my scripts will by default upgrade the application to the current version even if I don't want to. I can't easily install a new jail at an old version. > > * It is hard to reproduce the environment exactly, matching the application to the same version of Java that was available at the time of deployment. Again I'm fighting against the pkg system which always wants the latest version. > > * For failover I want each jail reproduced exactly on another host, or at least a snapshot which could be sent to another host within a few seconds. The jails are quite small (< 500Mb). Most of that is just the openjdk pkg. > > > As I understand, ezjail doesn't support multiple base jails. If it did, then I could simply install the application (and packages) to the base jail and have versions of the base. Then by shutting down a jail, switching the base to the new version and starting up, everything would upgrade easily. Even better would be some concept of hierarchy with customer_jail sitting on top of base_version_1.0 which in turn sits on top of base_jail. > > Would I need to abandon ezjail and be able to build all the above myself with a combination of nullfs (basejail) and unionfs (intermediate versioned jail)? Does unionfs now work with ZFS? > > > Alternatively I could simply use zfs clones to deploy a new version of the application by destroying the whole jail and replacing it with a new one. I'd need to then script (I use saltstack) deploying the 2-3 config files which are different in each jail. > > > > Thoughts? What seems like a more robust long term approach to jail management? > > > Thanks > Ari > > > From owner-freebsd-jail@freebsd.org Mon Feb 22 02:05:02 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4FA9AAB09B8 for ; Mon, 22 Feb 2016 02:05:02 +0000 (UTC) (envelope-from erdgeist@erdgeist.org) Received: from elektropost.org (elektropost.org [217.115.13.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 95E0311E2 for ; Mon, 22 Feb 2016 02:05:01 +0000 (UTC) (envelope-from erdgeist@erdgeist.org) Received: (qmail 54202 invoked from network); 22 Feb 2016 01:58:16 -0000 Received: from elektropost.org (HELO elektropost.org) (erdgeist@erdgeist.org) by elektropost.org with ESMTPS (DHE-RSA-AES256-SHA encrypted); 22 Feb 2016 01:58:16 -0000 Subject: Re: Jail management Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Content-Type: multipart/signed; boundary="Apple-Mail=_E05654EB-7CA4-426C-8D6C-7F83677712F7"; protocol="application/pgp-signature"; micalg=pgp-sha1 X-Pgp-Agent: GPGMail 2.6b2 From: erdgeist In-Reply-To: Date: Mon, 22 Feb 2016 14:57:49 +1300 Cc: freebsd-jail Message-Id: References: To: Aristedes Maniatis X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 02:05:02 -0000 --Apple-Mail=_E05654EB-7CA4-426C-8D6C-7F83677712F7 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On 22 Feb 2016, at 14:13, Aristedes Maniatis wrote: >=20 > Thoughts? What seems like a more robust long term approach to jail = management? Take a look at bsdploy https://github.com/ployground/bsdploy or just = come and ask ezjails author. ;) Also unionfs does not work very stable. erdgeist --Apple-Mail=_E05654EB-7CA4-426C-8D6C-7F83677712F7 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAlbKayYACgkQk9uKpYuqISTl0QCgp9y9iS8imutHzQQIBM2D51qy tF8AoLknsj/nlYjoM7Y0nehtkg2CqfTr =M8VL -----END PGP SIGNATURE----- --Apple-Mail=_E05654EB-7CA4-426C-8D6C-7F83677712F7-- From owner-freebsd-jail@freebsd.org Mon Feb 22 02:30:05 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3DE7BAAF611 for ; Mon, 22 Feb 2016 02:30:05 +0000 (UTC) (envelope-from ari@ish.com.au) Received: from mail13.tpgi.com.au (smtp-out13.tpgi.com.au [220.244.226.123]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.tpg.com.au", Issuer "RapidSSL SHA256 CA - G3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D013F1F33 for ; Mon, 22 Feb 2016 02:30:04 +0000 (UTC) (envelope-from ari@ish.com.au) X-TPG-Junk-Status: Message not scanned X-TPG-Abuse: host=[202.161.115.54]; ip=202.161.115.54; date=Mon, 22 Feb 2016 13:18:22 +1100 Received: from fish.ish.com.au (202-161-115-54.static.tpgi.com.au [202.161.115.54] (may be forged)) by mail13.tpgi.com.au (envelope-from ari@ish.com.au) (8.14.3/8.14.3) with ESMTP id u1M2IK3n013162 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 22 Feb 2016 13:18:22 +1100 Received: from ip-136.ish.com.au ([203.29.62.136]:59775) by fish.ish.com.au with esmtpsa (TLSv1.2:DHE-RSA-AES128-SHA:128) (Exim 4.82_1-5b7a7c0-XX) (envelope-from ) id 1aXg4g-0006dv-25; Mon, 22 Feb 2016 13:18:10 +1100 X-CTCH-RefID: str=0001.0A150206.56CA6FE2.00C4:SCFSTAT29393324, ss=1, re=-4.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 Subject: Re: Jail management To: erdgeist References: Cc: freebsd-jail From: Aristedes Maniatis X-Enigmail-Draft-Status: N1110 Message-ID: <0f5cae7e-7de3-2617-fcf6-3423d4caf13a@ish.com.au> Date: Mon, 22 Feb 2016 13:18:09 +1100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="t6cg0w4f0SCP2auHAqR3oVxhAg4e3gc1d" X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 02:30:05 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --t6cg0w4f0SCP2auHAqR3oVxhAg4e3gc1d Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 22/02/2016 12:57pm, erdgeist wrote: >=20 >> On 22 Feb 2016, at 14:13, Aristedes Maniatis wrote: >> >> Thoughts? What seems like a more robust long term approach to jail man= agement? >=20 > Take a look at bsdploy https://github.com/ployground/bsdploy or just co= me and ask ezjails author. ;) Hello there! Thanks for ezjail: a very useful tool for avoiding the pain = of setting up nullfs and friends. However I think that bsdploy is orthogonal to my problem. I'm already emb= edded in saltstack, so moving to ansible doesn't solve any problems for m= e. And I can't see how it solves the pkg versioning problem any better. That's why I was thinking to move to a snapshot clone/restore approach to= jail management. But that idea butts up against ezjail's assumptions. > Also unionfs does not work very stable. OK, I'll cross that option off my list. That then leaves just ZFS clone a= s the way to create a reproducible and deployable jail environment with t= he correct (old) package versions. I did have another idea: create a poudriere environment for each version = of the app and switch /usr/local/etc/pkg/repos/my.conf each time. But tha= t seems awkward and still very hard to go back in time and apply small fi= xes to an old deployed version. Have I just now outgrown ezjail and should set off on my own? I'm afraid = of how I'd go about upgrading the basejail for new FreeBSD host versions = without your tool :-) Thanks Ari --=20 --------------------------> Aristedes Maniatis ish http://www.ish.com.au Level 1, 30 Wilson Street Newtown 2042 Australia phone +61 2 9550 5001 fax +61 2 9550 4001 GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A --t6cg0w4f0SCP2auHAqR3oVxhAg4e3gc1d Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAlbKb+EACgkQ72p9Lj5JECrzgQCcCkNYJYydfF5aZjyZWTPLejep eCoAn3Co69zVuRyi2S5H6msgJKul6QQI =0eFD -----END PGP SIGNATURE----- --t6cg0w4f0SCP2auHAqR3oVxhAg4e3gc1d-- From owner-freebsd-jail@freebsd.org Mon Feb 22 03:21:28 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E8C9AB0938 for ; Mon, 22 Feb 2016 03:21:28 +0000 (UTC) (envelope-from markham@ssimicro.com) Received: from mail.ssimicro.com (mail.ssimicro.com [64.247.129.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.ssimicro.com", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CA5111938 for ; Mon, 22 Feb 2016 03:21:27 +0000 (UTC) (envelope-from markham@ssimicro.com) Received: from Markhams-MacBook-Pro.local (64-247-134-200.ssimicro.com [64.247.134.200]) (authenticated bits=0) by mail.ssimicro.com (8.14.7/8.14.7) with ESMTP id u1M3BpXF019568 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Sun, 21 Feb 2016 20:11:52 -0700 (MST) Subject: Re: Jail management To: freebsd-jail References: From: markham breitbach X-Enigmail-Draft-Status: N1110 Message-ID: <56CA7D5F.7060709@ssimicro.com> Date: Sun, 21 Feb 2016 20:15:43 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="V1rgdSU3Ftg6Ri1jerCssFQcPFKq2RehU" X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 03:21:28 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --V1rgdSU3Ftg6Ri1jerCssFQcPFKq2RehU Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable One of the solutions I have found to the version issue is to build my own package repo. I build the packages the way I want, and then upload them to my own package repo (which is just another jail running thttpd). I also keep a jail running with the ports tree frozen at the versions I am using for production. Add the following to /usr/local/etc/pkg.conf repos_dir: [ "/usr/local/etc/pkg/repos", "/etc/pkg", ] The tells pkg to look in your private repo first Then, create /usr/local/etc/pkg/repos/private.conf private: { url: "pkg+http://pkg.ssimicro.com/${ABI}/latest", enabled: true, signature_type: "PUBKEY", PUBKEY: "/usr/local/etc/pkg/repos/ssi.pub", mirror_type: "srv" } Note: you also need to create a public/private key pair for this using openssl. I don't recall the specifics though, but it looks like a pretty standard self-signed key/cert pair. The private key is stored on the repo and used to sign the packages when you add initialize the repo: pkg repo /home/pkg/repo/freebsd:10:x86:64/latest /home/pkg/repo.key Best, -Markham On 2016-02-21 6:13 PM, Aristedes Maniatis wrote: > I've been using FreeBSD jails (with ezjail) for many years and they wor= k very well. However I'm now reaching a critical mass (30+ jails) where I= want to be able to manage them in bulk more easily. > > In this environment, each jail runs just a single application, installe= d from a package built using poudriere from a custom port. That package d= epends on Java, so lots of other packages also get pulled in. That applic= ation gets new versions roughly once every 4 weeks. The problems I have r= ight now are: > > * FreeBSD's packaging system doesn't understand the concept of installi= ng a particular package version, so all my scripts will by default upgrad= e the application to the current version even if I don't want to. I can't= easily install a new jail at an old version. > > * It is hard to reproduce the environment exactly, matching the applica= tion to the same version of Java that was available at the time of deploy= ment. Again I'm fighting against the pkg system which always wants the la= test version. > > * For failover I want each jail reproduced exactly on another host, or = at least a snapshot which could be sent to another host within a few seco= nds. The jails are quite small (< 500Mb). Most of that is just the openjd= k pkg. > > > As I understand, ezjail doesn't support multiple base jails. If it did,= then I could simply install the application (and packages) to the base j= ail and have versions of the base. Then by shutting down a jail, switchin= g the base to the new version and starting up, everything would upgrade e= asily. Even better would be some concept of hierarchy with customer_jail = sitting on top of base_version_1.0 which in turn sits on top of base_jail= =2E > > Would I need to abandon ezjail and be able to build all the above mysel= f with a combination of nullfs (basejail) and unionfs (intermediate versi= oned jail)? Does unionfs now work with ZFS? > > > Alternatively I could simply use zfs clones to deploy a new version of = the application by destroying the whole jail and replacing it with a new = one. I'd need to then script (I use saltstack) deploying the 2-3 config f= iles which are different in each jail. > > > > Thoughts? What seems like a more robust long term approach to jail mana= gement? > > > Thanks > Ari > > > --V1rgdSU3Ftg6Ri1jerCssFQcPFKq2RehU Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2 Comment: GPGTools - https://gpgtools.org iEYEARECAAYFAlbKfWMACgkQKQ+fauj+jf6LuQCg1N3GkStrpq8jdLe7UmVRInfC R9YAoLGk9n1FZJ1F79MXK5cqHOKIWbO0 =GzxB -----END PGP SIGNATURE----- --V1rgdSU3Ftg6Ri1jerCssFQcPFKq2RehU-- From owner-freebsd-jail@freebsd.org Mon Feb 22 08:17:57 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6A8DAAFC9D for ; Mon, 22 Feb 2016 08:17:56 +0000 (UTC) (envelope-from ari@ish.com.au) Received: from mail13.tpgi.com.au (mail13.tpgi.com.au [203.12.160.181]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client CN "*.tpg.com.au", Issuer "RapidSSL SHA256 CA - G3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 889941CCA for ; Mon, 22 Feb 2016 08:17:55 +0000 (UTC) (envelope-from ari@ish.com.au) X-TPG-Junk-Status: Message not scanned X-TPG-Abuse: host=[202.161.115.54]; ip=202.161.115.54; date=Mon, 22 Feb 2016 19:17:52 +1100 Received: from fish.ish.com.au (202-161-115-54.static.tpgi.com.au [202.161.115.54] (may be forged)) by mail13.tpgi.com.au (envelope-from ari@ish.com.au) (8.14.3/8.14.3) with ESMTP id u1M8HohV025080 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 22 Feb 2016 19:17:52 +1100 Received: from ip-136.ish.com.au ([203.29.62.136]:61597) by fish.ish.com.au with esmtpsa (TLSv1.2:DHE-RSA-AES128-SHA:128) (Exim 4.82_1-5b7a7c0-XX) (envelope-from ) id 1aXlgb-000566-1f; Mon, 22 Feb 2016 19:17:42 +1100 X-CTCH-RefID: str=0001.0A150202.56CAC425.02E7:SCFSTAT29393324, ss=1, re=-4.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 Subject: Re: Jail management To: markham breitbach References: Cc: freebsd-jail From: Aristedes Maniatis X-Enigmail-Draft-Status: N1110 Message-ID: <7b947a1c-824b-193d-3dc3-49d876b21be9@ish.com.au> Date: Mon, 22 Feb 2016 19:17:40 +1100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="kejBCIdMMi0t6KHUxxURT5nll3bLgg39w" X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 08:17:57 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --kejBCIdMMi0t6KHUxxURT5nll3bLgg39w Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Markham wrote: > One of the solutions I have found to the version issue is to build my o= wn package repo. I build the packages the way I want, and then upload the= m to my own package repo (which is just another jail running thttpd). I a= lso keep a jail running with the ports tree frozen at the versions I am u= sing for production. Thanks for that idea. However I'm already doing all that with poudriere a= nd it works well. However creating a new poudriere repo every 4 weeks wou= ld be a little cumbersome. Steve Hartland wrote: > Checkout qjail from your description I think it will do what you want. I took a look at the documentation I could find (just some stuff on Sourc= eforge really) but nothing in qjail seemed to solve the issues of multipl= e basejails or anything else that was causing me issues with ezjail. I also discovered iocage which looks quite different and interesting. I'm= still reading about it, but it seems to: * have multiple basejails * use unionfs to create a "jail package" which looks like an overlay on a= jail. However there doesn't appear to be a feature to "undeploy" a packa= ge, so not sure if it is the best way to deploy a certain version of an a= pplication. * have also a template feature which looks like the ezjail 'flavour'. You= can't change the template after you make a jail. Again, it looks like "d= estroy jail" and make a new one. I can't really understand the different practical use cases of 'package' = and 'template' since they seem both very similar to ezjail flavours excep= t in how you create them in the first place. But the multiple basejail idea might just be what I need. I create a new = basejail once a month with each new release of the software. I can't swit= ch existing jails to a new basejail (I think), but I'd need to destroy an= d recreate it from a new base and add my jail specific config bits. Each jail could be upgraded as needed and not necessarily at the same tim= e. Nice. Ari --=20 --------------------------> Aristedes Maniatis ish http://www.ish.com.au Level 1, 30 Wilson Street Newtown 2042 Australia phone +61 2 9550 5001 fax +61 2 9550 4001 GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A --kejBCIdMMi0t6KHUxxURT5nll3bLgg39w Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAlbKxCQACgkQ72p9Lj5JECo7EQCfT46hq9aA62pBAbWVvjHQPe7j XAIAn3sxocIRXj0E42stY6JJgp80DvCh =9+Cj -----END PGP SIGNATURE----- --kejBCIdMMi0t6KHUxxURT5nll3bLgg39w-- From owner-freebsd-jail@freebsd.org Mon Feb 22 09:28:19 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1D091AAFDF0 for ; Mon, 22 Feb 2016 09:28:19 +0000 (UTC) (envelope-from lists@tomster.org) Received: from smtp.cachexia.de (smtp.cachexia.de [46.101.204.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AEB401344 for ; Mon, 22 Feb 2016 09:28:17 +0000 (UTC) (envelope-from lists@tomster.org) Received: (qmail 52100 invoked from network); 22 Feb 2016 09:28:08 -0000 Received: from unknown (HELO smiley-3.fritz.box) (tom@tomster.org@77.180.194.85) by smtp.cachexia.de with ESMTPS (DHE-RSA-AES256-SHA encrypted); 22 Feb 2016 09:28:08 -0000 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: Jail management From: Tom Lazar In-Reply-To: <7b947a1c-824b-193d-3dc3-49d876b21be9@ish.com.au> Date: Mon, 22 Feb 2016 10:28:07 +0100 Cc: markham breitbach , freebsd-jail Message-Id: <13A9C47A-86FE-4E44-83D6-4736488FB9CC@tomster.org> References: <7b947a1c-824b-193d-3dc3-49d876b21be9@ish.com.au> To: Aristedes Maniatis X-Mailer: Apple Mail (2.3112) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 09:28:19 -0000 > On 22 Feb 2016, at 09:17, Aristedes Maniatis wrote: >=20 > Markham wrote: >=20 > I also discovered iocage which looks quite different and interesting. = I'm still reading about it, but it seems to: another thing you might want to take a look at - given your requirements = and current setup - is jetpack[1] it basically implements the docker approach using zfs and jails as = underlying technology and pretty much replaces (the unstable) solution = of unionfs with its layers based on zfs snapshots. while it seems to be the least mature option discussed in this thread so = far, i think its container approach fills a niche that might fit your = use case very well. having said that, i=92d like to point out, that florian and myself (the = authors of bsdploy) are very open to using saltstack - bsdploy is = designed to be modular and we already have experimental support for it = [2] and the GPL licence of ansible is turning into a bigger annoyance = than expected[3] so we are motivated to continue along that path. just my two cents, cheers, tom =20 [1] https://github.com/3ofcoins/jetpack [2] https://github.com/ployground/ploy_salt = [3] https://github.com/ployground/bsdploy/issues/75= From owner-freebsd-jail@freebsd.org Mon Feb 22 10:00:16 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D7C9DAB0D28 for ; Mon, 22 Feb 2016 10:00:16 +0000 (UTC) (envelope-from ari@ish.com.au) Received: from mail13.tpgi.com.au (mail13.tpgi.com.au [203.12.160.181]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client CN "*.tpg.com.au", Issuer "RapidSSL SHA256 CA - G3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 775D71745 for ; Mon, 22 Feb 2016 10:00:15 +0000 (UTC) (envelope-from ari@ish.com.au) X-TPG-Junk-Status: Message not scanned X-TPG-Abuse: host=[202.161.115.54]; ip=202.161.115.54; date=Mon, 22 Feb 2016 21:00:12 +1100 Received: from fish.ish.com.au (202-161-115-54.static.tpgi.com.au [202.161.115.54] (may be forged)) by mail13.tpgi.com.au (envelope-from ari@ish.com.au) (8.14.3/8.14.3) with ESMTP id u1MA0A0u027475 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 22 Feb 2016 21:00:12 +1100 Received: from ip-136.ish.com.au ([203.29.62.136]:62849) by fish.ish.com.au with esmtpsa (TLSv1.2:DHE-RSA-AES128-SHA:128) (Exim 4.82_1-5b7a7c0-XX) (envelope-from ) id 1aXnHe-0003pZ-0H; Mon, 22 Feb 2016 21:00:02 +1100 X-CTCH-RefID: str=0001.0A150207.56CADC22.0108:SCFSTAT29393324, ss=1, re=-4.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 Subject: Re: Jail management To: Tom Lazar References: <7b947a1c-824b-193d-3dc3-49d876b21be9@ish.com.au> <13A9C47A-86FE-4E44-83D6-4736488FB9CC@tomster.org> Cc: markham breitbach , freebsd-jail From: Aristedes Maniatis X-Enigmail-Draft-Status: N1110 Message-ID: <20af917f-78c1-5a38-df36-6d8749377cc3@ish.com.au> Date: Mon, 22 Feb 2016 21:00:00 +1100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: <13A9C47A-86FE-4E44-83D6-4736488FB9CC@tomster.org> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="gDmNOn2ARHscJKVkrEGxjiqlhu3fa2w3D" X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 10:00:16 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --gDmNOn2ARHscJKVkrEGxjiqlhu3fa2w3D Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 22/02/2016 8:28pm, Tom Lazar wrote: >=20 >> On 22 Feb 2016, at 09:17, Aristedes Maniatis > wrote: >> >> Markham wrote: >> >> I also discovered iocage which looks quite different and interesting. = I'm still reading about it, but it seems to: >=20 > another thing you might want to take a look at - given your requirement= s and current setup - is jetpack[1] >=20 > it basically implements the docker approach using zfs and jails as unde= rlying technology and pretty much replaces (the unstable) solution of uni= onfs with its layers based on zfs snapshots. >=20 > while it seems to be the least mature option discussed in this thread s= o far, i think its container approach fills a niche that might fit your u= se case very well. Very interesting indeed. Thanks for that pointer. However, I think I'm st= ill on the fence about docker (and friends). It looks like a complex solu= tion to independent problems (bundling, jails, snapshots, configuration m= anagement). > having said that, i=92d like to point out, that florian and myself (the= authors of bsdploy) are very open to using saltstack - bsdploy is design= ed to be modular and we already have experimental support for it [2] and = the GPL licence of ansible is turning into a bigger annoyance than expect= ed[3] so we are motivated to continue along that path. Great, I think you'll like salt although it has a very steep initial lear= ning curve. I'm happy with my choice of saltstack and it appears to have = a couple of people contributing FreeBSD improvements reasonably regularly= =2E pkg support is pretty good now and it has limited jail support. The b= iggest issue I've found with salt is that there is no recommended best-pr= actices way of using it. Its like being given a shed full of wonderful to= ols and being told to build a house. But at this point I think my problem looks like a thin layer on top of ja= ils rather than something bigger. I still need to try more things and I j= ust found this which looks like a nice way to easily control iocage: https://github.com/bougie/salt-iocage-formula Maybe my workflow is: * destroy jail * create new jail from new template (with new version of app) * use salt to inject the little config files * start jail That means I lose all logs and other things at each upgrade, but with log= stash that's less of a problem than it was. On top of that I need a mechanism to create the jail templates, but somet= hing manual with FreeBSD pkg might be enough there. If I avoid the iocage 'packaging' thing then it looks like I avoid the un= ionfs which several people have warned about not being stable. Ari > just my two cents, >=20 > cheers, >=20 > tom > =20 > [1] https://github.com/3ofcoins/jetpack > [2] https://github.com/ployground/ploy_salt > [3] https://github.com/ployground/bsdploy/issues/75 --=20 --------------------------> Aristedes Maniatis ish http://www.ish.com.au Level 1, 30 Wilson Street Newtown 2042 Australia phone +61 2 9550 5001 fax +61 2 9550 4001 GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A --gDmNOn2ARHscJKVkrEGxjiqlhu3fa2w3D Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iEUEARECAAYFAlbK3CEACgkQ72p9Lj5JECrhGgCY6mS3YBbwzezquw8ea5UO0sOV UQCfdwvC4CRcMbNG9fO/3hE8uJphbZ8= =Dn4N -----END PGP SIGNATURE----- --gDmNOn2ARHscJKVkrEGxjiqlhu3fa2w3D-- From owner-freebsd-jail@freebsd.org Mon Feb 22 10:57:03 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BCB79AB0D14 for ; Mon, 22 Feb 2016 10:57:03 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 83B8F1CBC for ; Mon, 22 Feb 2016 10:57:02 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id E26BA2840C; Mon, 22 Feb 2016 11:56:53 +0100 (CET) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 11C7928417; Mon, 22 Feb 2016 11:56:53 +0100 (CET) Message-ID: <56CAE974.4050508@quip.cz> Date: Mon, 22 Feb 2016 11:56:52 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: Aristedes Maniatis CC: freebsd-jail Subject: Re: Jail management References: <0f5cae7e-7de3-2617-fcf6-3423d4caf13a@ish.com.au> In-Reply-To: <0f5cae7e-7de3-2617-fcf6-3423d4caf13a@ish.com.au> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 10:57:03 -0000 Aristedes Maniatis wrote on 02/22/2016 03:18: [...] > Have I just now outgrown ezjail and should set off on my own? I'm afraid of how I'd go about upgrading the basejail for new FreeBSD host versions without your tool :-) I don't know your environment and your FreeBSD jails skills but it seems you think jails are something complex and "magic". It is not. Managing jail by "hand" (own simple tools and scripts) is really simple and straight forward. Creating new base jail is just 'make installworld DESTDIR=/vol/jail/_basejail_XYZ' (or extracting base.txz installation archive) Movin your old jail to newer basejail is metter of change in fstab file where you will change path to new basejail. Once you tried it you will found how simple it is to write some own script perfectly fitting your needs. It is just a file manipulation - installing, unpacking, movin, deleting. Nothing more. As time and projects passed by, I had scripts to create+update jail from FTP, or from NFS mounted src and obj (by make installworld), or unpacking TGZ archive, or updated by rsyncing fails from hosts base or another directory... There are so many ways you can do this and I don't think you will find any existing tool fitting all your needs. Just don't be afraid of writing simple shell scripts :) For your problem with installing old versions of packages - I think you are still able to install whatever version you need if you have it locally on disk. Then you can use "pkg install my-package-1.2.3.txz" (you need all dependencies as well) Miroslav Lachman From owner-freebsd-jail@freebsd.org Mon Feb 22 11:26:14 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13D99AB18A4 for ; Mon, 22 Feb 2016 11:26:14 +0000 (UTC) (envelope-from ari@ish.com.au) Received: from mail13.tpgi.com.au (mail13.tpgi.com.au [203.12.160.181]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client CN "*.tpg.com.au", Issuer "RapidSSL SHA256 CA - G3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A83B91CA4 for ; Mon, 22 Feb 2016 11:26:12 +0000 (UTC) (envelope-from ari@ish.com.au) X-TPG-Junk-Status: Message not scanned X-TPG-Abuse: host=[202.161.115.54]; ip=202.161.115.54; date=Mon, 22 Feb 2016 22:26:09 +1100 Received: from fish.ish.com.au (202-161-115-54.static.tpgi.com.au [202.161.115.54] (may be forged)) by mail13.tpgi.com.au (envelope-from ari@ish.com.au) (8.14.3/8.14.3) with ESMTP id u1MBQ70q022002 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 22 Feb 2016 22:26:09 +1100 Received: from [10.242.2.22] (port=63340 helo=Aristedess-MacBook-Pro.local) by fish.ish.com.au with esmtpsa (TLSv1.2:DHE-RSA-AES128-SHA:128) (Exim 4.82_1-5b7a7c0-XX) (envelope-from ) id 1aXocu-0001Kb-1a; Mon, 22 Feb 2016 22:26:05 +1100 X-CTCH-RefID: str=0001.0A150205.56CAF04C.0241, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 Subject: Re: Jail management To: Miroslav Lachman <000.fbsd@quip.cz> References: <0f5cae7e-7de3-2617-fcf6-3423d4caf13a@ish.com.au> <56CAE974.4050508@quip.cz> Cc: freebsd-jail From: Aristedes Maniatis X-Enigmail-Draft-Status: N1110 Message-ID: <0eaf61d4-43e6-265a-f773-820244fc8931@ish.com.au> Date: Mon, 22 Feb 2016 22:26:02 +1100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: <56CAE974.4050508@quip.cz> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="QV4vnOEE2DruVug8s4bvnha6RXhU9n75K" X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 11:26:14 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --QV4vnOEE2DruVug8s4bvnha6RXhU9n75K Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 22/02/2016 9:56pm, Miroslav Lachman wrote: > I don't know your environment and your FreeBSD jails skills but it seem= s you think jails are something complex and "magic". It is not. =2E.. > Just don't be afraid of writing simple shell scripts :) You are right, and perhaps I should just bite the bullet. I am afraid of = only two things. * upgrade the basejail with FreeBSD upgrades. I am sure this is a simple = bit of chroot magic, but freebsd-update is a bit of a black box to me. * nullfs. I've never used it before and need to play with it more As for shell scripts: my only goal in life is to write *fewer* shell scri= pts. My adoption of saltstack was spurred by shell everywhere, mostly not= under version control. So less shell and more python centrally managed a= nd versioned is my dream. > For your problem with installing old versions of packages - I think you= are still able to install whatever version you need if you have it local= ly on disk. Then you can use "pkg install my-package-1.2.3.txz" > (you need all dependencies as well) I think 'pkg add' is needed there, but dependencies are very complicated = to make work in this way. And I need a new package distribution model not= based on the standard repo that poudriere makes. Which is why I started = this whole exploration. Thanks for your ideas. I should be less afraid of letting go of the jail = management tools. Ari --=20 --------------------------> Aristedes Maniatis ish http://www.ish.com.au Level 1, 30 Wilson Street Newtown 2042 Australia phone +61 2 9550 5001 fax +61 2 9550 4001 GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A --QV4vnOEE2DruVug8s4bvnha6RXhU9n75K Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAlbK8EsACgkQ72p9Lj5JECpKWwCfR0M4pQ4uzjaLySj74V2Iefcy T7QAn1D5glNJIYx9+pnDcwnrUNaD7OoS =ciXv -----END PGP SIGNATURE----- --QV4vnOEE2DruVug8s4bvnha6RXhU9n75K-- From owner-freebsd-jail@freebsd.org Mon Feb 22 11:34:15 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A3BADAB1BA9 for ; Mon, 22 Feb 2016 11:34:15 +0000 (UTC) (envelope-from lists@opsec.eu) Received: from home.opsec.eu (home.opsec.eu [IPv6:2001:14f8:200::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6AE531FE2 for ; Mon, 22 Feb 2016 11:34:15 +0000 (UTC) (envelope-from lists@opsec.eu) Received: from pi by home.opsec.eu with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1aXokm-0008h5-Sv; Mon, 22 Feb 2016 12:34:12 +0100 Date: Mon, 22 Feb 2016 12:34:12 +0100 From: Kurt Jaeger To: Aristedes Maniatis Cc: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-jail Subject: Re: Jail management Message-ID: <20160222113412.GB26283@home.opsec.eu> References: <0f5cae7e-7de3-2617-fcf6-3423d4caf13a@ish.com.au> <56CAE974.4050508@quip.cz> <0eaf61d4-43e6-265a-f773-820244fc8931@ish.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <0eaf61d4-43e6-265a-f773-820244fc8931@ish.com.au> X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 11:34:15 -0000 Hi! > * upgrade the basejail with FreeBSD upgrades. I am sure this is > a simple bit of chroot magic, but freebsd-update is a bit of a black > box to me. I use this script. My jails are in /vserv//. ------------- #!/usr/local/bin/bash if [ X$1 = 'X' ] then echo "usage: $0 vserv" echo " to update vservs with 10.1" exit 1 fi host=$1 UNAME_r=10.1-RELEASE-p6 export UNAME_r set -x freebsd-update -b /vserv/$host fetch freebsd-update -b /vserv/$host install ------------- -- pi@opsec.eu +49 171 3101372 4 years to go ! From owner-freebsd-jail@freebsd.org Mon Feb 22 11:57:14 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1E701AAF526 for ; Mon, 22 Feb 2016 11:57:14 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B69E11C9F for ; Mon, 22 Feb 2016 11:57:13 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 5FC9C28412; Mon, 22 Feb 2016 12:57:09 +0100 (CET) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 20B672840C; Mon, 22 Feb 2016 12:57:08 +0100 (CET) Message-ID: <56CAF793.2030104@quip.cz> Date: Mon, 22 Feb 2016 12:57:07 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: Aristedes Maniatis CC: freebsd-jail Subject: Re: Jail management References: <0f5cae7e-7de3-2617-fcf6-3423d4caf13a@ish.com.au> <56CAE974.4050508@quip.cz> <0eaf61d4-43e6-265a-f773-820244fc8931@ish.com.au> In-Reply-To: <0eaf61d4-43e6-265a-f773-820244fc8931@ish.com.au> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 11:57:14 -0000 Aristedes Maniatis wrote on 02/22/2016 12:26: > On 22/02/2016 9:56pm, Miroslav Lachman wrote: >> I don't know your environment and your FreeBSD jails skills but it seems you think jails are something complex and "magic". It is not. > ... >> Just don't be afraid of writing simple shell scripts :) > > > You are right, and perhaps I should just bite the bullet. I am afraid of only two things. > > * upgrade the basejail with FreeBSD upgrades. I am sure this is a simple bit of chroot magic, but freebsd-update is a bit of a black box to me. I tried it few years ago and it had some problems that doesn't fit well in to my environment, then I moved all our servers to own buildserver with make buildkernel + buildworld and then installworld through NFS in to destionation. Faster, safer and predictable solution. (I had problems with freebsd-update even on bare metal systems, not in jails) > * nullfs. I've never used it before and need to play with it more Nullfs is easy. You can "mount" one directory to another. If you have /vol0/jail/_basejail and jails in /vol0/jail/alpha, /vol0/jail/beta Then you can do mkdir /vol0/jail/alpha/basejail mkdir /vol0/jail/beta/basejail mount -t nullfs /vol0/jail/_basejail /vol0/jail/alpha/basejail mount -t nullfs /vol0/jail/_basejail /vol0/jail/beta/basejail Your basejail contains # ls -1 /vol0/jail/_basejail UPDATED bin boot lib libexec rescue sbin usr an jails (alpha, beta and you new jail template) contains symlinks to these directories # ls -lg /vol0/jail/alpha/ -rw-r--r-- 1 root wheel 798 Jan 13 2015 .cshrc -rw-r--r-- 2 root wheel 265 Jan 13 2015 .profile -r--r--r-- 1 root wheel 6197 May 12 2015 COPYRIGHT drwxr-xr-x 9 root wheel 10 May 12 2015 basejail lrwxr-xr-x 1 root wheel 13 Jan 13 2015 bin -> /basejail/bin lrwxr-xr-x 1 root wheel 14 Jan 13 2015 boot -> /basejail/boot dr-xr-xr-x 7 root wheel 512 Oct 18 17:52 dev lrwxr-xr-x 1 root wheel 12 Jan 20 2015 develop -> /usr/develop drwxr-xr-x 20 root wheel 105 Nov 12 19:37 etc lrwxr-xr-x 1 root wheel 8 Jan 13 2015 home -> usr/home lrwxr-xr-x 1 root wheel 13 Jan 13 2015 lib -> /basejail/lib lrwxr-xr-x 1 root wheel 17 Jan 13 2015 libexec -> /basejail/libexec dr-xr-xr-x 2 root wheel 2 Jan 13 2015 proc lrwxr-xr-x 1 root wheel 16 Jan 13 2015 rescue -> /basejail/rescue drwxr-xr-x 10 root wheel 29 May 12 2015 root lrwxr-xr-x 1 root wheel 14 Jan 13 2015 sbin -> /basejail/sbin lrwxr-xr-x 1 root wheel 11 Jan 13 2015 sys -> usr/src/sys drwxrwxrwt 9 root wheel 10 Feb 22 03:43 tmp drwxr-xr-x 7 root wheel 17 Jan 20 2015 usr drwxr-xr-x 22 root wheel 22 Oct 18 17:52 var Nullfs mounts can be specified in fstab files # cat /etc/fstab.alpha /vol0/jail/_basejail /vol0/jail/alpha/basejail nullfs ro 0 0 # cat /etc/fstab.beta /vol0/jail/_basejail /vol0/jail/beta/basejail nullfs ro 0 0 So if jails are running, you wil see this tank/vol0/jail/alpha on /vol0/jail/alpha (zfs, local, noatime, nfsv4acls) tank/vol0/jail/beta on /vol0/jail/beta (zfs, local, noatime, nfsv4acls) /vol0/jail/_basejail on /vol0/jail/alpha/basejail (nullfs, local, read-only) /vol0/jail/_basejail on /vol0/jail/beta/basejail (nullfs, local, read-only) And you can have gamma with another basejail called _basejail93 mounted as tank/vol0/jail/gamma on /vol0/jail/gamma (zfs, local, noatime, nfsv4acls) /vol0/jail/_basejail93 on /vol0/jail/gamma/basejail (nullfs, local, read-only) Migrate this jail to _basejail is just a matter of change one line if fstab.gamma All commong settings are in /etc/jail.conf It can be something like this ## Typical static defaults: ## Use the rc scripts to start and stop jails. Mount jail's /dev. exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.clean; exec.system_user = "root"; exec.jail_user = "root"; mount.devfs; devfs_ruleset = 4; enforce_statfs = 1; #allow.set_hostname = false; #allow.mount; allow.set_hostname = 0; allow.sysvipc = 0; allow.raw_sockets = 0; ## Dynamic wildcard parameter: ## Base the path off the jail name. path = "/vol0/jail/$name"; exec.consolelog = "/var/log/jail/$name.console"; mount.fstab = "/etc/fstab.$name"; ## Alpha alpha { host.hostname = "alpha.example.com"; ip4.addr = 10.10.10.20; allow.sysvipc = 1; } ## Beta beta { host.hostname = "beta.example.com"; ip4.addr = 10.10.10.30; } ## Gamma gamma { host.hostname = "gamma.example.com"; ip4.addr = 10.10.10.40; } > As for shell scripts: my only goal in life is to write *fewer* shell scripts. My adoption of saltstack was spurred by shell everywhere, mostly not under version control. So less shell and more python centrally managed and versioned is my dream. I understand this approach. You can look at it as your own port (package) and not as unversioned shell script. :) Miroslav Lachman From owner-freebsd-jail@freebsd.org Mon Feb 22 12:47:11 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CAF5BAB097D for ; Mon, 22 Feb 2016 12:47:11 +0000 (UTC) (envelope-from aimass@yabarana.com) Received: from mail-io0-x233.google.com (mail-io0-x233.google.com [IPv6:2607:f8b0:4001:c06::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9F6CC104C for ; Mon, 22 Feb 2016 12:47:11 +0000 (UTC) (envelope-from aimass@yabarana.com) Received: by mail-io0-x233.google.com with SMTP id l127so175006419iof.3 for ; Mon, 22 Feb 2016 04:47:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yabarana-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=sRPypmtr1sAob4+AeKMGKiINP9FX6crYD1umoQyp8Fo=; b=SHbzwQ+rAv02KbbRcldAS4xFLF6ojtl9kM8SBR+VmUqpAzxtpGa4V003j3BtQKwyK/ LdkRm0TrMRumS7wbK+dkG1bYQ05RhiEe82/5wu1nwY8teKIkSr+X6aqbU50IMozeRh5q NZy9k9RjzCF1QIAw+Tb7Ua3reedKaKCAeJgAbp4kgQGSvY0e5Rzqx9eXg97REsUvB5FK r6UCQIPmJd7bGEGVqdRNDaa24PEU6Jjiu9DMS/FvO6qE7MVL9ejutiQVwUiTvVW7HAx7 MyFYJZKkbhdcrO6vUkmoV/F8yZg5+AmPBp4Dpj7DmX+5+bBfBlFbmy46zpuR0H+Izm+0 GY9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=sRPypmtr1sAob4+AeKMGKiINP9FX6crYD1umoQyp8Fo=; b=JlLe1z8pfPISNVewE2JO49Sn9FoilEnWkdlfKRCxl1rMrZlEkq4q2eyRsScZDxvreV yf/8DUWvIX5p2bc6EhnFl4Ste/GDmCUEaAO18ArpUVXrJT14Op5O4GasXZ9L3fjQZZEZ BKxCAyoII2W1wElwu89Mah1dRqpa62FtEFZVYV8Vg0P+vlsrg1Q84Z0GKfFr4dZPsaIU M2tMpIigGEoqOcugO1h23K9pB12tn7gM2ScMv2Wy/LscdilhTrhLYfyaUyCuKkDm8m6b OKE+5SdonXGJvvuH3t6cedPw7v3uv0uZufVFPA062xXweU45CxFawRJDTVXGzcWW83oc TIAw== X-Gm-Message-State: AG10YOQVxRIbEPC0nrJm08d+qCVKG9gNs+zc1fk6RIfGfKhoHFewAFw8py115x2HXfCv4Y8kL2xLn3uGDAax0g== MIME-Version: 1.0 X-Received: by 10.107.5.149 with SMTP id 143mr33602041iof.129.1456145231053; Mon, 22 Feb 2016 04:47:11 -0800 (PST) Received: by 10.107.152.142 with HTTP; Mon, 22 Feb 2016 04:47:10 -0800 (PST) In-Reply-To: References: Date: Mon, 22 Feb 2016 07:47:10 -0500 Message-ID: Subject: Re: Jail management From: Alejandro Imass To: Aristedes Maniatis Cc: freebsd-jail Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 12:47:11 -0000 On Sun, Feb 21, 2016 at 8:13 PM, Aristedes Maniatis wrote: > I've been using FreeBSD jails (with ezjail) for many years and they work > very well. However I'm now reaching a critical mass (30+ jails) where I > want to be able to manage them in bulk more easily. > > > [...] > * FreeBSD's packaging system doesn't understand the concept of installing > a particular package version, so all my scripts will by default upgrade the > application to the current version even if I don't want to. I can't easily > install a new jail at an old version. > > * It is hard to reproduce the environment exactly, matching the > application to the same version of Java that was available at the time of > deployment. Again I'm fighting against the pkg system which always wants > the latest version. > > * For failover I want each jail reproduced exactly on another host, or at > least a snapshot which could be sent to another host within a few seconds. > The jails are quite small (< 500Mb). Most of that is just the openjdk pkg. > > Hi Aristides, I read most of this thread and since you are already using EzJail why not just create a "base" jail with and then use EzJail's archive feature and then create/replace your existing jails with the archive as base? We did something similar for Perl Catalyst applications, precisely to support specific versions of Catalyst, albeit I did never automated alot, the create from archive even worked on multiple servers so long the base system and EzJail set-up was the same on all servers. Best, Alejandro Imass From owner-freebsd-jail@freebsd.org Mon Feb 22 13:37:58 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6C22AB0405; Mon, 22 Feb 2016 13:37:58 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from mx1.sbone.de (bird.sbone.de [46.4.1.90]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 980411CB1; Mon, 22 Feb 2016 13:37:58 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 22FAB25D3891; Mon, 22 Feb 2016 13:37:49 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 6E146C76FCD; Mon, 22 Feb 2016 13:37:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id kmD75uP3DbNN; Mon, 22 Feb 2016 13:37:46 +0000 (UTC) Received: from [IPv6:fde9:577b:c1a9:4420:cabc:c8ff:fe8b:4fe6] (orange-tun0-ula.sbone.de [IPv6:fde9:577b:c1a9:4420:cabc:c8ff:fe8b:4fe6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 68E83C76FE8; Mon, 22 Feb 2016 13:37:46 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Subject: VNET teardown changes (part I) Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) From: "Bjoern A. Zeeb" Date: Mon, 22 Feb 2016 13:37:44 +0000 Cc: FreeBSD Net , freebsd-jail@freebsd.org Content-Transfer-Encoding: 7bit Reply-To: bz@FreeBSD.org Message-Id: To: freebsd-virtualization@freebsd.org X-Mailer: Apple Mail (2.2104) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 13:37:58 -0000 Hi, sorry for the cross-post; Reply-To set. I extracted a patch from projects VNET which tries to get the VNET teardown more robust (and in a next step plug the remaining [TCP] memory leaks). If anyone has an interest in testing some parts on a non-production setup (you have been warned) please do so and report back to me (privately) in case of success or panics. There is more to come. https://people.freebsd.org/~bz/20160222-01-projects-vnets.diff Thanks, Bjoern From owner-freebsd-jail@freebsd.org Mon Feb 22 13:41:17 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72D93AB0587; Mon, 22 Feb 2016 13:41:17 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (bird.sbone.de [46.4.1.90]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 313781F4A; Mon, 22 Feb 2016 13:41:16 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id B62CA25D3891; Mon, 22 Feb 2016 13:41:14 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 108C6C76FE8; Mon, 22 Feb 2016 13:41:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id 64qPPgAHf4da; Mon, 22 Feb 2016 13:41:12 +0000 (UTC) Received: from [IPv6:fde9:577b:c1a9:4420:cabc:c8ff:fe8b:4fe6] (orange-tun0-ula.sbone.de [IPv6:fde9:577b:c1a9:4420:cabc:c8ff:fe8b:4fe6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 66E2CC76FCD; Mon, 22 Feb 2016 13:41:12 +0000 (UTC) From: "Bjoern A. Zeeb" Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: VNET jails not going away Date: Mon, 22 Feb 2016 13:41:10 +0000 Message-Id: Cc: freebsd-jail@freebsd.org To: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) X-Mailer: Apple Mail (2.2104) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 13:41:17 -0000 Hi, has anyone else experienced VNET jails to not fully go away anymore on a = recent HEAD kernel (or possibly an older kernel)? I have test cases with which I can have them in DYING state (see jls = -av) for ever or at least more than half a day. I am in the process of = trying to find the cause but would be good to know if anyone else is = experiencing this? Thanks, Bjoern Example (after more than 12 hours of jail -r ..): # jls -av JID Hostname Path Name State CPUSetID IP Address(es) 1 left.example.net / lef827 DYING 18 2 center.example.net / mid827 DYING 19 3 right.example.net / right827 DYING 20 6 right.example.net / right923 DYING 23= From owner-freebsd-jail@freebsd.org Tue Feb 23 11:42:41 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2CA3CAB1439; Tue, 23 Feb 2016 11:42:41 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:130:3ffc::401:25]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E6B4DF03; Tue, 23 Feb 2016 11:42:40 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id C499125D37C2; Tue, 23 Feb 2016 11:42:38 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 04B89C77037; Tue, 23 Feb 2016 11:42:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id yoPKKbXOYK9q; Tue, 23 Feb 2016 11:42:36 +0000 (UTC) Received: from [IPv6:fde9:577b:c1a9:4420:cabc:c8ff:fe8b:4fe6] (orange-tun0-ula.sbone.de [IPv6:fde9:577b:c1a9:4420:cabc:c8ff:fe8b:4fe6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 36BF3C77007; Tue, 23 Feb 2016 11:42:36 +0000 (UTC) Subject: Re: (VNET) jails not going away Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Content-Type: text/plain; charset=utf-8 From: "Bjoern A. Zeeb" In-Reply-To: Date: Tue, 23 Feb 2016 11:42:34 +0000 Cc: freebsd-jail@freebsd.org Reply-To: bz@FreeBSD.org Content-Transfer-Encoding: quoted-printable Message-Id: References: To: freebsd-virtualization@freebsd.org X-Mailer: Apple Mail (2.2104) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2016 11:42:41 -0000 Hi, sorry for the cross-post, Reply-To: set. > On 22 Feb 2016, at 13:41 , Bjoern A. Zeeb = wrote: >=20 > Hi, >=20 > has anyone else experienced VNET jails to not fully go away anymore on = a recent HEAD kernel (or possibly an older kernel)? >=20 > I have test cases with which I can have them in DYING state (see jls = -av) for ever or at least more than half a day. I am in the process of = trying to find the cause but would be good to know if anyone else is = experiencing this? Ok, I found more funny behaviour that I can get rid of the previous jail = by cleaning up the next one. root@rabbit4:/home/test # jail -i -c -n test19 host.hostname=3Dfoo vnet = persist 19 root@rabbit4:/home/test # jexec 19 /bin/csh root@foo:/ # ifconfig lo0 inet 127.19/8 root@foo:/ # exit root@rabbit4:/home/test # jail -r 19 Jail 19 is in DYING and hangs there forever; If I repeat this upon exit = from jail 20, jail 19 will go away. If I=E2=80=99ll just do this root@rabbit4:/home/test # jail -i -c -n test20 host.hostname=3Dfoo vnet = persist 21 root@rabbit4:/home/test # jail -r 21 20 and 21 are going. I=E2=80=99ll keep tracing this but if it ring a bell for anyone please = let me know ;-) > Thanks, > Bjoern >=20 > Example (after more than 12 hours of jail -r ..): >=20 > # jls -av > JID Hostname Path > Name State > CPUSetID > IP Address(es) > 1 left.example.net / > lef827 DYING > 18 > 2 center.example.net / > mid827 DYING > 19 > 3 right.example.net / > right827 DYING > 20 > 6 right.example.net / > right923 DYING > 23 =E2=80=94=20 Bjoern A. Zeeb Charles Haddon Spurgeon: "Friendship is one of the sweetest joys of life. Many might have failed beneath the bitterness of their trial had they not found a friend." From owner-freebsd-jail@freebsd.org Wed Feb 24 19:59:33 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9A32FAA64F1 for ; Wed, 24 Feb 2016 19:59:33 +0000 (UTC) (envelope-from mlists@mail.ru) Received: from fallback1.mail.ru (fallback1.mail.ru [94.100.181.184]) by mx1.freebsd.org (Postfix) with ESMTP id D5E502F0 for ; Wed, 24 Feb 2016 19:59:32 +0000 (UTC) (envelope-from mlists@mail.ru) Received: from f56.i.mail.ru (f56.i.mail.ru [94.100.185.15]) by fallback1.mail.ru (mPOP.Fallback_MX) with ESMTP id 843146AC46EF; Wed, 24 Feb 2016 22:21:07 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=References:In-Reply-To:Content-Type:Message-ID:Reply-To:Date:MIME-Version:Subject:Cc:To:From; bh=MZk9bkx3TM9e08PtmgIDwtCUQDBMiKp5y0E6pyMHOEo=; b=gySuRN37cnfLDHETRJFdZKx0NYb0hWe+NubWe8HSms21lC7Y3h3ax34pvHlypNxAcnUlZ4W8qXFdYHUDVD7z188lpeKl6gI/0Y7Jf1seVvdkj+/eZPX98eGsyGr2yQZPmOM41JYG4NX+MjcJ697FZBg1tx+e16h3AQUrCMT8Gw0=; Received: from [95.211.187.223] (ident=mail) by f56.i.mail.ru with local (envelope-from ) id 1aYeza-0000qr-Jf; Wed, 24 Feb 2016 22:20:59 +0300 Received: from [95.211.187.223] by e.mail.ru with HTTP; Wed, 24 Feb 2016 22:20:58 +0300 From: =?UTF-8?B?TWFpbCBMaXN0cw==?= To: =?UTF-8?B?QmpvZXJuIEEuIFplZWI=?= Cc: freebsd-jail@freebsd.org Subject: =?UTF-8?B?UmU6IFZORVQgamFpbHMgbm90IGdvaW5nIGF3YXk=?= MIME-Version: 1.0 X-Mailer: Mail.Ru Mailer 1.0 X-Originating-IP: [95.211.187.223] Date: Wed, 24 Feb 2016 22:20:58 +0300 Reply-To: =?UTF-8?B?TWFpbCBMaXN0cw==?= X-Priority: 3 (Normal) Message-ID: <1456341658.824576028@f56.i.mail.ru> X-Mras: Ok X-Spam: undefined In-Reply-To: References: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2016 19:59:33 -0000 CgpIaSwKCkkgaGF2ZSB0aGUgc2FtZS9zaW1pbGFyIHByb2JsZW0sIGl0J3MgcXVpdGUgYW5ub3lp bmcsIHdpdGggUjEwLjEgYW5kIFIxMC4yOgpqYWlsIC1yIHNodXRzIGRvd24gdGhlICh2bmV0LSlq YWlsLCBqbHMgZG9lcyBub3QgbGlzdCB0aGVtIGFueW1vcmUsIGJ1dCB3aXRoIGpscyAtZCwgdGhl eSBhcmUgc3RpbGwgdGhlcmUgLSAKYXBwYXJlbnRseSBpbiBhICdkeWluZyBzdGF0ZScgPwoKSSBj YW5ub3QgcmVzdGFydCB0aGUgamFpbCBhcyBsb25nIGFzIHRoZSBvbGQgamFpbCBzdGlsbCBhcHBl YXJzIGluICdqbHMgLWQnLgpSZWFsbHkgYW5ub3lpbmcuCgpCdXQgdGhleSBnbyBhd2F5LCBldmVu dHVhbGx5LCBzb21ldGltZXMgYWZ0ZXIgNSBtaW51dGVzLCBzb21ldGltZXMgYWZ0ZXIgaGFsZiBh biBob3VyIG9yIHNvLgoKS2FqLgoKPk1vbmRheSwgRmVicnVhcnkgMjIsIDIwMTYgMTo0MSBQTSBV VEMgZnJvbSAiQmpvZXJuIEEuIFplZWIiIDxiemVlYi1saXN0c0BsaXN0cy56YWJiYWRvei5uZXQ+ Ogo+Cj5IaSwKPgo+aGFzIGFueW9uZSBlbHNlIGV4cGVyaWVuY2VkIFZORVQgamFpbHMgdG8gbm90 IGZ1bGx5IGdvIGF3YXkgYW55bW9yZSBvbiBhIHJlY2VudCBIRUFEIGtlcm5lbCAob3IgcG9zc2li bHkgYW4gb2xkZXIga2VybmVsKT8KPgo+SSBoYXZlIHRlc3QgY2FzZXMgd2l0aCB3aGljaCBJIGNh biBoYXZlIHRoZW0gaW4gRFlJTkcgc3RhdGUgKHNlZSBqbHMgLWF2KSBmb3IgZXZlciBvciBhdCBs ZWFzdCBtb3JlIHRoYW4gaGFsZiBhIGRheS4gICBJIGFtIGluIHRoZSBwcm9jZXNzIG9mIHRyeWlu ZyB0byBmaW5kIHRoZSBjYXVzZSBidXQgd291bGQgYmUgZ29vZCB0byBrbm93IGlmIGFueW9uZSBl bHNlIGlzIGV4cGVyaWVuY2luZyB0aGlzPwo+Cj4KPlRoYW5rcywKPkJqb2Vybgo+Cj5FeGFtcGxl IChhZnRlciBtb3JlIHRoYW4gMTIgaG91cnMgb2YgamFpbCAtciAuLik6Cj4KPiMgamxzIC1hdgo+ wqDCoMKgSklEICBIb3N0bmFtZSAgICAgICAgICAgICAgICAgICAgICBQYXRoCj7CoMKgwqDCoMKg wqDCoMKgTmFtZSAgICAgICAgICAgICAgICAgICAgICAgICAgU3RhdGUKPsKgwqDCoMKgwqDCoMKg wqBDUFVTZXRJRAo+wqDCoMKgwqDCoMKgwqDCoElQIEFkZHJlc3MoZXMpCj7CoMKgwqDCoMKgMSAg bGVmdC5leGFtcGxlLm5ldCAgICAgICAgICAgICAgLwo+wqDCoMKgwqDCoMKgwqDCoGxlZjgyNyAg ICAgICAgICAgICAgICAgICAgICAgIERZSU5HCj7CoMKgwqDCoMKgwqDCoMKgMTgKPsKgwqDCoMKg wqAyICBjZW50ZXIuZXhhbXBsZS5uZXQgICAgICAgICAgICAvCj7CoMKgwqDCoMKgwqDCoMKgbWlk ODI3ICAgICAgICAgICAgICAgICAgICAgICAgRFlJTkcKPsKgwqDCoMKgwqDCoMKgwqAxOQo+wqDC oMKgwqDCoDMgIHJpZ2h0LmV4YW1wbGUubmV0ICAgICAgICAgICAgIC8KPsKgwqDCoMKgwqDCoMKg wqByaWdodDgyNyAgICAgICAgICAgICAgICAgICAgICBEWUlORwo+wqDCoMKgwqDCoMKgwqDCoDIw Cj7CoMKgwqDCoMKgNiAgcmlnaHQuZXhhbXBsZS5uZXQgICAgICAgICAgICAgLwo+wqDCoMKgwqDC oMKgwqDCoHJpZ2h0OTIzICAgICAgICAgICAgICAgICAgICAgIERZSU5HCj7CoMKgwqDCoMKgwqDC oMKgMjMKPl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCj5m cmVlYnNkLWphaWxAZnJlZWJzZC5vcmcgbWFpbGluZyBsaXN0Cj5odHRwczovL2xpc3RzLmZyZWVi c2Qub3JnL21haWxtYW4vbGlzdGluZm8vZnJlZWJzZC1qYWlsCj5UbyB1bnN1YnNjcmliZSwgc2Vu ZCBhbnkgbWFpbCB0byAiIGZyZWVic2QtamFpbC11bnN1YnNjcmliZUBmcmVlYnNkLm9yZyAiCgoK QmVzdCByZWdhcmRzLApNYWlsIExpc3RzCm1saXN0c0BtYWlsLnJ1Cg== From owner-freebsd-jail@freebsd.org Wed Feb 24 20:02:10 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2520CAA66B9 for ; Wed, 24 Feb 2016 20:02:10 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:130:3ffc::401:25]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D437C665 for ; Wed, 24 Feb 2016 20:02:09 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 447E825D37C2; Wed, 24 Feb 2016 20:02:06 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 681A6C76FDA; Wed, 24 Feb 2016 20:02:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id jUGBtVYXymrv; Wed, 24 Feb 2016 20:02:03 +0000 (UTC) Received: from [IPv6:fde9:577b:c1a9:4420:cabc:c8ff:fe8b:4fe6] (orange-tun0-ula.sbone.de [IPv6:fde9:577b:c1a9:4420:cabc:c8ff:fe8b:4fe6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id A21D5C76FCE; Wed, 24 Feb 2016 20:02:03 +0000 (UTC) Subject: Re: VNET jails not going away Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Content-Type: text/plain; charset=utf-8 From: "Bjoern A. Zeeb" X-Priority: 3 (Normal) In-Reply-To: <1456341658.824576028@f56.i.mail.ru> Date: Wed, 24 Feb 2016 20:02:02 +0000 Cc: freebsd-jail@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <1456341658.824576028@f56.i.mail.ru> To: Mail Lists X-Mailer: Apple Mail (2.2104) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2016 20:02:10 -0000 > On 24 Feb 2016, at 19:20 , Mail Lists wrote: >=20 >=20 >=20 > Hi, >=20 > I have the same/similar problem, it's quite annoying, with R10.1 and = R10.2: > jail -r shuts down the (vnet-)jail, jls does not list them anymore, = but with jls -d, they are still there -=20 > apparently in a 'dying state' ? >=20 > I cannot restart the jail as long as the old jail still appears in = 'jls -d'. > Really annoying. >=20 > But they go away, eventually, sometimes after 5 minutes, sometimes = after half an hour or so. Yeah, if they eventually go away that=E2=80=99s fine. Hanging around = for a bit can be expected. That=E2=80=99s, e.g., TCP timeouts from = sockets, or similar. The problem is if they never go away. Bjoern= From owner-freebsd-jail@freebsd.org Wed Feb 24 20:52:59 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D81FCAB2E65 for ; Wed, 24 Feb 2016 20:52:59 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8FFC8919 for ; Wed, 24 Feb 2016 20:52:59 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 5CD3C27FB2 for ; Wed, 24 Feb 2016 15:52:58 -0500 (EST) Received: from web6 ([10.202.2.216]) by compute2.internal (MEProxy); Wed, 24 Feb 2016 15:52:58 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=ZLhbUQfAgP6ISuC MjlshD4hAW34=; b=SBXWJYeLSkun3CzLftIehtdbFkrJDeabnwGzwk59BLNlw7W 7MaIAOKuK2sQ9E82fyd66ED/Eghz/eIohdAsbz7xkHCJNpRtUgN4yyW3XfbDws+M aEuR3X609hij73E5Ludw3XT3AaoNuriXTYaOU5AK17pdmDDt94izjqArL8Os= Received: by web6.nyi.internal (Postfix, from userid 99) id 305EC4A26C; Wed, 24 Feb 2016 15:52:58 -0500 (EST) Message-Id: <1456347178.2985454.530948890.160132B1@webmail.messagingengine.com> X-Sasl-Enc: yQxds4F4Rqd+za9mb/GNHwC4iAOQj1WkyybdfYIOSX5W 1456347178 From: Mark Felder To: Aristedes Maniatis , "freebsd-jail" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Mailer: MessagingEngine.com Webmail Interface - ajax-aeec9b65 Subject: Re: Jail management Date: Wed, 24 Feb 2016 14:52:58 -0600 In-Reply-To: References: X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2016 20:52:59 -0000 On Sun, Feb 21, 2016, at 19:13, Aristedes Maniatis wrote: > I've been using FreeBSD jails (with ezjail) for many years and they work > very well. However I'm now reaching a critical mass (30+ jails) where I > want to be able to manage them in bulk more easily. > > In this environment, each jail runs just a single application, installed > from a package built using poudriere from a custom port. That package > depends on Java, so lots of other packages also get pulled in. That > application gets new versions roughly once every 4 weeks. The problems I > have right now are: > > * FreeBSD's packaging system doesn't understand the concept of installing > a particular package version, so all my scripts will by default upgrade > the application to the current version even if I don't want to. I can't > easily install a new jail at an old version. > > * It is hard to reproduce the environment exactly, matching the > application to the same version of Java that was available at the time of > deployment. Again I'm fighting against the pkg system which always wants > the latest version. The package system *could* handle this, but it doesn't fit our design. We aren't like RedHat/Debian where we "freeze" packages at a certain version at the OS release and then backport only changes. With that method different versions of packages will just work with everything else in the system. With FreeBSD's ports system it's really a rolling release as the entire ports tree moves together. Mixing packages build from different checkouts of the ports tree is dangerous and not guaranteed to work. You may be better served with the quarterly branch of the ports tree where things are mostly static for 4 months at a time. Only security and major bugfixes trickle in. Software will get upgraded to fix security issues -- the fixes are not "backported" as that overhead is unmaintainable and is even being criticized in the Linux world these days. The entire software ecosystem needs to stay nimble if we are to stay secure. Upstream projects need to either learn to not break functionality or to provide Long Term Support releases that people can rely on. We definitely have growing pains. > > * For failover I want each jail reproduced exactly on another host, or at > least a snapshot which could be sent to another host within a few > seconds. The jails are quite small (< 500Mb). Most of that is just the > openjdk pkg. > > > As I understand, ezjail doesn't support multiple base jails. If it did, > then I could simply install the application (and packages) to the base > jail and have versions of the base. Then by shutting down a jail, > switching the base to the new version and starting up, everything would > upgrade easily. Even better would be some concept of hierarchy with > customer_jail sitting on top of base_version_1.0 which in turn sits on > top of base_jail. > > Would I need to abandon ezjail and be able to build all the above myself > with a combination of nullfs (basejail) and unionfs (intermediate > versioned jail)? Does unionfs now work with ZFS? > > > Alternatively I could simply use zfs clones to deploy a new version of > the application by destroying the whole jail and replacing it with a new > one. I'd need to then script (I use saltstack) deploying the 2-3 config > files which are different in each jail. > > > > Thoughts? What seems like a more robust long term approach to jail > management? > > I don't use ezjail. It doesn't upgrade well, and changes to the base jail require you stop all your jails. FreeBSD fat jails are so small (300MB?) it's not worth it in my opinion. I simply wrote a shell script to create fat jails and another script to handle updating them all. They're all treated like full servers/VMs, and configs/roles are managed with Ansible/Salt/etc. -- Mark Felder ports-secteam member feld@FreeBSD.org From owner-freebsd-jail@freebsd.org Thu Feb 25 01:39:58 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2CC5EAB27AD for ; Thu, 25 Feb 2016 01:39:58 +0000 (UTC) (envelope-from ari@ish.com.au) Received: from mail12.tpgi.com.au (mail12.tpgi.com.au [203.12.160.162]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.tpg.com.au", Issuer "RapidSSL SHA256 CA - G3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BFEF41A75 for ; Thu, 25 Feb 2016 01:39:57 +0000 (UTC) (envelope-from ari@ish.com.au) X-TPG-Junk-Status: Message not scanned X-TPG-Antivirus: Passed X-TPG-Abuse: host=[202.161.115.54]; ip=202.161.115.54; date=Thu, 25 Feb 2016 11:50:43 +1100 Received: from fish.ish.com.au (202-161-115-54.static.tpgi.com.au [202.161.115.54] (may be forged)) by mail12.tpgi.com.au (envelope-from ari@ish.com.au) (8.14.3/8.14.3) with ESMTP id u1P0off7006897 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 25 Feb 2016 11:50:43 +1100 Received: from ip-136.ish.com.au ([203.29.62.136]:58456) by fish.ish.com.au with esmtpsa (TLSv1.2:DHE-RSA-AES128-SHA:128) (Exim 4.82_1-5b7a7c0-XX) (envelope-from ) id 1aYk8c-0005WX-18; Thu, 25 Feb 2016 11:50:38 +1100 X-CTCH-RefID: str=0001.0A150203.56CE4FDE.008D:SCFSTAT29393324, ss=1, re=-4.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 Subject: Re: Jail management To: Mark Felder , freebsd-jail References: <1456347178.2985454.530948890.160132B1@webmail.messagingengine.com> From: Aristedes Maniatis X-Enigmail-Draft-Status: N1110 Message-ID: Date: Thu, 25 Feb 2016 11:50:37 +1100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: <1456347178.2985454.530948890.160132B1@webmail.messagingengine.com> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Vx58mOf2efqg0dl5cDTduxfIfolEQIx5v" X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Feb 2016 01:39:58 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Vx58mOf2efqg0dl5cDTduxfIfolEQIx5v Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 25/02/2016 7:52am, Mark Felder wrote: >=20 >=20 > On Sun, Feb 21, 2016, at 19:13, Aristedes Maniatis wrote: >> * It is hard to reproduce the environment exactly, matching the >> application to the same version of Java that was available at the time= of >> deployment. Again I'm fighting against the pkg system which always wan= ts >> the latest version. >=20 > The package system *could* handle this, but it doesn't fit our design. > We aren't like RedHat/Debian where we "freeze" packages at a certain > version at the OS release and then backport only changes. With that > method different versions of packages will just work with everything > else in the system. With FreeBSD's ports system it's really a rolling > release as the entire ports tree moves together. Mixing packages build > from different checkouts of the ports tree is dangerous and not > guaranteed to work. Hi Mark Yes, that makes sense and I've frequently struggled in Linux systems to g= et all the bits to match each other properly. So the FreeBSD solution her= e works. But for me, where I use poudriere and roll my own custom packages, versio= ning becomes complicated. I'd need to either create a new poudriere jail = for every release of my software, or go down the path of snap-shotting th= e entire jail (I'm choosing the second). > I don't use ezjail. It doesn't upgrade well, and changes to the base > jail require you stop all your jails. FreeBSD fat jails are so small > (300MB?) it's not worth it in my opinion. I simply wrote a shell script= > to create fat jails and another script to handle updating them all. > They're all treated like full servers/VMs, and configs/roles are manage= d > with Ansible/Salt/etc. That's a good point. And after all the excellent advice here, this is pro= bably what I'll do: 1. Discard salt-minions inside every jail. That's become more trouble to = look after as the number of jails grows. That's also really tricky to han= dle in salt when I want to fail over jails to another host. Since then ev= ery jail is running in two places and that confuses salt. 2. Create a single master/template jail. That might include the basejail = or perhaps I'll keep the nullfs thing which works fine. 3. With every release of my software, upgrade that jail using 'pkg upgrad= e', test and 'zfs snapshot pool/template@v8.10'. I'll keep accumulating s= napshots for every release, bundling up the changes to my software plus a= ll the dependency packages and config. 4. When I want to upgrade a customer jail, I stop the jail and: # zfs destroy pool/customerJail # zfs clone pool/template@v8.10 pool/customerJail I'll have salt help with automating this of course. By using the zfs clon= e command, even the 300Mb of FreeBSD userland takes zero bytes to clone. = These commands should really take less than a second to execute, so the u= pgrade speed is great. 5. Then salt will write out the appropriate customer specific configurati= on into the newly cloned jail (for us that's just 2-3 files) 6. Start jail Where having no basejail falls down is when the host system goes from Fre= eBSD 10 to 11, then I'll need to upgrade every jail but I'll have no way = to go backwards to an older version once this is done since the old snaps= hot will include the old userland. That's probably not a problem for some= thing that is rare. Also, all the above works only because we use logstash and therefore don'= t care about losing all the logs inside each jail with every upgrade. I g= uess syslog would do the same thing for other people. I hope this little summary helps other people facing similar challenges. Ari --=20 --------------------------> Aristedes Maniatis CEO, ish https://www.ish.com.au GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A --Vx58mOf2efqg0dl5cDTduxfIfolEQIx5v Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAlbOT94ACgkQ72p9Lj5JECpUfQCfSlzdHFgqdFugV4UsPeaYQjQG 2EMAnjD99kAmItp4EeIh/FOo7MP4/ppA =7ker -----END PGP SIGNATURE----- --Vx58mOf2efqg0dl5cDTduxfIfolEQIx5v-- From owner-freebsd-jail@freebsd.org Thu Feb 25 15:41:57 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 53619AB2A53 for ; Thu, 25 Feb 2016 15:41:57 +0000 (UTC) (envelope-from eto.freebsd@ethome.sk) Received: from smtpout6.dnsserver.eu (smtpout6.dnsserver.eu [92.240.253.144]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 12C2F1008 for ; Thu, 25 Feb 2016 15:41:56 +0000 (UTC) (envelope-from eto.freebsd@ethome.sk) Received: from [92.240.253.67] (helo=smtp3s109.dnsserver.eu) by smtpout6.dnsserver.eu with esmtp (Exim 4.84 (FreeBSD)) (envelope-from ) id 1aYxiQ-000ATr-8m for freebsd-jail@freebsd.org; Thu, 25 Feb 2016 16:20:30 +0100 Received: from [80.242.44.220] (helo=eto-mona.office.smartweb.sk) by smtp3s109.dnsserver.eu with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128) (Exim 4.83 (FreeBSD)) (envelope-from ) id 1aYxiR-000Gd4-QB for freebsd-jail@freebsd.org; Thu, 25 Feb 2016 16:20:31 +0100 Date: Thu, 25 Feb 2016 16:14:13 +0100 From: "Martin \"eto\" Misuth" To: freebsd-jail@freebsd.org Subject: Re: Jail management Message-ID: <20160225161413.25f17811@eto-mona.office.smartweb.sk> In-Reply-To: <0eaf61d4-43e6-265a-f773-820244fc8931@ish.com.au> References: <0f5cae7e-7de3-2617-fcf6-3423d4caf13a@ish.com.au> <56CAE974.4050508@quip.cz> <0eaf61d4-43e6-265a-f773-820244fc8931@ish.com.au> Organization: ethome.sk MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 80.242.44.220 X-SA-Exim-Mail-From: eto.freebsd@ethome.sk X-SA-Exim-Scanned: No (on smtp3s109.dnsserver.eu); SAEximRunCond expanded to false X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Feb 2016 15:41:57 -0000 On Mon, 22 Feb 2016 22:26:02 +1100 Aristedes Maniatis wrote: > > You are right, and perhaps I should just bite the bullet. I am afraid of only > two things. > Hi I am jails newcomer as well, and this is kinda report of how far I got. But depending on how much you want bet on this technology, I strongly suggest killing some serious time with jails. It's really worth it, if you intend to use them more extensively on FreeBSD, eg. besides "standard" add few lines to rc.conf using ezjail or whatever. For all intent and purposes jail is actually just glorified chroot (although VNET ones are running separate TCP/IP stacks and I have not have had time to look at them). And thus in actual usage jails are pretty "simple" as most things on FreeBSD are (which is benefit). I considered them magical too in the past. In the end it seems same base rules as for managing chroots apply. Although chroot is with us for long time, I don't see much easily accessible information on how to efficiently build and manage chroots themselves and large numbers of chroots concurently. As such the rule is, if there are mentions of chroot parameter in manpages of tool it can be used as both jail and chroot (operating directly on jail tree) management tool. Seems like at certain point most admins doing this kind of work absorb enough knowledge to build chroots on their own, either by some custom scripts stack or with help of package manager and other tools. Several observations: - you can assemble your jail tree at any location, although most setups pick certain path and use that as jail "hive" - if you have your given jail tree mounted in place, you can always chroot to it, without spawning it as jail (this might have security implications!) to test programs are dynamically linking together or whatever - if jail is offline, you can pkg "do" (install whatever) into it with chroot parameter (offline here is just to avoid locking and general clashing with "running" jail) - you can pkg "do" (install, whatever) into running jail as well, with jail parameter - you can use literally any mounted fs under jail's root, and unless you allow given jail to do mounts and stat for fs information, it sees it as one single / it cannot modify - not sure about Miroslav's problems with freebsd-update, but it seems to work pretty well with -basedir /jail/tree parameter nowadays (there might be corner cases) - you can have older jail-base run on newest kernel (other way around is not possible) - you can kill many files in given jail to get bare minimal running setup (this seems completely driven by gut, from what I gathered, as some things might have un-obvious dependencies) - you can mount many things into jail read-only (this makes them more rigid and harder to "manage" "live") - jails can have limits on number of procs living in them and can be allowed to be nested(!) (jail-in-jail) - with rctl you can cap resources per jail - you don't need to run all daemons usually run on host jail - depending on your setup you can get rid of native syslog and use something lighter - jails have their own user databases, but if you do this, uids don't map to usernames when viewed from host (ps, htop etc) - imho simplest possible setup with "live" uid database (eg not copying files around) was for me to use nss ldap extension, that way all jails and host see same synced uids (if you store them in single ldap space, this might have security implications as users in other jails can see others in the system) - as ports tree uses same uids on every machine it made sense to me to import those into main ldap setup, then have per jail subtree with jail local users this requires from you to adjust nss ldap search rules As the idea of jail is having chroot like tree with more protections, once you add ZFS into the mix, number possibilities starts exploding : - by various ways of nesting your jailtree ZFS datasets - you can get to instant "full jails snapshots". Once snapshotted, given your connection limitations and zfs skillset, jails can be sent over to other zfs pools (within same machine or between multiple machines) by virtues of 'zfs send' - similarly you can implement automatic incremental snapshots - you can instruct zfs to attach given dataset to jail (from there jail can control it's own snapshots or other zfs operations from inside) - taking advantage of FreeBSD's / and /usr/local split you can reuse (by zfs clone) same "root" dataset in multiple jails and treat is as easily discardable - this allows you to keep /usr/local subtree and swap roots around after freebsd-update-ing template (by cloning) With last most interesting option you can insert nullfs into the mix this allows you to share some part of host tree within multiple jails. Keep in mind nullfs-es don't mount recursively and this becomes managemnt issue with deeper mounts. So now because it's so easy, when you made 60+ jails starts the fun - all these jails need love to keep them up-to date - packages in these jails need to be kept up-to date - fs partitioning or zfs nesting you picked might turn out to not be exactly optimal - if you use fstab option of jail.conf, unmounting and re-mounting might sometimes fail - some daemons (I suspect native syslog being one of them for me) might leave jail in dying state because of what I guess, are TCP/IP and unix socket timeouts, this might, make "remount" with custom fstab impossible in quick succession (after teardown) - as mentioned nullfs mounts don't nest so you might need some kind of script glue if you want to use that So with all tools mentioned in this thread, you get many of those decisions made for you and you can mostly use your hive only in the way tool's author envisioned. The more you use it directly more convoluted and flexible setups are possible provided you can manage them. Generally, the more static (all jail mounts at system boot + persistent jails with fixed jail ids) setup is, more reliably it operates, but more rigid it becomes to change. I personally don't run heavy load hosts. My main use currently is to test various combinations of http servers with different versions of php-s with various versions of our php stuff.