From owner-freebsd-jail@freebsd.org Wed Feb 24 20:52:59 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D81FCAB2E65 for ; Wed, 24 Feb 2016 20:52:59 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8FFC8919 for ; Wed, 24 Feb 2016 20:52:59 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 5CD3C27FB2 for ; Wed, 24 Feb 2016 15:52:58 -0500 (EST) Received: from web6 ([10.202.2.216]) by compute2.internal (MEProxy); Wed, 24 Feb 2016 15:52:58 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=ZLhbUQfAgP6ISuC MjlshD4hAW34=; b=SBXWJYeLSkun3CzLftIehtdbFkrJDeabnwGzwk59BLNlw7W 7MaIAOKuK2sQ9E82fyd66ED/Eghz/eIohdAsbz7xkHCJNpRtUgN4yyW3XfbDws+M aEuR3X609hij73E5Ludw3XT3AaoNuriXTYaOU5AK17pdmDDt94izjqArL8Os= Received: by web6.nyi.internal (Postfix, from userid 99) id 305EC4A26C; Wed, 24 Feb 2016 15:52:58 -0500 (EST) Message-Id: <1456347178.2985454.530948890.160132B1@webmail.messagingengine.com> X-Sasl-Enc: yQxds4F4Rqd+za9mb/GNHwC4iAOQj1WkyybdfYIOSX5W 1456347178 From: Mark Felder To: Aristedes Maniatis , "freebsd-jail" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Mailer: MessagingEngine.com Webmail Interface - ajax-aeec9b65 Subject: Re: Jail management Date: Wed, 24 Feb 2016 14:52:58 -0600 In-Reply-To: References: X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2016 20:52:59 -0000 On Sun, Feb 21, 2016, at 19:13, Aristedes Maniatis wrote: > I've been using FreeBSD jails (with ezjail) for many years and they work > very well. However I'm now reaching a critical mass (30+ jails) where I > want to be able to manage them in bulk more easily. > > In this environment, each jail runs just a single application, installed > from a package built using poudriere from a custom port. That package > depends on Java, so lots of other packages also get pulled in. That > application gets new versions roughly once every 4 weeks. The problems I > have right now are: > > * FreeBSD's packaging system doesn't understand the concept of installing > a particular package version, so all my scripts will by default upgrade > the application to the current version even if I don't want to. I can't > easily install a new jail at an old version. > > * It is hard to reproduce the environment exactly, matching the > application to the same version of Java that was available at the time of > deployment. Again I'm fighting against the pkg system which always wants > the latest version. The package system *could* handle this, but it doesn't fit our design. We aren't like RedHat/Debian where we "freeze" packages at a certain version at the OS release and then backport only changes. With that method different versions of packages will just work with everything else in the system. With FreeBSD's ports system it's really a rolling release as the entire ports tree moves together. Mixing packages build from different checkouts of the ports tree is dangerous and not guaranteed to work. You may be better served with the quarterly branch of the ports tree where things are mostly static for 4 months at a time. Only security and major bugfixes trickle in. Software will get upgraded to fix security issues -- the fixes are not "backported" as that overhead is unmaintainable and is even being criticized in the Linux world these days. The entire software ecosystem needs to stay nimble if we are to stay secure. Upstream projects need to either learn to not break functionality or to provide Long Term Support releases that people can rely on. We definitely have growing pains. > > * For failover I want each jail reproduced exactly on another host, or at > least a snapshot which could be sent to another host within a few > seconds. The jails are quite small (< 500Mb). Most of that is just the > openjdk pkg. > > > As I understand, ezjail doesn't support multiple base jails. If it did, > then I could simply install the application (and packages) to the base > jail and have versions of the base. Then by shutting down a jail, > switching the base to the new version and starting up, everything would > upgrade easily. Even better would be some concept of hierarchy with > customer_jail sitting on top of base_version_1.0 which in turn sits on > top of base_jail. > > Would I need to abandon ezjail and be able to build all the above myself > with a combination of nullfs (basejail) and unionfs (intermediate > versioned jail)? Does unionfs now work with ZFS? > > > Alternatively I could simply use zfs clones to deploy a new version of > the application by destroying the whole jail and replacing it with a new > one. I'd need to then script (I use saltstack) deploying the 2-3 config > files which are different in each jail. > > > > Thoughts? What seems like a more robust long term approach to jail > management? > > I don't use ezjail. It doesn't upgrade well, and changes to the base jail require you stop all your jails. FreeBSD fat jails are so small (300MB?) it's not worth it in my opinion. I simply wrote a shell script to create fat jails and another script to handle updating them all. They're all treated like full servers/VMs, and configs/roles are managed with Ansible/Salt/etc. -- Mark Felder ports-secteam member feld@FreeBSD.org