From owner-freebsd-questions@freebsd.org Wed Feb 19 16:02:09 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5EB5E23E73C for ; Wed, 19 Feb 2020 16:02:09 +0000 (UTC) (envelope-from tim@timpreston.net) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48N2Xw4DgMz4bMs for ; Wed, 19 Feb 2020 16:02:08 +0000 (UTC) (envelope-from tim@timpreston.net) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 7DF592223B; Wed, 19 Feb 2020 11:02:07 -0500 (EST) Received: from imap35 ([10.202.2.85]) by compute3.internal (MEProxy); Wed, 19 Feb 2020 11:02:07 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=timpreston.net; h=mime-version:message-id:in-reply-to:references:date:from:to :cc:subject:content-type:content-transfer-encoding; s=mesmtp; bh=rVIAss2V5/aMDFKUXXsu3mvYiJm9S5+cdPcWd56ReYg=; b=kOjHIJSSRVXi HBv2TuRXpOfDs2oQHx2nCAcWIFtL9pGujus0ETzSidCWZs0tfr0QhhlEObBHfPt2 OdF3+xVfqXRoi1aWtT/hVEuCZmfVfldDOXEJT5gMF1Il4VkSQPKx4Vu/cdEQA9S4 qQc7eUdvG3azojxMOUp13GtQCk33T5c= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=rVIAss2V5/aMDFKUXXsu3mvYiJm9S5+cdPcWd56Re Yg=; b=orj4Upi5zeZl1VuBnISgexa/jeznn+ibHutL/Aoa8+72ltwsot0TdiHUB osN+O+x74mUFtC+FGbrdpJRHlr6A9yRm5jFGbcyXlstAqyBUvrJ0o+CYxJcXKUM4 Z8kGLEFvX+IJXFddARGaeU9GjS+i7YchOSG+SvY33S+m2CgLGBmIihiVeAEw0fCN UfA8Iy8zCgVEAfvkyBOUtHM7FkCVQjxryBMqeJzmjvEQHNd/eB50U1cJ4Ron6BdC +STEEpiC7FrJtAAwDrpsAvt6zEPqGgkRhDnCa/X7jSg+g9jRqoI/f7hCA6RltoHs OpNVe4VeaiwlFFboU8ZJzBph4kAtQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrkedtgdekgecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefofgggkfgjfhffhffvufgtgfesthhqredtreerjeenucfhrhhomhepfdfvihhm ucfrrhgvshhtohhnfdcuoehtihhmsehtihhmphhrvghsthhonhdrnhgvtheqnecuffhomh grihhnpeguohgtkhgvrhdrtghomhenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehtihhmsehtihhmphhrvghsthhonhdrnhgvth X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 186DA14C013D; Wed, 19 Feb 2020 11:02:07 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.7-802-g7a41c81-fmstable-20200203v1 Mime-Version: 1.0 Message-Id: <20aa26cd-4b39-4724-a444-30b902dfadd4@www.fastmail.com> In-Reply-To: <20200217194207.rxmcomsn4jvmoc7c@sea-ll-10936> References: <8a9a33b3-4eb1-419c-a9e3-fca4db430619@www.fastmail.com> <20200217194207.rxmcomsn4jvmoc7c@sea-ll-10936> Date: Wed, 19 Feb 2020 17:00:58 +0100 From: "Tim Preston" To: "Ihor Antonov" Cc: freebsd-questions@freebsd.org Subject: Re: Technological advantages over Linux Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 48N2Xw4DgMz4bMs X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=timpreston.net header.s=mesmtp header.b=kOjHIJSS; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=orj4Upi5; dmarc=none; spf=pass (mx1.freebsd.org: domain of tim@timpreston.net designates 66.111.4.29 as permitted sender) smtp.mailfrom=tim@timpreston.net X-Spamd-Result: default: False [-5.08 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[timpreston.net:s=mesmtp,messagingengine.com:s=fm2]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.29]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[timpreston.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[timpreston.net:+,messagingengine.com:+]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-3.49)[ip: (-9.84), ipnet: 66.111.4.0/24(-4.89), asn: 11403(-2.68), country: US(-0.05)]; RCVD_IN_DNSWL_LOW(-0.10)[29.4.111.66.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MID_RHS_WWW(0.50)[] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Feb 2020 16:02:09 -0000 I apologize for the slow reply. Responses inline: On Mon, 17 Feb 2020, at 8:42 PM, Ihor Antonov wrote: > On 2020-02-17 09:47, Tim Preston wrote: > > Thank you Ihor, this is a great summary. > >=20 > > On thing I'd like to mention regarding Docker is the inherent > > abstraction leakage around pre-built images as they tend to be tied = to > > the host they were built on. >=20 > > For example, I've seen quite a few images in Docker Hub with a > > hardcoded UID which causes file permissions issue when mounting a > > volume from the host. Or often authors just assume you'll run the > > container as root. >=20 > Yes, this is a known problem. For reasons unknow to me (could be > technical limitation back when docker started) docker daemon was not > performing UID mapping, so that root > UID inside container (0) was also same UID outside containers, which > created all sorts of problems. Docker now allow to do that, but the > feature is not default >=20 > https://docs.docker.com/engine/security/userns-remap/ >=20 > Newer tool - podman - offers rootless containers feature and perform t= his > by default. Podman got rit of privileged daemon process, so that now > entire container tooling runs in user namespace. And so it has to do > UID/GID mapping between parent/child namespaces, >=20 > This is a step in right direction since now joes don't need root > privileges to work with container tooling.=20 >=20 > Can you create jails in FreeBSD as non-root user and have root inside > jail? >=20 That=E2=80=99s a good question. I haven=E2=80=99t looked into it, maybe = someone else knows? >=20 > Initial design of docker has a flaw, that > was necessary back then, but not anymore - it has a privileged daemon > running, listening on a socket for commands from CLI tool. >=20 >=20 > > Another is a mismatch of kernel versions or capabilities between ima= ge > > build host and your host, for example, Redis usually needs Transpare= nt > > Huge Pages to be turned off in the kernel. >=20 > While this is probably true, I never encountered this issue myslef. An= d > if software requires specific kernel settings - does jail solve this > problem better? (I don't know if there are per-jail sysctl configs..) >=20 I don=E2=80=99t think this is a container problem, but rather a problem = of running images created on a different host. > > It's for these reasons (and the previously mentioned security risks)= > > I'd hope that an 'image' model isn't implemented for FreeBSD jails. > > Recipes to build jails are a much better idea, as per iocage and > > Bastille. >=20 > Pre-build images can emerge as inevitable need to speed up build > process. If your recepie(dockerfile) relies on another recipie and tha= t > one relies on another - it could take A LOT of time to build all the > layers you rely on.=20 >=20 > Basically docker "image" is just collection of layers.=20 > When you work on the dockerfile and > rebuild it regularly - you don't want to rebuild parts that have not > changed. And so docker came up with the idea of image layers. Each > command in Dockerfile creates a layer. And if you did not touch that > specific line in dockerfile - layer will be re-used > When you are finished - your "image" is just a resulting set of layers= . > (overly simplified, but the gist of it)=20 >=20 > And since linux folks did not have proper COW file system(ZFS) they ha= d=20 > to > invent things like overlayfs to quickly take snapshots of the image -=20= > because simply > gziping the image every time somethig chages there was VERY SLOW. >=20 > Dockerhub also stores all the layers, because it appers to be > storage-efficient, since many images can have shared layers. >=20 > So as much as I am with you on=20 > > hope that an 'image' model isn't implemented=20 > I see it as inevitable result of ecosystem development... unless a > radically different approach is taken towards solving "long build time= s" > problem. >=20 I wonder how much of the long build time problem was created by introduc= ing images and layers? I=E2=80=99ve only just started using jails but I = haven=E2=80=99t seen this problem in FreeBSD. As far as I know the longe= st step is downloading a release to base your containers on.=20 In ezjail, for example, you can update all dependent jails by updating t= he base jail once. In Docker, updating the base image would mean rebuild= ing all dependent images, since a change in one layer means rebuilding a= ll higher layers. Docker was important because it paved the way for wide scale kubernetes = adoption. But to answer the original question, no I don=E2=80=99t think = Linux has any technological advantages over FreeBSD in regard to contain= erization. Outside of the ephemeral, horizontally scalable container use= -case I think that jails are more useful.