From owner-freebsd-questions@freebsd.org Mon Feb 17 08:47:35 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6C601252007 for ; Mon, 17 Feb 2020 08:47:35 +0000 (UTC) (envelope-from tim@timpreston.net) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48Ld0P6nVPz4Kv1 for ; Mon, 17 Feb 2020 08:47:33 +0000 (UTC) (envelope-from tim@timpreston.net) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 234BC21F41 for ; Mon, 17 Feb 2020 03:47:33 -0500 (EST) Received: from imap35 ([10.202.2.85]) by compute3.internal (MEProxy); Mon, 17 Feb 2020 03:47:33 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=timpreston.net; h=mime-version:message-id:in-reply-to:references:date:from:to :subject:content-type; s=mesmtp; bh=Q88It4Aiw2fwSO6M6CXUR8IuNVol s2EzKlnuFVY56Ac=; b=H2Tl4S2+MkuR9TpLdfnbmwm0ePJqr5iwPPRDCTXnLFnv fdoJPISIvfDXmEWqPK8ctJC6ROMCNCSdKKZfWwWnHY4E1VFQ/fHlrdXGpEXtCzvA 32jY0PHksVT6P+zCgFyDZKUn+V+fVrH01XSzAETOvynQZLpTrZ7vfv7U+Pnd0R0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=Q88It4 Aiw2fwSO6M6CXUR8IuNVols2EzKlnuFVY56Ac=; b=TMAcCgtqfuMYtZsQvw1Gmx 6eb6R86GmtOmKZSV1L7Y6EAsgdunh7ZsQCA9HTKfLB4BzIVt3sCR8pUxkxzFS3yA Gxvo/NSXC0PvjXcgf+hdFcfn2rh1hQ1nmDbdYKhULLtJ85wFaHGYyd001djuyKgg MenqZ7sdawYwa0CkPqNNR0WkwPUNE0bn7lRXhhpWxNxDUOk8nBBt1tiBy5Pv7g5p XQwtjlHn9Rk13OEmeQOpEjfSRbMmlKaW9+X/lwTAi30uF1ScUcibgl7Qlekiwuof brIzeIzg49ER60p3KBlNYwDH09X5TqwrN+Tt5xs5a9z02UfERxw2T0hDsySTdJhA == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrjeehgdduvdegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtsehttd ertderredtnecuhfhrohhmpedfvfhimhcurfhrvghsthhonhdfuceothhimhesthhimhhp rhgvshhtohhnrdhnvghtqeenucffohhmrghinhepghhithhhuhgsrdgtohhmnecuvehluh hsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepthhimhesthhimhhp rhgvshhtohhnrdhnvght X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id CA59D14C0136; Mon, 17 Feb 2020 03:47:32 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.7-802-g7a41c81-fmstable-20200203v1 Mime-Version: 1.0 Message-Id: <8a9a33b3-4eb1-419c-a9e3-fca4db430619@www.fastmail.com> In-Reply-To: References: Date: Mon, 17 Feb 2020 09:47:11 +0100 From: "Tim Preston" To: freebsd-questions@freebsd.org Subject: Re: Technological advantages over Linux Content-Type: text/plain X-Rspamd-Queue-Id: 48Ld0P6nVPz4Kv1 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=timpreston.net header.s=mesmtp header.b=H2Tl4S2+; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=TMAcCgtq; dmarc=none; spf=pass (mx1.freebsd.org: domain of tim@timpreston.net designates 66.111.4.28 as permitted sender) smtp.mailfrom=tim@timpreston.net X-Spamd-Result: default: False [-5.08 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[timpreston.net:s=mesmtp,messagingengine.com:s=fm2]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[28.4.111.66.rep.mailspike.net : 127.0.0.18]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.28]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[timpreston.net]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[timpreston.net:+,messagingengine.com:+]; IP_SCORE(-3.49)[ip: (-9.85), ipnet: 66.111.4.0/24(-4.89), asn: 11403(-2.68), country: US(-0.05)]; RCVD_IN_DNSWL_LOW(-0.10)[28.4.111.66.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MID_RHS_WWW(0.50)[] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Feb 2020 08:47:35 -0000 Thank you Ihor, this is a great summary. On thing I'd like to mention regarding Docker is the inherent abstraction leakage around pre-built images as they tend to be tied to the host they were built on. For example, I've seen quite a few images in Docker Hub with a hardcoded UID which causes file permissions issue when mounting a volume from the host. Or often authors just assume you'll run the container as root. Another is a mismatch of kernel versions or capabilities between image build host and your host, for example, Redis usually needs Transparent Huge Pages to be turned off in the kernel. It's for these reasons (and the previously mentioned security risks) I'd hope that an 'image' model isn't implemented for FreeBSD jails. Recipes to build jails are a much better idea, as per iocage and Bastille. On Sun, 16 Feb 2020 13:32:29 -0800, Ihor Antonov wrote: > > > I've been reading this tread for a while, and now I can't help but to > add my 2 cents: > > I am long-time Linux sysadmin/devops and I work with "docker" on a daily > basis. Reading this thread I got an impression that a lot of folks on > BSD side have vague/wrong/incomplete understanding of Linux containers > so I want to introduce more structure into this topic. > > First off, "docker" is really a misnomer. Nowadays linux world has a > whole bunch of container tools: moby (former docker), podman, kata > containers, cri-o etc. Not all of them are equal, some of them are complete > user ecosystems, and some are just "bare" runtimes. There was a tool > named "docker" once with that name and the name really stuck, so people > call things "docker" left and right. > > Second, there is no such thing as "linux containers" per se. There are 2 > kernel mechanisms: namespaces(allow isolating a process from a the rest > of the system, like network namespace, user namespace, pid namespace > etc) and cgroups(allow limit resource usage, like cpu, ram, bandwitdh). > Combing various combinations of namespaces and cgroups you get > "containers". On a low level tools like docker et al do is manipulate > namespaces and cgroups. > > The design of namespaces is really the opposite to jails. With > jails you start with a completely isolated environment and then you can > add different capabilites if necessary. With namespaces you start with > non-isolated process (process that shares namespaces with rest of the > system) and you unshare namespaces one by one. (I can't compare resource > limiiting part as I am not familiar with how it is done on FreeBSD) > > It does not mean that namespaces are less secure than jails, it is a different > design, more involved, probably harder to get righ, but also more > flexible. > > Before docker it was very hard to use namespaces and cgroups for a > regular linux user. There was no one "jail" command. There were only > some system calls and scattered docs.(Well there was LXC, but not the > point) > What docker did(and was first to do it) is > provided a very convenient and pretty complete ecosystem to manage > namespaces and cgroups, including features like: > - scripting container creation (aka Dockerfile) and sharing it as code > - sharing compiled images > - Dockerhub is a centralized location for sharing images( it is just > glorified fileserver that hosts a lot of tar.gz + some indexing ) > - sharing/re-using iamges ( FROM clasue in Dockerfile ) > - nice CLI tool to manage containers and images > > And it hid deeply notion of namespaces and cgroups, so regular joes were > able to use it without learning what kernel mechanisms make it possible. > Writing a dockerfile is not very different from writing a shell script > really. It helped widespread adoption of the tool, but with this also > created a lot of misconceptions too. > > One can argue that "docker" is too bloated and is not really secure. > > Yes, it is partially true: > - it makes some choices about how namespaces and cgroups are used, maybe > not the way YOU want. > - It is also a pretty big codebase in golang, that YOU did not audit and > which is not really necessary if you want to manage things manually > and customize to you needs. > - Yes, re-using images from the internet also introduces lots of risks. > - And yes, big army of regular joes who don't know how the tool works > allows misuse, miscofiguration etc. > > But if you understand how it kerlnel works and when you understand your > requirements it is becomes pretty easy to find a proper solutoin. > > > Now coming to jails. jail is pretty low level tool. It should not be > compared to "docker". It can be compared to namespaces though. > > I think it would be more productive to compare capabilities of ecosystems. > - Can you securely sandbox the process with jails or namespaces? > - Can you easily script sanbox creation? > - Can you share/re-use recepies or built images? > - What tools provides more control and what provides more productivity > insread? > - etc... > > Where FreeBSD can improve IMHO is building ecosystem tools around > jails. IOCage and > Bastile are good projects, doing the right thing. But there are still > little to none ways to re-use/share images and build recepies > (AFAIK BasitleBSD is working in that direction). Some might argue that > BSD community does not need those - could be. > > > I use docker for things that are not very important on machines that > > are (relatively) unimportant. I would never use it on something like a > > mail server or web server that has other people?s data on it. > > Yes, use bubblewrap instead - really inspired by jails, minimal, > oriented for maximum security. https://github.com/containers/bubblewrap > > > ------------ > Ihor Antonov