From owner-freebsd-jail@freebsd.org Wed Dec 14 04:28:15 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BB850C73C7B for ; Wed, 14 Dec 2016 04:28:15 +0000 (UTC) (envelope-from ike@blackskyresearch.net) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 90C971D3C for ; Wed, 14 Dec 2016 04:28:14 +0000 (UTC) (envelope-from ike@blackskyresearch.net) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id A003C20977; Tue, 13 Dec 2016 23:28:13 -0500 (EST) Received: from frontend1 ([10.202.2.160]) by compute1.internal (MEProxy); Tue, 13 Dec 2016 23:28:13 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= blackskyresearch.net; h=cc:content-transfer-encoding :content-type:date:from:in-reply-to:message-id:mime-version :references:subject:to:x-me-sender:x-me-sender:x-sasl-enc :x-sasl-enc; s=mesmtp; bh=7glIgxIqjVrpSdko05S7ZHxIdwo=; b=K2+Pun 4s0VmxYmhJNfvHsYQVmFgZNofHd9TcpZwr+BteS4/yB6zuW6DSSnijsc5+HEtgHG zee4atYNXPKarqyDL6WLa0toieodvAWtPH8Lp7+8YyhfEv10/s1OsmAHKs8sL4Sd hpcs9ybQS8W9hnPtCidajRr7HHR9+n4+cTJ/o= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc:x-sasl-enc; s= smtpout; bh=7glIgxIqjVrpSdko05S7ZHxIdwo=; b=he868T/Ar4Ygr0afWlgg qlf7m71uTCjAv+TEP5jLgXSMIXarCYftmaruTMxnWMQAFpAuzmob2sX7SprLHnlj PZi54bKCZEnvFc2tKAso9QaNpdHDTPGnFEziq4rbASXrl69LiUIrlhFdUfgULKA6 aGpoaBCXEs6zX8+Nzzxr4Mk= X-ME-Sender: X-Sasl-enc: 6RjKpU4rVAi9rAocq5ZtrlcX3R+AQ3Dn7nXLe+T7J6yc 1481689693 Received: from [10.0.224.105] (cpe-24-90-119-105.nyc.res.rr.com [24.90.119.105]) by mail.messagingengine.com (Postfix) with ESMTPA id 55E497EA6B; Tue, 13 Dec 2016 23:28:13 -0500 (EST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: multiple interfaces for jail.conf(1) and jail_set(2) From: "Isaac (.ike) Levy" In-Reply-To: <5850A9F6.2090501@gmail.com> Date: Tue, 13 Dec 2016 23:28:12 -0500 Cc: freebsd-jail@FreeBSD.org Content-Transfer-Encoding: quoted-printable Message-Id: <8E3BBF75-D2A2-4B42-A693-41D0B3F16D19@blackskyresearch.net> References: <0ED7F403-F14E-4A72-8E54-AF74AAE15061@blackskyresearch.net> <5850A9F6.2090501@gmail.com> To: Ernie Luzar X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2016 04:28:15 -0000 Thanks Ernie, But, that straight out did not work for me, > On Dec 13, 2016, at 9:09 PM, Ernie Luzar wrote: >=20 > Isaac (.ike) Levy wrote: >> Hi All, >> Can I specify multiple IP interfaces and assign IP=E2=80=99s to them = using jail.conf? >> I have jails with IPv4/IPv6 addresses on multiple physical = interfaces, as well as assigning a loopback. >> I have not found answers in the respective man pages or digging = online. >> I=E2=80=99m finally starting to poke around to start using the = impressively simple jail.conf subsystem to manage jails. I have been = managing jails with simple custom start scripts since 99=E2=80=99, and = custom devfs rulesets since ~2006, so jail.conf(1) and jail_set(2) are a = big welcome change for me- really awesome and clean :) >> -- >> Additional detail to clarify my loopback use: >> In general, I always assign each jail it=E2=80=99s own a loopback IP = somewhere in the RFC5735 specified range, 127.0.0.0/8 - (simply saving = 127.0.0.1 for the jailing host), and then I simply set localhost to = point at it=E2=80=99s IP in /etc/hosts for the jail. On the host, I = simply add the IP alias to lo0 like any other interface. >> This is often overlooked in common jailing practice, but often = eliminates complexity and confusion for many userland daemons. For full = Virtual Server applications, loopback is simply dotting the i=E2=80=99s = and crossing the t=E2=80=99s. >> I can see how localhost would be challenging to automate for easy = jail.conf configuration, mostly, in picking a loopback IP for the jail = and not letting that get messy- etc=E2=80=A6 >> Thanks in advance for any info! >> Best, >> .ike >=20 > Using native jail.conf you can assign multiple NICs with both ipv4 & = ipv6 ip address. By native I mean use the jail(8) command to start/stop = your jails IE. not [service jail start] command. Use this format > ip.addr =3D "rlo:10.0.10.02,xl0:10.20.10.07,lo0:127.10.0.02" This will = also automatically create and remove the required aliases. That does not appear to work- which is sad, I was excited by the syntax. I am getting the following error, r# jail -c myjail jail: medial: ip4.addr: not an IPv4 address: em0:10.0.0.22 jail: myjail: ip6.addr: not an IPv6 address: em0:2:2:2:2::22 # uname -r 11.0-RELEASE-p2 My jail.conf contains precisely the following, myjail { path =3D /foo/bar; mount.devfs; host.hostname =3D bar.blackskyresearch.net; ip4.addr =3D "em0:10.0.0.22,lo0:127.0.0.22"; ip6.addr =3D "em0:2:2:2:2::22"; exec.start =3D "/bin/sh /etc/rc"; exec.stop =3D "/bin/sh /etc/rc.shutdown"; } Noteworthy- the error thrown for ip4.addr does not even get to = mentioning the second listed IP on lo0. >=20 > A word about loopback. Just like on the host, most services do not use = the loopback interface, this is also true for jailed services. Only = services that default to using the loopback interface need one defined = in the jail to work correctly. Sure sometimes, but not always. While 127.0.0.1 is hardcoded into many = apps and configs, this is certainly more controllable in my world where = I can physically slap whomever wrote the daemon with hardcoded IP=E2=80=99= s- even for using local inet sockets :) >=20 > Take note, the services that use the loopback interface default to = using 127.0.0.1 ip address. For a service in a jail that uses loopback = MUST have it's configuration changed to use the 127.10.0.02 ip address = assigned on the jails jail.conf ip.addr parameter. No service in a jail = can be assigned the hosts 127.0.0.1 ip address. Certainly. Yet, I=E2=80=99ve found very few headaches after changing a = /etc/hosts to reflect the localhost IP for the jail. =E2=80=9Clocalhost=E2= =80=9D just resolves, as it should. >=20 > I recommend you check out these ports, > jail-primer gives background on jails across Freebsd releases. I believe I gave the author of that document extensive feedback when it = was originally authored- as a submission rewrite for the handbook. While this jail-primer doc was filled with many useful and practical = words of advice, it was also a document which I provided a great deal of = constructive feedback for the author, (pre 9.2 release). I was particularly worried about the way the =E2=80=9Cjail cell=E2=80=9D = vocabulary abstraction was introduced and used. I cited a relentless = =E2=80=9Cuse my port=E2=80=9D approach to jail administration. And = finally, in that doc, there was far too much of an overall fundamental = shift away from base UNIX ways of doing things- and even the FreeBSD way = of doing things. I find documentation like this to be frustrating for = oldschoolers because it is not concise or technically informative, and = detracts for new users- by presenting jail(8) in a manner which is = abstracted into something so from the FreeBSD operating system. On a quick skim, the jail-primer project you posted appears to be = roughly the same document- and it also does not have the information = about IP interfaces jail.conf syntax you mention above. > qjail a utility that simplifies jail admin. Thanks, but I=E2=80=99m not really interested in qjail or else I would = have asked about it wherever they run their list! While I do see tools like qjail, good ol=E2=80=99 ezjail, and iocage as = being very valuable, they have little to do with my question. -- Back to the original post- did I do something wrong or interpret your = instructions incorrectly? Thanks! Best, .ike >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20