From owner-freebsd-jail@freebsd.org Thu Jun 2 18:24:46 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9FD9EB65E2E for ; Thu, 2 Jun 2016 18:24:46 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [IPv6:2001:41d0:1008:bcb:1:1:0:1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3927C1FC2 for ; Thu, 2 Jun 2016 18:24:45 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e03:e201:1cf:2a54:f28f:3bb1] (p2003008C2E03E20101CF2A54F28F3BB1.dip0.t-ipconnect.de [IPv6:2003:8c:2e03:e201:1cf:2a54:f28f:3bb1]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3rLFxb70D7zL6J for ; Thu, 2 Jun 2016 20:24:35 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? From: Michael Grimm In-Reply-To: <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com> Date: Thu, 2 Jun 2016 20:24:34 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org> References: <140851342.3380283.1464808961455.JavaMail.yahoo.ref@mail.yahoo.com> <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com> To: "freebsd-jail@freebsd.org" X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 18:24:46 -0000 Sebasti=C3=A1n Maruca via freebsd-jail wrote: > Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, = has anyone tried it? Roger, it seems you are thumbing up my challenge... > But I guess i'll have to stick with netgraph instead epair/if_bridge = because the later is not so documented as the first one=E2=80=A6 Preamble: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, = now 10.3-STABLE, and haven't seen any issues, sofar. Currently I do have = 10 jails running, firewall is pf at the host, only. My servers are not = big scaled ISP like, more small business-like, though. I am considering = myself a hobby admin.=20 Here's my configuration that may show you one way to get that running, = but I am sure your will have to tweak it to your needs: 1) Jails have been created by ezjail in the past, thus they are still at = ezjail's infrastructure. But I do no longer use ezjail for starting or = stopping my jails due to ezjail's lack of dealing with VNET jails (yet). = So I do still have fstab definitions in /etc for all jails, e.g.: /etc/fstab.www /path-to-your/jails/basejail = /path-to-your/jails/www/basejail nullfs ro 0 0=20 2) All external IPv4 or IPv6 addresses are NAT'ed or NAT66'ed to = 10.1.1.x or fd00:dead:dead:beef::x 3) Networking regarding VNET jails defined in /etc/rc.conf: # set up one bridge interface cloned_interfaces=3D"bridge0" # needed for default routes within jails ifconfig_bridge0=3D"inet 10.1.1.254 netmask 255.255.255.0" ifconfig_bridge0_ipv6=3D"inet6 fd00:dead:dead:beef::254 = prefixlen 64" 4) Thus, jails are controlled by jail(8) (shown for 3 example jails): /etc/rc.conf = =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94BEGIN---------= --------------- jail_enable=3D"YES" jail_reverse_stop=3D"YES" jail_list=3D"dns www mail" = =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94-END=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94 /etc/jail.conf: # # host dependent global settings # $ip6prefixLOCAL =3D "fd00:dead:dead:beef"; =09 # # global jail settings # host.hostname =3D "${name}"; path =3D = "/path-to-your/jails/${name}"; mount.fstab =3D "/etc/fstab.${name}"; exec.consolelog =3D = "/var/log/jail_${name}_console.log"; vnet =3D "new"; vnet.interface =3D "epair${jailID}b"; exec.clean; mount.devfs; persist; =09 # # network settings to apply/destroy during start/stop of = every jail # exec.prestart =3D "sleep 2"; exec.prestart +=3D "ifconfig epair${jailID} = create up"; exec.prestart +=3D "ifconfig bridge0 addm = epair${jailID}a"; exec.start =3D "/sbin/ifconfig lo0 = 127.0.0.1 up"; exec.start +=3D "/sbin/ifconfig = epair${jailID}b inet ${ip4_addr}"; exec.start +=3D "/sbin/ifconfig = epair${jailID}b inet6 ${ip6_addr}"; exec.start +=3D "/sbin/route add default = -gateway 10.1.1.254"; exec.start +=3D "/sbin/route add -inet6 = default -gateway ${ip6prefixLOCAL}::254"; #exec.stop =3D "/sbin/route del default"; #exec.stop +=3D "/sbin/route del -inet6 = default"; exec.stop +=3D "/bin/sh /etc/rc.shutdown"; exec.poststop =3D "ifconfig epair${jailID}a = destroy"; =09 # # individual jail settings # mail { $jailID =3D 1; $ip4_addr =3D 10.1.1.1; $ip6_addr =3D ${ip6prefixLOCAL}::1/64; exec.start +=3D "/bin/sh /etc/rc"; } =09 www { $jailID =3D 2; $ip4_addr =3D 10.1.1.2; $ip6_addr =3D ${ip6prefixLOCAL}::2/64; exec.start +=3D "/bin/sh /etc/rc"; } =09 dns { $jailID =3D 3; $ip4_addr =3D 10.1.1.3; $ip4_addr_2 =3D 10.1.1.4; $ip6_addr =3D ${ip6prefixLOCAL}::3/64; $ip6_addr_2 =3D ${ip6prefixLOCAL}::4/64; exec.start +=3D "/sbin/ifconfig = epair${jailID}b inet ${ip4_addr_2} alias"; exec.start +=3D "/sbin/ifconfig = epair${jailID}b inet6 ${ip6_addr_2} alias"; exec.start +=3D "/bin/sh /etc/rc"; } =09 Now you can use "service jail" to start/stop your jails, e.g.: service jail stop=20 service jail restart dns service jail start dns mail 5) NOTE: I am refraining from restarting VNET jails the hard way as = shown above, and I am using a similar approach as iocage, namely "soft = restarts". As this functionality isn't available in 10.3-STABLE (IIRC) I = am using a homemade shell script instead. This script has to be run = *inside* a jail which can be triggered from the outside (still using = ezjail-admin) by e.g.: "sudo ezjail-admin console -e = '/usr/local/etc/_JAIL_SOFT_RESTART' www" #!/bin/csh =09 # # restart jail services without removing jail and its network # =09 # # global definitions # set LOGGER =3D "/usr/bin/logger -p user.info -t _JAIL_SOFT_RC" set RCDIR =3D "/usr/local/etc/rc.d" set TAB =3D " " =09 # # evaluate list of rc files in /usr/local/etc/rc.d # set RCFILES =3D `rcorder ${RCDIR}/* |& grep -v ^rcorder:` =09 # # evaluate reverse order of RCFILES # set RCFILES_REVERSE =3D "" foreach rcname ( ${RCFILES} ) set RCFILES_REVERSE =3D "${rcname} ${RCFILES_REVERSE}" end =09 # # stop rc services # echo "stopping:" foreach rcname ( ${RCFILES_REVERSE} ) ${LOGGER} stopping ${rcname} ${rcname} stop >& /dev/null echo "${TAB}" ${rcname} end =09 # # start rc services # echo "starting:" foreach rcname ( ${RCFILES} ) ${LOGGER} starting ${rcname} ${rcname} start >& /dev/null echo "${TAB}" ${rcname} end =09 exit 0 This script isn't perfect, and if you start or stop a jail you need to = separate the relevant part. This can easily be coded into that script, I = know. But I was lazy ;-) I hope that helps for a start. Again, I am sure you may need some = tweaking at your site. Regards, Michael