From owner-freebsd-jail@freebsd.org Thu Jun 2 21:29:34 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1DD89B67523 for ; Thu, 2 Jun 2016 21:29:34 +0000 (UTC) (envelope-from juanperiz@yahoo.com.ar) Received: from nm39-vm8.bullet.mail.gq1.yahoo.com (nm39-vm8.bullet.mail.gq1.yahoo.com [98.136.217.111]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E13D81330 for ; Thu, 2 Jun 2016 21:29:33 +0000 (UTC) (envelope-from juanperiz@yahoo.com.ar) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.ar; s=s2048; t=1464902967; bh=KtSsZJcyOA8MSGsJpiTIast853JgFeDurd5sIwI3+So=; h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject; b=Wo4EXu3gA+mAqe6RfRuqAcl/95/Nvi80WvYS/j8huiIknjdrG0eEFRczPCkfEaMhnLEWDk2kVE/7Y4djOz2aUBc1hFbRpALqWxJ/AS1DM+K8kazaQDvo5uaFMuRTH/q/FzgbFcweabCNDfBQURuP96P+MjZrJvbmrIRGG3IYTjaziH4lM5R4N6lq2BXPHHbDNm4uM57XfJ/UKcOcZVBkAQpITQzVF/vwcthCYBglAT7qv0ar652oRMGEtjfiHxpX/1/t7gCq2kdOu0XUApXWCTo5icpHQyiM7c4QySIctUexWXauSSrBlFuTjZmUcZLbOsuy3dR/95pu89V65FxS5g== Received: from [127.0.0.1] by nm39.bullet.mail.gq1.yahoo.com with NNFMP; 02 Jun 2016 21:29:27 -0000 Received: from [98.137.12.190] by nm39.bullet.mail.gq1.yahoo.com with NNFMP; 02 Jun 2016 21:26:42 -0000 Received: from [98.137.12.212] by tm11.bullet.mail.gq1.yahoo.com with NNFMP; 02 Jun 2016 21:26:42 -0000 Received: from [127.0.0.1] by omp1020.mail.gq1.yahoo.com with NNFMP; 02 Jun 2016 21:26:42 -0000 X-Yahoo-Newman-Property: ymail-4 X-Yahoo-Newman-Id: 64175.6436.bm@omp1020.mail.gq1.yahoo.com X-YMail-OSG: EYqV43MVM1kzETPCbJQlvk8bVUbD32nJ1ecVnjdpn53GwJMEIRCZxaEO97bs.Ng cnzkcbhi_HKhpBLQMJlEJpS_3wWFLOBunXsDs9_eZO24pHdxMm9ECRVWP44e5U8Va75aAFH9hcJp HsGAntnxZL1zfWK8jhiJOASQ5NZ4ePARcciUhM6Jdsk6ISSSxC4qeu8YCqg42ay_mjiUWzdpSjLA vSvXahE0hS5xrS96odEi2aIffNQSvO7CuRonofv6RutTgi7Xgt7BZOKUKMTQEfmNxAviuJEjbZjx Blr58vAsjuj7hPidVmCrWDJ.zloNxyQUaebReC7HMEx.Lnf2TFpJFhvlyztpucE_5.XrZyBeaIy3 CLlyZPC7M6B3Fjs5WwAWzwqyoBwovvoIvYDinE5pGOZEYnuRojiMi1WJN750RhBj649DOps.r.ub Ftp6ARG2H7QZcPWmcPnIFfX9Z5ZX6Ez.hyw0LudU_33zDoPdEAUZRqGv4k8KgZyTppMtTMGxv4eW 9R41VaWACCki3g7LtsxAwz9h.7HbYZH8lChk- Received: from jws10746.mail.gq1.yahoo.com by sendmailws128.mail.gq1.yahoo.com; Thu, 02 Jun 2016 21:26:41 +0000; 1464902801.582 Date: Thu, 2 Jun 2016 21:26:41 +0000 (UTC) From: =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= Reply-To: =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= To: Michael Grimm , "freebsd-jail@freebsd.org" Message-ID: <377963018.4245125.1464902801251.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org> References: <140851342.3380283.1464808961455.JavaMail.yahoo.ref@mail.yahoo.com> <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com> <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org> Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 21:29:34 -0000 Michael... even though you consider yourself as a admin hobbier, I can tell= you have the "lend hander" top grade you're honored ;) I'll start from this big step you're posting (and all the other which repli= ed too) and carry on dancing 'til I got my jails running DMZ, VLAN and WAN = like a pro... Best Regards,Seba De: Michael Grimm Para: "freebsd-jail@freebsd.org" =20 Enviado: Jueves, 2 de junio, 2016 15:24:34 Asunto: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? =20 Sebasti=C3=A1n Maruca via freebsd-jail wrote: > Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, has = anyone tried it? Roger, it seems you are thumbing up my challenge... > But I guess i'll have to stick with netgraph instead epair/if_bridge beca= use the later is not so documented as the first one=E2=80=A6 Preamble: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, no= w 10.3-STABLE, and haven't seen any issues, sofar. Currently I do have 10 j= ails running, firewall is pf at the host, only. My servers are not big scal= ed ISP like, more small business-like, though. I am considering myself a ho= bby admin.=20 Here's my configuration that may show you one way to get that running, but = I am sure your will have to tweak it to your needs: 1) Jails have been created by ezjail in the past, thus they are still at ez= jail's infrastructure. But I do no longer use ezjail for starting or stoppi= ng my jails due to ezjail's lack of dealing with VNET jails (yet). So I do = still have fstab definitions in /etc for all jails, e.g.: =C2=A0=C2=A0=C2=A0 /etc/fstab.www =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 /path-to-your/jails/basejail /path-to= -your/jails/www/basejail nullfs ro 0 0=20 2) All external IPv4 or IPv6 addresses are NAT'ed or NAT66'ed to 10.1.1.x o= r fd00:dead:dead:beef::x 3) Networking regarding VNET jails defined in /etc/rc.conf: =C2=A0=C2=A0=C2=A0 # set up one bridge interface =C2=A0=C2=A0=C2=A0 cloned_interfaces=3D"bridge0" =C2=A0=C2=A0=C2=A0 # needed for default routes within jails =C2=A0=C2=A0=C2=A0 ifconfig_bridge0=3D"inet 10.1.1.254 netmask 255.255.255.= 0" =C2=A0=C2=A0=C2=A0 ifconfig_bridge0_ipv6=3D"inet6 fd00:dead:dead:beef::254 = prefixlen 64" 4) Thus, jails are controlled by jail(8) (shown for 3 example jails): =C2=A0=C2=A0=C2=A0 /etc/rc.conf =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94BEGIN------------------------ =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 jail_enable=3D"YES" =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 jail_reverse_stop=3D"YES" =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 jail_list=3D"dns www mail" =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94-END=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 =C2=A0=C2=A0=C2=A0 /etc/jail.conf: =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # host dependent global settings =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6prefixLOCAL=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 =3D "fd00:dead:dead:beef"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # global jail settings =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 host.hostname=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 =3D "${name}"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 path=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0= =C2=A0 =C2=A0=C2=A0=C2=A0 =3D "/path-to-your/jails/${name}"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 mount.fstab=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 =3D "/etc/fstab.${name}"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.consolelog =C2=A0=C2=A0=C2=A0 = =3D "/var/log/jail_${name}_console.log"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 vnet=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0= =C2=A0 =C2=A0=C2=A0=C2=A0 =3D "new"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 vnet.interface=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 =3D "epair${jailID}b"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.clean; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 mount.devfs; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 persist; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # network settings to apply/destroy d= uring start/stop of every jail =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.prestart=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 =3D "sleep 2"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.prestart=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 +=3D "ifconfig epair${jailID} create up"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.prestart=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 +=3D "ifconfig bridge0 addm epair${jailID}a"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 =3D "/sbin/ifconfig lo0 127.0.0.1 up"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 +=3D "/sbin/ifconfig epair${jailID}b inet ${ip4_addr}"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 +=3D "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr}"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 +=3D "/sbin/route add default -gateway 10.1.1.254"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 +=3D "/sbin/route add -inet6 default -gateway ${ip6prefixLOCAL= }::254"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #exec.stop=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 =3D "/sbin/route del default"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #exec.stop=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 +=3D "/sbin/route del -inet6 default"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.stop=C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 +=3D "/bin/sh /etc/rc.shutdown"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.poststop =C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 =3D "ifconfig epair${jailID}a destroy"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # individual jail settings =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 mail { =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $jailID=C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =3D 1; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip4_addr=C2=A0=C2= =A0=C2=A0 =3D 10.1.1.1; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6_addr=C2=A0=C2= =A0=C2=A0 =3D ${ip6prefixLOCAL}::1/64; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0= =C2=A0=C2=A0 +=3D "/bin/sh /etc/rc"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 } =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 www { =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $jailID=C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =3D 2; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip4_addr=C2=A0=C2= =A0=C2=A0 =3D 10.1.1.2; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6_addr=C2=A0=C2= =A0=C2=A0 =3D ${ip6prefixLOCAL}::2/64; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0= =C2=A0=C2=A0 +=3D "/bin/sh /etc/rc"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 } =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 dns { =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $jailID=C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =3D 3; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip4_addr=C2=A0=C2= =A0=C2=A0 =3D 10.1.1.3; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip4_addr_2=C2=A0= =C2=A0=C2=A0 =3D 10.1.1.4; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6_addr=C2=A0=C2= =A0=C2=A0 =3D ${ip6prefixLOCAL}::3/64; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6_addr_2=C2=A0= =C2=A0=C2=A0 =3D ${ip6prefixLOCAL}::4/64; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0= =C2=A0=C2=A0 +=3D "/sbin/ifconfig epair${jailID}b inet=C2=A0 ${ip4_addr_2} = alias"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0= =C2=A0=C2=A0 +=3D "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr_2} alias= "; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0= =C2=A0=C2=A0 +=3D "/bin/sh /etc/rc"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 } =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 Now you can use "service jail" to start/stop your jails,= e.g.: =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 service jail stop=20 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 service jail restart dns =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 service jail start dns mail 5) NOTE: I am refraining from restarting VNET jails the hard way as shown a= bove, and I am using a similar approach as iocage, namely "soft restarts". = As this functionality isn't available in 10.3-STABLE (IIRC) I am using a ho= memade shell script instead. This script has to be run *inside* a jail whic= h can be triggered from the outside (still using ezjail-admin) by e.g.: "su= do ezjail-admin console -e '/usr/local/etc/_JAIL_SOFT_RESTART' www" =C2=A0=C2=A0=C2=A0 #!/bin/csh =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 # restart jail services without removing jail and its ne= twork =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 # global definitions =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 set LOGGER =3D "/usr/bin/logger -p user.info -t _JAIL_SO= FT_RC" =C2=A0=C2=A0=C2=A0 set RCDIR =3D "/usr/local/etc/rc.d" =C2=A0=C2=A0=C2=A0 set TAB =3D "=C2=A0 =C2=A0 =C2=A0 =C2=A0 " =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 # evaluate list of rc files in /usr/local/etc/rc.d =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 set RCFILES =3D `rcorder ${RCDIR}/* |& grep -v ^rcorder:= ` =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 # evaluate reverse order of RCFILES =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 set RCFILES_REVERSE =3D "" =C2=A0=C2=A0=C2=A0 foreach rcname ( ${RCFILES} ) =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 set RCFILES_REVERSE =3D "${rcname} ${= RCFILES_REVERSE}" =C2=A0=C2=A0=C2=A0 end =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 # stop rc services =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 echo "stopping:" =C2=A0=C2=A0=C2=A0 foreach rcname ( ${RCFILES_REVERSE} ) =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 ${LOGGER} stopping ${rcname} =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 ${rcname} stop >& /dev/null =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 echo "${TAB}" ${rcname} =C2=A0=C2=A0=C2=A0 end =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 # start rc services =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 echo "starting:" =C2=A0=C2=A0=C2=A0 foreach rcname ( ${RCFILES} ) =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 ${LOGGER} starting ${rcname} =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 ${rcname} start >& /dev/null =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 echo "${TAB}" ${rcname} =C2=A0=C2=A0=C2=A0 end =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 exit 0 This script isn't perfect, and if you start or stop a jail you need to sepa= rate the relevant part. This can easily be coded into that script, I know. = But I was lazy ;-) I hope that helps for a start. Again, I am sure you may need some tweaking = at your site. Regards, Michael _______________________________________________ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscribe@freebsd.org" From owner-freebsd-jail@freebsd.org Thu Jun 2 22:05:25 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6EB8B67FF8 for ; Thu, 2 Jun 2016 22:05:25 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-it0-x22c.google.com (mail-it0-x22c.google.com [IPv6:2607:f8b0:4001:c0b::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9F8661796 for ; Thu, 2 Jun 2016 22:05:25 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-it0-x22c.google.com with SMTP id i127so63881716ita.1 for ; Thu, 02 Jun 2016 15:05:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=e/rPc/1dbW5I8qmQan0CHYWFwykS7g60OSrNYbmOLuw=; b=ez18eA1R9mIHJy2OcyJ7nnJ1sAMy8GI2FI9ztcN7jQ8Ouo45XzHAyO2pNKgZYIaU4s 5Z0BfVTWsXIhxFQYTYUTV3o9elsBYVWTskoO1iCYp+Lsw3Rgb/AVbIfw+xXmWBPeAfNO +SY/Va+Gq8DUbd9Zm07yGsMPrTjdve/JtT2h6tKkgwNnLoRCq3GFC/JBV9h89EucdX8J hmWh2TsJOGSE06YrubZSH50OG3OwpIq+dTKF5wxLo2xyqa5orivIxXezXXRyKNDFkasl QdBAdUAh4swLyLt24VctsuZzMVH21F1oAORvIS7F58JMoYHJ/5s65kbSMCJWHXm3gTCC I8uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=e/rPc/1dbW5I8qmQan0CHYWFwykS7g60OSrNYbmOLuw=; b=aIfBAlq0U2QmtEzQFDWe7gPxDqO03IMyx0qEtsTWPyvWAoENyGjIjegRoDQczvY3kL G8Nn2MrzJTE5bSkW4jlcUhzghZz5z5nWAVh630EP+rl7QA2qufkGtFLLOgd/hduWTC2s 4ty3RxP1BXr0XSCt5w8Ts2bmyXOCbZlpj6MTvho2GW/mgtTPQgSzjjC9u0XheTfT0AZ4 SFTAva3jRTzggo6xRwzbAnBwx4e3/KkLNJE/M4AOyTY+npRdJK7V77emVxD3Lrnbg+VG RnM6pSC8flmn5HNd5C3Z1MWFU517HMD+kOyE62F38cPwF9ZZnXmH39hzHFZ9Smai7zjE myRQ== X-Gm-Message-State: ALyK8tLbG2iVwFeT1cpKvTB6KQAoNbDIZQA5nP50mC9quw4oOROaSAyonuB4IVdd9OOaIQ== X-Received: by 10.36.19.16 with SMTP id 16mr1257845itz.76.1464905123864; Thu, 02 Jun 2016 15:05:23 -0700 (PDT) Received: from [10.0.10.3] (cpe-184-56-210-236.neo.res.rr.com. [184.56.210.236]) by smtp.googlemail.com with ESMTPSA id l131sm1407098iol.13.2016.06.02.15.05.23 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 02 Jun 2016 15:05:23 -0700 (PDT) Message-ID: <5750ADB7.8010409@gmail.com> Date: Thu, 02 Jun 2016 18:05:43 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Michael Grimm CC: "freebsd-jail@freebsd.org" Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? References: <140851342.3380283.1464808961455.JavaMail.yahoo.ref@mail.yahoo.com> <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com> <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org> In-Reply-To: <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 22:05:25 -0000 Michael Grimm wrote: > Sebastián Maruca via freebsd-jail wrote: > >> Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, has anyone tried it? Roger, it seems you are thumbing up my challenge... >> But I guess i'll have to stick with netgraph instead epair/if_bridge because the later is not so documented as the first one… > > Preamble: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, now 10.3-STABLE, and haven't seen any issues, sofar. Currently I do have 10 jails running, firewall is pf at the host, only. My servers are not big scaled ISP like, more small business-like, though. I am considering myself a hobby admin. > > > Here's my configuration that may show you one way to get that running, but I am sure your will have to tweak it to your needs: > > 1) Jails have been created by ezjail in the past, thus they are still at ezjail's infrastructure. But I do no longer use ezjail for starting or stopping my jails due to ezjail's lack of dealing with VNET jails (yet). So I do still have fstab definitions in /etc for all jails, e.g.: > > /etc/fstab.www > /path-to-your/jails/basejail /path-to-your/jails/www/basejail nullfs ro 0 0 > > 2) All external IPv4 or IPv6 addresses are NAT'ed or NAT66'ed to 10.1.1.x or fd00:dead:dead:beef::x > > 3) Networking regarding VNET jails defined in /etc/rc.conf: > > # set up one bridge interface > cloned_interfaces="bridge0" > > # needed for default routes within jails > ifconfig_bridge0="inet 10.1.1.254 netmask 255.255.255.0" > ifconfig_bridge0_ipv6="inet6 fd00:dead:dead:beef::254 prefixlen 64" > > 4) Thus, jails are controlled by jail(8) (shown for 3 example jails): > > /etc/rc.conf > ———————————————BEGIN------------------------ > jail_enable="YES" > jail_reverse_stop="YES" > jail_list="dns www mail" > ———————————————-END———————————— > > /etc/jail.conf: > # > # host dependent global settings > # > $ip6prefixLOCAL = "fd00:dead:dead:beef"; > > # > # global jail settings > # > host.hostname = "${name}"; > path = "/path-to-your/jails/${name}"; > mount.fstab = "/etc/fstab.${name}"; > exec.consolelog = "/var/log/jail_${name}_console.log"; > vnet = "new"; > vnet.interface = "epair${jailID}b"; > exec.clean; > mount.devfs; > persist; > > # > # network settings to apply/destroy during start/stop of every jail > # > exec.prestart = "sleep 2"; > exec.prestart += "ifconfig epair${jailID} create up"; > exec.prestart += "ifconfig bridge0 addm epair${jailID}a"; > exec.start = "/sbin/ifconfig lo0 127.0.0.1 up"; > exec.start += "/sbin/ifconfig epair${jailID}b inet ${ip4_addr}"; > exec.start += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr}"; > exec.start += "/sbin/route add default -gateway 10.1.1.254"; > exec.start += "/sbin/route add -inet6 default -gateway ${ip6prefixLOCAL}::254"; > #exec.stop = "/sbin/route del default"; > #exec.stop += "/sbin/route del -inet6 default"; > exec.stop += "/bin/sh /etc/rc.shutdown"; > exec.poststop = "ifconfig epair${jailID}a destroy"; > > # > # individual jail settings > # > mail { > $jailID = 1; > $ip4_addr = 10.1.1.1; > $ip6_addr = ${ip6prefixLOCAL}::1/64; > exec.start += "/bin/sh /etc/rc"; > } > > www { > $jailID = 2; > $ip4_addr = 10.1.1.2; > $ip6_addr = ${ip6prefixLOCAL}::2/64; > exec.start += "/bin/sh /etc/rc"; > } > > dns { > $jailID = 3; > $ip4_addr = 10.1.1.3; > $ip4_addr_2 = 10.1.1.4; > $ip6_addr = ${ip6prefixLOCAL}::3/64; > $ip6_addr_2 = ${ip6prefixLOCAL}::4/64; > exec.start += "/sbin/ifconfig epair${jailID}b inet ${ip4_addr_2} alias"; > exec.start += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr_2} alias"; > exec.start += "/bin/sh /etc/rc"; > } > > Now you can use "service jail" to start/stop your jails, e.g.: > > service jail stop > service jail restart dns > service jail start dns mail > > 5) NOTE: I am refraining from restarting VNET jails the hard way as shown above, and I am using a similar approach as iocage, namely "soft restarts". As this functionality isn't available in 10.3-STABLE (IIRC) I am using a homemade shell script instead. This script has to be run *inside* a jail which can be triggered from the outside (still using ezjail-admin) by e.g.: "sudo ezjail-admin console -e '/usr/local/etc/_JAIL_SOFT_RESTART' www" > > #!/bin/csh > > # > # restart jail services without removing jail and its network > # > > # > # global definitions > # > set LOGGER = "/usr/bin/logger -p user.info -t _JAIL_SOFT_RC" > set RCDIR = "/usr/local/etc/rc.d" > set TAB = " " > > # > # evaluate list of rc files in /usr/local/etc/rc.d > # > set RCFILES = `rcorder ${RCDIR}/* |& grep -v ^rcorder:` > > # > # evaluate reverse order of RCFILES > # > set RCFILES_REVERSE = "" > foreach rcname ( ${RCFILES} ) > set RCFILES_REVERSE = "${rcname} ${RCFILES_REVERSE}" > end > > # > # stop rc services > # > echo "stopping:" > foreach rcname ( ${RCFILES_REVERSE} ) > ${LOGGER} stopping ${rcname} > ${rcname} stop >& /dev/null > echo "${TAB}" ${rcname} > end > > # > # start rc services > # > echo "starting:" > foreach rcname ( ${RCFILES} ) > ${LOGGER} starting ${rcname} > ${rcname} start >& /dev/null > echo "${TAB}" ${rcname} > end > > exit 0 > > This script isn't perfect, and if you start or stop a jail you need to separate the relevant part. This can easily be coded into that script, I know. But I was lazy ;-) > > I hope that helps for a start. Again, I am sure you may need some tweaking at your site. > > Regards, > Michael > > Michael, You left out whether you had to compile the kernel with the vimage option or whether vimage was already included in the kernel?