Date: Thu, 2 Jun 2016 21:26:41 +0000 (UTC) From: =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= <juanperiz@yahoo.com.ar> To: Michael Grimm <trashcan@ellael.org>, "freebsd-jail@freebsd.org" <freebsd-jail@freebsd.org> Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? Message-ID: <377963018.4245125.1464902801251.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org> References: <140851342.3380283.1464808961455.JavaMail.yahoo.ref@mail.yahoo.com> <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com> <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Michael... even though you consider yourself as a admin hobbier, I can tell=
you have the "lend hander" top grade you're honored ;)
I'll start from this big step you're posting (and all the other which repli=
ed too) and carry on dancing 'til I got my jails running DMZ, VLAN and WAN =
like a pro...
Best Regards,Seba
De: Michael Grimm <trashcan@ellael.org>
Para: "freebsd-jail@freebsd.org" <freebsd-jail@freebsd.org>=20
Enviado: Jueves, 2 de junio, 2016 15:24:34
Asunto: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready?
=20
Sebasti=C3=A1n Maruca via freebsd-jail <freebsd-jail@freebsd.org> wrote:
> Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, has =
anyone tried it? Roger, it seems you are thumbing up my challenge...
> But I guess i'll have to stick with netgraph instead epair/if_bridge beca=
use the later is not so documented as the first one=E2=80=A6
Preamble: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, no=
w 10.3-STABLE, and haven't seen any issues, sofar. Currently I do have 10 j=
ails running, firewall is pf at the host, only. My servers are not big scal=
ed ISP like, more small business-like, though. I am considering myself a ho=
bby admin.=20
Here's my configuration that may show you one way to get that running, but =
I am sure your will have to tweak it to your needs:
1) Jails have been created by ezjail in the past, thus they are still at ez=
jail's infrastructure. But I do no longer use ezjail for starting or stoppi=
ng my jails due to ezjail's lack of dealing with VNET jails (yet). So I do =
still have fstab definitions in /etc for all jails, e.g.:
=C2=A0=C2=A0=C2=A0 /etc/fstab.www
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 /path-to-your/jails/basejail /path-to=
-your/jails/www/basejail nullfs ro 0 0=20
2) All external IPv4 or IPv6 addresses are NAT'ed or NAT66'ed to 10.1.1.x o=
r fd00:dead:dead:beef::x
3) Networking regarding VNET jails defined in /etc/rc.conf:
=C2=A0=C2=A0=C2=A0 # set up one bridge interface
=C2=A0=C2=A0=C2=A0 cloned_interfaces=3D"bridge0"
=C2=A0=C2=A0=C2=A0 # needed for default routes within jails
=C2=A0=C2=A0=C2=A0 ifconfig_bridge0=3D"inet 10.1.1.254 netmask 255.255.255.=
0"
=C2=A0=C2=A0=C2=A0 ifconfig_bridge0_ipv6=3D"inet6 fd00:dead:dead:beef::254 =
prefixlen 64"
4) Thus, jails are controlled by jail(8) (shown for 3 example jails):
=C2=A0=C2=A0=C2=A0 /etc/rc.conf
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =E2=80=94=E2=80=94=E2=80=94=E2=80=94=
=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=
=80=94=E2=80=94=E2=80=94BEGIN------------------------
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 jail_enable=3D"YES"
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 jail_reverse_stop=3D"YES"
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 jail_list=3D"dns www mail"
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =E2=80=94=E2=80=94=E2=80=94=E2=80=94=
=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=
=80=94=E2=80=94=E2=80=94-END=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=
=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94
=C2=A0=C2=A0=C2=A0 /etc/jail.conf:
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # host dependent global settings
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6prefixLOCAL=C2=A0=C2=A0=C2=A0 =C2=
=A0=C2=A0=C2=A0 =3D "fd00:dead:dead:beef";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # global jail settings
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 host.hostname=C2=A0=C2=A0=C2=A0 =C2=
=A0=C2=A0=C2=A0 =3D "${name}";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 path=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=
=C2=A0 =C2=A0=C2=A0=C2=A0 =3D "/path-to-your/jails/${name}";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 mount.fstab=C2=A0=C2=A0=C2=A0 =C2=A0=
=C2=A0=C2=A0 =3D "/etc/fstab.${name}";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.consolelog =C2=A0=C2=A0=C2=A0 =
=3D "/var/log/jail_${name}_console.log";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 vnet=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=
=C2=A0 =C2=A0=C2=A0=C2=A0 =3D "new";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 vnet.interface=C2=A0=C2=A0=C2=A0 =C2=
=A0=C2=A0=C2=A0 =3D "epair${jailID}b";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.clean;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 mount.devfs;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 persist;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # network settings to apply/destroy d=
uring start/stop of every jail
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.prestart=C2=A0=C2=A0=C2=A0 =C2=
=A0=C2=A0=C2=A0 =3D "sleep 2";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.prestart=C2=A0=C2=A0=C2=A0 =C2=
=A0=C2=A0=C2=A0 +=3D "ifconfig epair${jailID} create up";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.prestart=C2=A0=C2=A0=C2=A0 =C2=
=A0=C2=A0=C2=A0 +=3D "ifconfig bridge0 addm epair${jailID}a";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0=
=C2=A0=C2=A0 =3D "/sbin/ifconfig lo0 127.0.0.1 up";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0=
=C2=A0=C2=A0 +=3D "/sbin/ifconfig epair${jailID}b inet ${ip4_addr}";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0=
=C2=A0=C2=A0 +=3D "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr}";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0=
=C2=A0=C2=A0 +=3D "/sbin/route add default -gateway 10.1.1.254";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0=
=C2=A0=C2=A0 +=3D "/sbin/route add -inet6 default -gateway ${ip6prefixLOCAL=
}::254";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #exec.stop=C2=A0=C2=A0=C2=A0 =C2=A0=
=C2=A0=C2=A0 =3D "/sbin/route del default";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #exec.stop=C2=A0=C2=A0=C2=A0 =C2=A0=
=C2=A0=C2=A0 +=3D "/sbin/route del -inet6 default";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.stop=C2=A0=C2=A0=C2=A0 =C2=A0=C2=
=A0=C2=A0 +=3D "/bin/sh /etc/rc.shutdown";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.poststop =C2=A0=C2=A0=C2=A0 =C2=
=A0=C2=A0=C2=A0 =3D "ifconfig epair${jailID}a destroy";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # individual jail settings
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 mail {
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $jailID=C2=A0=C2=
=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =3D 1;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip4_addr=C2=A0=C2=
=A0=C2=A0 =3D 10.1.1.1;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6_addr=C2=A0=C2=
=A0=C2=A0 =3D ${ip6prefixLOCAL}::1/64;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=
=C2=A0=C2=A0 +=3D "/bin/sh /etc/rc";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 }
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 www {
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $jailID=C2=A0=C2=
=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =3D 2;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip4_addr=C2=A0=C2=
=A0=C2=A0 =3D 10.1.1.2;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6_addr=C2=A0=C2=
=A0=C2=A0 =3D ${ip6prefixLOCAL}::2/64;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=
=C2=A0=C2=A0 +=3D "/bin/sh /etc/rc";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 }
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 dns {
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $jailID=C2=A0=C2=
=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =3D 3;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip4_addr=C2=A0=C2=
=A0=C2=A0 =3D 10.1.1.3;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip4_addr_2=C2=A0=
=C2=A0=C2=A0 =3D 10.1.1.4;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6_addr=C2=A0=C2=
=A0=C2=A0 =3D ${ip6prefixLOCAL}::3/64;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6_addr_2=C2=A0=
=C2=A0=C2=A0 =3D ${ip6prefixLOCAL}::4/64;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=
=C2=A0=C2=A0 +=3D "/sbin/ifconfig epair${jailID}b inet=C2=A0 ${ip4_addr_2} =
alias";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=
=C2=A0=C2=A0 +=3D "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr_2} alias=
";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=
=C2=A0=C2=A0 +=3D "/bin/sh /etc/rc";
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 }
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20
=C2=A0=C2=A0=C2=A0 Now you can use "service jail" to start/stop your jails,=
e.g.:
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 service jail stop=20
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 service jail restart dns
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 service jail start dns mail
5) NOTE: I am refraining from restarting VNET jails the hard way as shown a=
bove, and I am using a similar approach as iocage, namely "soft restarts". =
As this functionality isn't available in 10.3-STABLE (IIRC) I am using a ho=
memade shell script instead. This script has to be run *inside* a jail whic=
h can be triggered from the outside (still using ezjail-admin) by e.g.: "su=
do ezjail-admin console -e '/usr/local/etc/_JAIL_SOFT_RESTART' www"
=C2=A0=C2=A0=C2=A0 #!/bin/csh
=C2=A0=C2=A0=C2=A0=20
=C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 # restart jail services without removing jail and its ne=
twork
=C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0=20
=C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 # global definitions
=C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 set LOGGER =3D "/usr/bin/logger -p user.info -t _JAIL_SO=
FT_RC"
=C2=A0=C2=A0=C2=A0 set RCDIR =3D "/usr/local/etc/rc.d"
=C2=A0=C2=A0=C2=A0 set TAB =3D "=C2=A0 =C2=A0 =C2=A0 =C2=A0 "
=C2=A0=C2=A0=C2=A0=20
=C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 # evaluate list of rc files in /usr/local/etc/rc.d
=C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 set RCFILES =3D `rcorder ${RCDIR}/* |& grep -v ^rcorder:=
`
=C2=A0=C2=A0=C2=A0=20
=C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 # evaluate reverse order of RCFILES
=C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 set RCFILES_REVERSE =3D ""
=C2=A0=C2=A0=C2=A0 foreach rcname ( ${RCFILES} )
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 set RCFILES_REVERSE =3D "${rcname} ${=
RCFILES_REVERSE}"
=C2=A0=C2=A0=C2=A0 end
=C2=A0=C2=A0=C2=A0=20
=C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 # stop rc services
=C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 echo "stopping:"
=C2=A0=C2=A0=C2=A0 foreach rcname ( ${RCFILES_REVERSE} )
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 ${LOGGER} stopping ${rcname}
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 ${rcname} stop >& /dev/null
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 echo "${TAB}" ${rcname}
=C2=A0=C2=A0=C2=A0 end
=C2=A0=C2=A0=C2=A0=20
=C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 # start rc services
=C2=A0=C2=A0=C2=A0 #
=C2=A0=C2=A0=C2=A0 echo "starting:"
=C2=A0=C2=A0=C2=A0 foreach rcname ( ${RCFILES} )
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 ${LOGGER} starting ${rcname}
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 ${rcname} start >& /dev/null
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 echo "${TAB}" ${rcname}
=C2=A0=C2=A0=C2=A0 end
=C2=A0=C2=A0=C2=A0=20
=C2=A0=C2=A0=C2=A0 exit 0
This script isn't perfect, and if you start or stop a jail you need to sepa=
rate the relevant part. This can easily be coded into that script, I know. =
But I was lazy ;-)
I hope that helps for a start. Again, I am sure you may need some tweaking =
at your site.
Regards,
Michael
_______________________________________________
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscribe@freebsd.org"
From owner-freebsd-jail@freebsd.org Thu Jun 2 22:05:25 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6EB8B67FF8
for <freebsd-jail@mailman.ysv.freebsd.org>;
Thu, 2 Jun 2016 22:05:25 +0000 (UTC)
(envelope-from luzar722@gmail.com)
Received: from mail-it0-x22c.google.com (mail-it0-x22c.google.com
[IPv6:2607:f8b0:4001:c0b::22c])
(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
(Client CN "smtp.gmail.com",
Issuer "Google Internet Authority G2" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 9F8661796
for <freebsd-jail@freebsd.org>; Thu, 2 Jun 2016 22:05:25 +0000 (UTC)
(envelope-from luzar722@gmail.com)
Received: by mail-it0-x22c.google.com with SMTP id i127so63881716ita.1
for <freebsd-jail@freebsd.org>; Thu, 02 Jun 2016 15:05:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
h=message-id:date:from:user-agent:mime-version:to:cc:subject
:references:in-reply-to:content-transfer-encoding;
bh=e/rPc/1dbW5I8qmQan0CHYWFwykS7g60OSrNYbmOLuw=;
b=ez18eA1R9mIHJy2OcyJ7nnJ1sAMy8GI2FI9ztcN7jQ8Ouo45XzHAyO2pNKgZYIaU4s
5Z0BfVTWsXIhxFQYTYUTV3o9elsBYVWTskoO1iCYp+Lsw3Rgb/AVbIfw+xXmWBPeAfNO
+SY/Va+Gq8DUbd9Zm07yGsMPrTjdve/JtT2h6tKkgwNnLoRCq3GFC/JBV9h89EucdX8J
hmWh2TsJOGSE06YrubZSH50OG3OwpIq+dTKF5wxLo2xyqa5orivIxXezXXRyKNDFkasl
QdBAdUAh4swLyLt24VctsuZzMVH21F1oAORvIS7F58JMoYHJ/5s65kbSMCJWHXm3gTCC
I8uw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20130820;
h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to
:cc:subject:references:in-reply-to:content-transfer-encoding;
bh=e/rPc/1dbW5I8qmQan0CHYWFwykS7g60OSrNYbmOLuw=;
b=aIfBAlq0U2QmtEzQFDWe7gPxDqO03IMyx0qEtsTWPyvWAoENyGjIjegRoDQczvY3kL
G8Nn2MrzJTE5bSkW4jlcUhzghZz5z5nWAVh630EP+rl7QA2qufkGtFLLOgd/hduWTC2s
4ty3RxP1BXr0XSCt5w8Ts2bmyXOCbZlpj6MTvho2GW/mgtTPQgSzjjC9u0XheTfT0AZ4
SFTAva3jRTzggo6xRwzbAnBwx4e3/KkLNJE/M4AOyTY+npRdJK7V77emVxD3Lrnbg+VG
RnM6pSC8flmn5HNd5C3Z1MWFU517HMD+kOyE62F38cPwF9ZZnXmH39hzHFZ9Smai7zjE
myRQ==
X-Gm-Message-State: ALyK8tLbG2iVwFeT1cpKvTB6KQAoNbDIZQA5nP50mC9quw4oOROaSAyonuB4IVdd9OOaIQ==
X-Received: by 10.36.19.16 with SMTP id 16mr1257845itz.76.1464905123864;
Thu, 02 Jun 2016 15:05:23 -0700 (PDT)
Received: from [10.0.10.3] (cpe-184-56-210-236.neo.res.rr.com.
[184.56.210.236])
by smtp.googlemail.com with ESMTPSA id l131sm1407098iol.13.2016.06.02.15.05.23
(version=TLSv1/SSLv3 cipher=OTHER);
Thu, 02 Jun 2016 15:05:23 -0700 (PDT)
Message-ID: <5750ADB7.8010409@gmail.com>
Date: Thu, 02 Jun 2016 18:05:43 -0400
From: Ernie Luzar <luzar722@gmail.com>
User-Agent: Thunderbird 2.0.0.24 (Windows/20100228)
MIME-Version: 1.0
To: Michael Grimm <trashcan@ellael.org>
CC: "freebsd-jail@freebsd.org" <freebsd-jail@freebsd.org>
Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready?
References: <140851342.3380283.1464808961455.JavaMail.yahoo.ref@mail.yahoo.com>
<140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com>
<2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org>
In-Reply-To: <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
<mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>;
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
<mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 02 Jun 2016 22:05:25 -0000
Michael Grimm wrote:
> Sebastián Maruca via freebsd-jail <freebsd-jail@freebsd.org> wrote:
>
>> Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, has anyone tried it? Roger, it seems you are thumbing up my challenge...
>> But I guess i'll have to stick with netgraph instead epair/if_bridge because the later is not so documented as the first one…
>
> Preamble: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, now 10.3-STABLE, and haven't seen any issues, sofar. Currently I do have 10 jails running, firewall is pf at the host, only. My servers are not big scaled ISP like, more small business-like, though. I am considering myself a hobby admin.
>
>
> Here's my configuration that may show you one way to get that running, but I am sure your will have to tweak it to your needs:
>
> 1) Jails have been created by ezjail in the past, thus they are still at ezjail's infrastructure. But I do no longer use ezjail for starting or stopping my jails due to ezjail's lack of dealing with VNET jails (yet). So I do still have fstab definitions in /etc for all jails, e.g.:
>
> /etc/fstab.www
> /path-to-your/jails/basejail /path-to-your/jails/www/basejail nullfs ro 0 0
>
> 2) All external IPv4 or IPv6 addresses are NAT'ed or NAT66'ed to 10.1.1.x or fd00:dead:dead:beef::x
>
> 3) Networking regarding VNET jails defined in /etc/rc.conf:
>
> # set up one bridge interface
> cloned_interfaces="bridge0"
>
> # needed for default routes within jails
> ifconfig_bridge0="inet 10.1.1.254 netmask 255.255.255.0"
> ifconfig_bridge0_ipv6="inet6 fd00:dead:dead:beef::254 prefixlen 64"
>
> 4) Thus, jails are controlled by jail(8) (shown for 3 example jails):
>
> /etc/rc.conf
> ———————————————BEGIN------------------------
> jail_enable="YES"
> jail_reverse_stop="YES"
> jail_list="dns www mail"
> ———————————————-END————————————
>
> /etc/jail.conf:
> #
> # host dependent global settings
> #
> $ip6prefixLOCAL = "fd00:dead:dead:beef";
>
> #
> # global jail settings
> #
> host.hostname = "${name}";
> path = "/path-to-your/jails/${name}";
> mount.fstab = "/etc/fstab.${name}";
> exec.consolelog = "/var/log/jail_${name}_console.log";
> vnet = "new";
> vnet.interface = "epair${jailID}b";
> exec.clean;
> mount.devfs;
> persist;
>
> #
> # network settings to apply/destroy during start/stop of every jail
> #
> exec.prestart = "sleep 2";
> exec.prestart += "ifconfig epair${jailID} create up";
> exec.prestart += "ifconfig bridge0 addm epair${jailID}a";
> exec.start = "/sbin/ifconfig lo0 127.0.0.1 up";
> exec.start += "/sbin/ifconfig epair${jailID}b inet ${ip4_addr}";
> exec.start += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr}";
> exec.start += "/sbin/route add default -gateway 10.1.1.254";
> exec.start += "/sbin/route add -inet6 default -gateway ${ip6prefixLOCAL}::254";
> #exec.stop = "/sbin/route del default";
> #exec.stop += "/sbin/route del -inet6 default";
> exec.stop += "/bin/sh /etc/rc.shutdown";
> exec.poststop = "ifconfig epair${jailID}a destroy";
>
> #
> # individual jail settings
> #
> mail {
> $jailID = 1;
> $ip4_addr = 10.1.1.1;
> $ip6_addr = ${ip6prefixLOCAL}::1/64;
> exec.start += "/bin/sh /etc/rc";
> }
>
> www {
> $jailID = 2;
> $ip4_addr = 10.1.1.2;
> $ip6_addr = ${ip6prefixLOCAL}::2/64;
> exec.start += "/bin/sh /etc/rc";
> }
>
> dns {
> $jailID = 3;
> $ip4_addr = 10.1.1.3;
> $ip4_addr_2 = 10.1.1.4;
> $ip6_addr = ${ip6prefixLOCAL}::3/64;
> $ip6_addr_2 = ${ip6prefixLOCAL}::4/64;
> exec.start += "/sbin/ifconfig epair${jailID}b inet ${ip4_addr_2} alias";
> exec.start += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr_2} alias";
> exec.start += "/bin/sh /etc/rc";
> }
>
> Now you can use "service jail" to start/stop your jails, e.g.:
>
> service jail stop
> service jail restart dns
> service jail start dns mail
>
> 5) NOTE: I am refraining from restarting VNET jails the hard way as shown above, and I am using a similar approach as iocage, namely "soft restarts". As this functionality isn't available in 10.3-STABLE (IIRC) I am using a homemade shell script instead. This script has to be run *inside* a jail which can be triggered from the outside (still using ezjail-admin) by e.g.: "sudo ezjail-admin console -e '/usr/local/etc/_JAIL_SOFT_RESTART' www"
>
> #!/bin/csh
>
> #
> # restart jail services without removing jail and its network
> #
>
> #
> # global definitions
> #
> set LOGGER = "/usr/bin/logger -p user.info -t _JAIL_SOFT_RC"
> set RCDIR = "/usr/local/etc/rc.d"
> set TAB = " "
>
> #
> # evaluate list of rc files in /usr/local/etc/rc.d
> #
> set RCFILES = `rcorder ${RCDIR}/* |& grep -v ^rcorder:`
>
> #
> # evaluate reverse order of RCFILES
> #
> set RCFILES_REVERSE = ""
> foreach rcname ( ${RCFILES} )
> set RCFILES_REVERSE = "${rcname} ${RCFILES_REVERSE}"
> end
>
> #
> # stop rc services
> #
> echo "stopping:"
> foreach rcname ( ${RCFILES_REVERSE} )
> ${LOGGER} stopping ${rcname}
> ${rcname} stop >& /dev/null
> echo "${TAB}" ${rcname}
> end
>
> #
> # start rc services
> #
> echo "starting:"
> foreach rcname ( ${RCFILES} )
> ${LOGGER} starting ${rcname}
> ${rcname} start >& /dev/null
> echo "${TAB}" ${rcname}
> end
>
> exit 0
>
> This script isn't perfect, and if you start or stop a jail you need to separate the relevant part. This can easily be coded into that script, I know. But I was lazy ;-)
>
> I hope that helps for a start. Again, I am sure you may need some tweaking at your site.
>
> Regards,
> Michael
>
>
Michael,
You left out whether you had to compile the kernel with the vimage
option or whether vimage was already included in the kernel?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?377963018.4245125.1464902801251.JavaMail.yahoo>