Date: Thu, 2 Jun 2016 20:24:34 +0200 From: Michael Grimm <trashcan@ellael.org> To: "freebsd-jail@freebsd.org" <freebsd-jail@freebsd.org> Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? Message-ID: <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org> In-Reply-To: <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com> References: <140851342.3380283.1464808961455.JavaMail.yahoo.ref@mail.yahoo.com> <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Sebasti=C3=A1n Maruca via freebsd-jail <freebsd-jail@freebsd.org> wrote:
> Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, =
has anyone tried it? Roger, it seems you are thumbing up my challenge...
> But I guess i'll have to stick with netgraph instead epair/if_bridge =
because the later is not so documented as the first one=E2=80=A6
Preamble: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, =
now 10.3-STABLE, and haven't seen any issues, sofar. Currently I do have =
10 jails running, firewall is pf at the host, only. My servers are not =
big scaled ISP like, more small business-like, though. I am considering =
myself a hobby admin.=20
Here's my configuration that may show you one way to get that running, =
but I am sure your will have to tweak it to your needs:
1) Jails have been created by ezjail in the past, thus they are still at =
ezjail's infrastructure. But I do no longer use ezjail for starting or =
stopping my jails due to ezjail's lack of dealing with VNET jails (yet). =
So I do still have fstab definitions in /etc for all jails, e.g.:
/etc/fstab.www
/path-to-your/jails/basejail =
/path-to-your/jails/www/basejail nullfs ro 0 0=20
2) All external IPv4 or IPv6 addresses are NAT'ed or NAT66'ed to =
10.1.1.x or fd00:dead:dead:beef::x
3) Networking regarding VNET jails defined in /etc/rc.conf:
# set up one bridge interface
cloned_interfaces=3D"bridge0"
# needed for default routes within jails
ifconfig_bridge0=3D"inet 10.1.1.254 netmask 255.255.255.0"
ifconfig_bridge0_ipv6=3D"inet6 fd00:dead:dead:beef::254 =
prefixlen 64"
4) Thus, jails are controlled by jail(8) (shown for 3 example jails):
/etc/rc.conf
=
=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=
=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94BEGIN---------=
---------------
jail_enable=3D"YES"
jail_reverse_stop=3D"YES"
jail_list=3D"dns www mail"
=
=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=
=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94-END=E2=80=94=E2=
=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=
=94=E2=80=94=E2=80=94
/etc/jail.conf:
#
# host dependent global settings
#
$ip6prefixLOCAL =3D "fd00:dead:dead:beef";
=09
#
# global jail settings
#
host.hostname =3D "${name}";
path =3D =
"/path-to-your/jails/${name}";
mount.fstab =3D "/etc/fstab.${name}";
exec.consolelog =3D =
"/var/log/jail_${name}_console.log";
vnet =3D "new";
vnet.interface =3D "epair${jailID}b";
exec.clean;
mount.devfs;
persist;
=09
#
# network settings to apply/destroy during start/stop of =
every jail
#
exec.prestart =3D "sleep 2";
exec.prestart +=3D "ifconfig epair${jailID} =
create up";
exec.prestart +=3D "ifconfig bridge0 addm =
epair${jailID}a";
exec.start =3D "/sbin/ifconfig lo0 =
127.0.0.1 up";
exec.start +=3D "/sbin/ifconfig =
epair${jailID}b inet ${ip4_addr}";
exec.start +=3D "/sbin/ifconfig =
epair${jailID}b inet6 ${ip6_addr}";
exec.start +=3D "/sbin/route add default =
-gateway 10.1.1.254";
exec.start +=3D "/sbin/route add -inet6 =
default -gateway ${ip6prefixLOCAL}::254";
#exec.stop =3D "/sbin/route del default";
#exec.stop +=3D "/sbin/route del -inet6 =
default";
exec.stop +=3D "/bin/sh /etc/rc.shutdown";
exec.poststop =3D "ifconfig epair${jailID}a =
destroy";
=09
#
# individual jail settings
#
mail {
$jailID =3D 1;
$ip4_addr =3D 10.1.1.1;
$ip6_addr =3D ${ip6prefixLOCAL}::1/64;
exec.start +=3D "/bin/sh /etc/rc";
}
=09
www {
$jailID =3D 2;
$ip4_addr =3D 10.1.1.2;
$ip6_addr =3D ${ip6prefixLOCAL}::2/64;
exec.start +=3D "/bin/sh /etc/rc";
}
=09
dns {
$jailID =3D 3;
$ip4_addr =3D 10.1.1.3;
$ip4_addr_2 =3D 10.1.1.4;
$ip6_addr =3D ${ip6prefixLOCAL}::3/64;
$ip6_addr_2 =3D ${ip6prefixLOCAL}::4/64;
exec.start +=3D "/sbin/ifconfig =
epair${jailID}b inet ${ip4_addr_2} alias";
exec.start +=3D "/sbin/ifconfig =
epair${jailID}b inet6 ${ip6_addr_2} alias";
exec.start +=3D "/bin/sh /etc/rc";
}
=09
Now you can use "service jail" to start/stop your jails, e.g.:
service jail stop=20
service jail restart dns
service jail start dns mail
5) NOTE: I am refraining from restarting VNET jails the hard way as =
shown above, and I am using a similar approach as iocage, namely "soft =
restarts". As this functionality isn't available in 10.3-STABLE (IIRC) I =
am using a homemade shell script instead. This script has to be run =
*inside* a jail which can be triggered from the outside (still using =
ezjail-admin) by e.g.: "sudo ezjail-admin console -e =
'/usr/local/etc/_JAIL_SOFT_RESTART' www"
#!/bin/csh
=09
#
# restart jail services without removing jail and its network
#
=09
#
# global definitions
#
set LOGGER =3D "/usr/bin/logger -p user.info -t _JAIL_SOFT_RC"
set RCDIR =3D "/usr/local/etc/rc.d"
set TAB =3D " "
=09
#
# evaluate list of rc files in /usr/local/etc/rc.d
#
set RCFILES =3D `rcorder ${RCDIR}/* |& grep -v ^rcorder:`
=09
#
# evaluate reverse order of RCFILES
#
set RCFILES_REVERSE =3D ""
foreach rcname ( ${RCFILES} )
set RCFILES_REVERSE =3D "${rcname} ${RCFILES_REVERSE}"
end
=09
#
# stop rc services
#
echo "stopping:"
foreach rcname ( ${RCFILES_REVERSE} )
${LOGGER} stopping ${rcname}
${rcname} stop >& /dev/null
echo "${TAB}" ${rcname}
end
=09
#
# start rc services
#
echo "starting:"
foreach rcname ( ${RCFILES} )
${LOGGER} starting ${rcname}
${rcname} start >& /dev/null
echo "${TAB}" ${rcname}
end
=09
exit 0
This script isn't perfect, and if you start or stop a jail you need to =
separate the relevant part. This can easily be coded into that script, I =
know. But I was lazy ;-)
I hope that helps for a start. Again, I am sure you may need some =
tweaking at your site.
Regards,
Michael
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2CD81649-9D95-44B8-B0E3-DA38B8C3F31B>