Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 04 Feb 2012 17:05:45 +0000
From:      Poul-Henning Kamp <phk@freebsd.org>
To:        hackers@freebsd.org
Subject:   A dual-ISP hack with jail/vnet and ipfw
Message-ID:  <12192.1328375145@critter.freebsd.dk>

next in thread | raw e-mail | index | archive | help

Natd(8) knows how to deal with multiple NAT instances for different
interfaces, which is useful when you have multiple ISPs.

The problem with it, is that it becomes incredibly hairy to configure
your IPFW rules, in particular if you have other policy to implement
too.

I spent some quality time with a 9.0-Stable nanobsd image today,
and the script below is my proof of concept of a simpler way to
do that.

The idea is to let a jail deal with the two ISPs and use an epair
to deliver a "normal default route interface" to the rest of the
firewall, making its configuration simpler and easier to understand.

A discard interface is used to hold 127/8 and the default route in
the jail.  If the default route were to one of the ISP interfaces,
and that if_ goes down, you loose the default route, and packets
don't even reach ipfw(8) in the first place.

Depending on how paranoid you are, you can run the natd(8) in an
empty jail, (see last line).

In a sense this is "DMZ-in-a-box", and there are a number of
interesting ideas to explore, for instance running an openvpn
instance in the jail, but put its TUN/TAP interface in the
default (non-jail) vnet.

The only disadvantage I have found yet, is that you see all packets
with EPAIR_IN destination, instead of the physical destination IP,
the source address however is OK.

Another thing I noticed is that we should probably consider giving
tcpdump/pcap in the unjailed part the ability to packet-dump
interfaces in all vnets.  This would match all the other the
"semi-transparent" properties of jails.

NB: This is only a proof of concept, you may want
to think more about the IPFW rules before going live.

Enjoy,

Poul-Henning

PS: feel free to adopt for any purpose you like, including doc/wiki/etc.

#!/bin/sh

set -x

# ISP #1
VR2_IP=192.168.60.101
VR2_GW=192.168.60.1

# ISP #2
VR3_IP=10.0.0.1
VR3_GW=10.0.0.2

# IN/OUT ethernet pair
EPAIR_OUT=192.168.5.2
EPAIR_IN=192.168.5.1
EPAIR_WID=/30

# Kill old jail
jail -r ext > /dev/null 2>&1 || true 

jdir=/var/tmp/jail_ext
rm -rf $jdir
if true ; then
	mkdir $jdir
	(
		cd /
		find \
		    libexec/ld-elf.so.1 \
		    sbin/ipfw \
		    sbin/natd \
		    sbin/dhclient \
		    sbin/ifconfig \
		    sbin/sysctl \
		    lib/libalias.so.7 \
		    lib/libbsdxml.so.4 \
		    lib/libjail.so.1 \
		    lib/libsbuf.so.6 \
		    lib/libipx.so.5 \
		    lib/libc.so.7 \
		    lib/libutil.so.9 \
		    lib/libalias_*.so \
		    etc/libalias.conf \
		    etc/services \
		    -print | cpio -dumpv $jdir
		
	)
else
	jdir=/
fi


# Create new jail
jail -c vnet name=ext path=$jdir persist

F="jexec ext ipfw"
$F -f flush

$F add	1	deny		ip from any to any

# No filtering on the epair
$F add 100	allow		ip from any to any via epair0b

# Dispatch to proper natd instance
$F add 1200      skipto 22000    ip from any to any in via vr2
$F add 1300      skipto 23000    ip from any to any in via vr3

# The global instance, outgoing packets
$F add 1400	divert 40000    ip from any to any
$F add 1410	fwd $VR3_GW	ip from $VR3_IP to any

# Non-matched // TRAFIC POLICY //
$F add 1420	skipto 12000	ip from any to any prob .5
$F add 1430	skipto 13000	ip from any to any

# Outgoing vr2
$F add 12000	divert 20000	ip from any to any
$F add 12100    fwd $VR2_GW	ip from $VR2_IP to any
$F add 12200	deny		ip from any to any

# Outgoing vr3
$F add 13000	divert 30000	ip from any to any
$F add 13100    fwd $VR3_GW	ip from $VR3_IP to any
$F add 13200	deny		ip from any to any

# Incoming vr2
$F add 22000    divert 20000    ip from any to any
$F add 22100	allow		ip from any to any

# Incoming vr3
$F add 23000    divert 30000    ip from any to any
$F add 23100	allow		ip from any to any

# Set up a discard interface to hold the default route
ifconfig disc0 destroy
ifconfig disc0 create
ifconfig disc0 vnet ext
jexec ext ifconfig disc0 127.0.0.1/8
jexec ext route add default -iface disc0

# Create ethernet pair
ifconfig epair0a destroy
ifconfig epair0 create
ifconfig epair0a ${EPAIR_IN}${EPAIR_WID}

# Default route, (not quite default for my tests)
route del -net 10/8
route add -net 10/8 ${EPAIR_OUT}

# Move other end into jail
ifconfig epair0b vnet ext

# Move external interfaces to jail
ifconfig vr2 vnet ext
ifconfig vr3 vnet ext

jexec ext ifconfig epair0b ${EPAIR_OUT}${EPAIR_WID}

# Get addresses from you ISP's (DHCP/static)
jexec ext dhclient -b vr2 
jexec ext ifconfig vr3 10.0.0.1/30

# Enable forwarding in the jail
jexec ext sysctl net.inet.ip.forwarding=1

# Build a natd.conf for the jail, allow inbound ssh
(
echo deny_incoming
echo globalport 40000
echo alias_address 127.0.0.1
echo
echo instance vr2
echo port 20000
echo alias_address $VR2_IP
echo redirect_port tcp ${EPAIR_IN}:22 ${VR2_IP}:22
echo 
echo instance vr3
echo port 30000
echo alias_address $VR3_IP
echo redirect_port tcp ${EPAIR_IN}:22 ${VR3_IP}:22
) > $jdir/etc/natd_ext.conf

jexec ext natd -f /etc/natd_ext.conf

# Remove the roadblock
$F delete 1

# Remove the evidence
# XXX: Even safer: put jail in md(4) disk, rm, remount r/o
rm -rf $jdir/*


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?12192.1328375145>