From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 4 17:22:07 2012 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07F6A106566B for ; Sat, 4 Feb 2012 17:22:07 +0000 (UTC) (envelope-from phk@freebsd.org) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 4D02C8FC08 for ; Sat, 4 Feb 2012 17:22:06 +0000 (UTC) Received: from critter.freebsd.dk (critter.freebsd.dk [192.168.61.3]) by phk.freebsd.dk (Postfix) with ESMTP id 5BCB95DFA for ; Sat, 4 Feb 2012 17:05:46 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.5/8.14.5) with ESMTP id q14H5jTR012193 for ; Sat, 4 Feb 2012 17:05:46 GMT (envelope-from phk@freebsd.org) To: hackers@freebsd.org From: Poul-Henning Kamp Content-Type: text/plain; charset=ISO-8859-1 Date: Sat, 04 Feb 2012 17:05:45 +0000 Message-ID: <12192.1328375145@critter.freebsd.dk> Cc: Subject: A dual-ISP hack with jail/vnet and ipfw X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Feb 2012 17:22:07 -0000 Natd(8) knows how to deal with multiple NAT instances for different interfaces, which is useful when you have multiple ISPs. The problem with it, is that it becomes incredibly hairy to configure your IPFW rules, in particular if you have other policy to implement too. I spent some quality time with a 9.0-Stable nanobsd image today, and the script below is my proof of concept of a simpler way to do that. The idea is to let a jail deal with the two ISPs and use an epair to deliver a "normal default route interface" to the rest of the firewall, making its configuration simpler and easier to understand. A discard interface is used to hold 127/8 and the default route in the jail. If the default route were to one of the ISP interfaces, and that if_ goes down, you loose the default route, and packets don't even reach ipfw(8) in the first place. Depending on how paranoid you are, you can run the natd(8) in an empty jail, (see last line). In a sense this is "DMZ-in-a-box", and there are a number of interesting ideas to explore, for instance running an openvpn instance in the jail, but put its TUN/TAP interface in the default (non-jail) vnet. The only disadvantage I have found yet, is that you see all packets with EPAIR_IN destination, instead of the physical destination IP, the source address however is OK. Another thing I noticed is that we should probably consider giving tcpdump/pcap in the unjailed part the ability to packet-dump interfaces in all vnets. This would match all the other the "semi-transparent" properties of jails. NB: This is only a proof of concept, you may want to think more about the IPFW rules before going live. Enjoy, Poul-Henning PS: feel free to adopt for any purpose you like, including doc/wiki/etc. #!/bin/sh set -x # ISP #1 VR2_IP=192.168.60.101 VR2_GW=192.168.60.1 # ISP #2 VR3_IP=10.0.0.1 VR3_GW=10.0.0.2 # IN/OUT ethernet pair EPAIR_OUT=192.168.5.2 EPAIR_IN=192.168.5.1 EPAIR_WID=/30 # Kill old jail jail -r ext > /dev/null 2>&1 || true jdir=/var/tmp/jail_ext rm -rf $jdir if true ; then mkdir $jdir ( cd / find \ libexec/ld-elf.so.1 \ sbin/ipfw \ sbin/natd \ sbin/dhclient \ sbin/ifconfig \ sbin/sysctl \ lib/libalias.so.7 \ lib/libbsdxml.so.4 \ lib/libjail.so.1 \ lib/libsbuf.so.6 \ lib/libipx.so.5 \ lib/libc.so.7 \ lib/libutil.so.9 \ lib/libalias_*.so \ etc/libalias.conf \ etc/services \ -print | cpio -dumpv $jdir ) else jdir=/ fi # Create new jail jail -c vnet name=ext path=$jdir persist F="jexec ext ipfw" $F -f flush $F add 1 deny ip from any to any # No filtering on the epair $F add 100 allow ip from any to any via epair0b # Dispatch to proper natd instance $F add 1200 skipto 22000 ip from any to any in via vr2 $F add 1300 skipto 23000 ip from any to any in via vr3 # The global instance, outgoing packets $F add 1400 divert 40000 ip from any to any $F add 1410 fwd $VR3_GW ip from $VR3_IP to any # Non-matched // TRAFIC POLICY // $F add 1420 skipto 12000 ip from any to any prob .5 $F add 1430 skipto 13000 ip from any to any # Outgoing vr2 $F add 12000 divert 20000 ip from any to any $F add 12100 fwd $VR2_GW ip from $VR2_IP to any $F add 12200 deny ip from any to any # Outgoing vr3 $F add 13000 divert 30000 ip from any to any $F add 13100 fwd $VR3_GW ip from $VR3_IP to any $F add 13200 deny ip from any to any # Incoming vr2 $F add 22000 divert 20000 ip from any to any $F add 22100 allow ip from any to any # Incoming vr3 $F add 23000 divert 30000 ip from any to any $F add 23100 allow ip from any to any # Set up a discard interface to hold the default route ifconfig disc0 destroy ifconfig disc0 create ifconfig disc0 vnet ext jexec ext ifconfig disc0 127.0.0.1/8 jexec ext route add default -iface disc0 # Create ethernet pair ifconfig epair0a destroy ifconfig epair0 create ifconfig epair0a ${EPAIR_IN}${EPAIR_WID} # Default route, (not quite default for my tests) route del -net 10/8 route add -net 10/8 ${EPAIR_OUT} # Move other end into jail ifconfig epair0b vnet ext # Move external interfaces to jail ifconfig vr2 vnet ext ifconfig vr3 vnet ext jexec ext ifconfig epair0b ${EPAIR_OUT}${EPAIR_WID} # Get addresses from you ISP's (DHCP/static) jexec ext dhclient -b vr2 jexec ext ifconfig vr3 10.0.0.1/30 # Enable forwarding in the jail jexec ext sysctl net.inet.ip.forwarding=1 # Build a natd.conf for the jail, allow inbound ssh ( echo deny_incoming echo globalport 40000 echo alias_address 127.0.0.1 echo echo instance vr2 echo port 20000 echo alias_address $VR2_IP echo redirect_port tcp ${EPAIR_IN}:22 ${VR2_IP}:22 echo echo instance vr3 echo port 30000 echo alias_address $VR3_IP echo redirect_port tcp ${EPAIR_IN}:22 ${VR3_IP}:22 ) > $jdir/etc/natd_ext.conf jexec ext natd -f /etc/natd_ext.conf # Remove the roadblock $F delete 1 # Remove the evidence # XXX: Even safer: put jail in md(4) disk, rm, remount r/o rm -rf $jdir/* -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.