Date: Tue, 10 Jan 2006 02:16:10 -0800 From: Andrew Fremantle <freebsd@skyhawk.ca> To: freebsd-questions@freebsd.org, freebsd-ipfw@freebsd.org Subject: Advanced IPFW2 Forward rule problem / bug / misunderstanding Message-ID: <43C3896A.7090704@skyhawk.ca>
next in thread | raw e-mail | index | archive | help
Hello,
I have searched the lists for information pertaining to this problem,
but I haven't been able to find anything relevant to my attempted usage
of IPFWs "forward" action. If there are any preexisting threads that
address my concern, please direct me to them.
I have also sent this message to the freebsd-ipfw list, as this seems
consistent with some other threads I see in their archive.
Here's the situation :
I have two ISPs, each providing two IPs. One of these ISPs is providing
IPs on totally different subnets, and is MAC sensitive. I have two
internal servers (Actually, just one listening on two addresses), and I
want this server to be available externally to both ISPs. (We're
migrating ISPs, and we don't want any interruption in service). I am
using port forwarding in NATd to allow the necessary ports through to
the server. My problem comes with the replies - FreeBSD has only one
default gateway, and all traffic going out, regardless of which external
IP address it is from, goes to that gateway. Since ISP2 doesen't care
much for routing traffic from ISP1, and vice-versa, I have a problem.
I should note here that I am not trying to load balance - I am perfectly
happy with all outbound LAN connections being NATted over one link, I
just need the ability to service inbound connections on all four IPs.
I am using forward rules in my firewall to match packets belonging to
these other interfaces, to forward them to the appropriate gateway.
According to the manpage for ipfw, "If /ipaddr/ is not a local address,
then the port number (if specified) is ignored, and the packet will be
forwarded to the remote address, using the route as found in the local
routing table for that IP." I interpret that as "The packet's next hop
will be compared to the routing table, and routed out the appropriate
interface to reach that next hop". The problem is that doesen't seem to
be happening. I have tried fiddling a few knobs to no effect -
specifically net.inet.ip.fastforwarding, net.inet.ip.sourceroute and
net.inet.ip.accept_sourceroute.
Telus is the "legacy" ISP, so when I'm trying these rules all the
inbound server requests are from the two Telus interfaces. I have
numbered a rule here "42000". This rule will catch all kinds of packets
outbound from ${ext1_ip}:80 and ${ext1_ip}:443 to clients on the internet.
This tells me my inbound NAT translation is working, the packets are
getting to the server, replies are coming back, they're matching my
forward rules, but still going out the wrong interface anyways!
As attached as I am to the idea of doing this via ipfw, if anyone has
any suggestions on alternate methods to achieve the same results, I'd
love to hear them!!!
On to the technical details - I have obscured IP addresses here, but the
networks and subnet masks remain the same
-------------------------------------------
bsdbox# uname -a
FreeBSD bsdbox 6.0-RELEASE-p1 FreeBSD 6.0-RELEASE-p1 #1: Mon Jan 9
08:15:08 PST 2006 root@bsdbox:/usr/obj/usr/src/sys/BSDBOX i386
-------------------------------------------
bsdbox# cat /usr/src/sys/i386/conf/BSDBOX
.
.
.
### FIREWALLING
options IPFIREWALL
options IPFIREWALL_FORWARD
options IPFIREWALL_FORWARD_EXTENDED ( I just did this to test -
it made no difference)
options IPDIVERT
-------------------------------------------
bsdbox# cat /etc/rc.conf
.
.
.
##########
## Networking
##########
gateway_enable="YES"
## Ensure interface configuration and Firewall script remain consistent!!
defaultrouter="24.85.92.1"
ifconfig_rl0="192.168.1.1"
ifconfig_vr0="142.179.109.xxx netmask 255.255.248.0"
ifconfig_vr1="216.232.85.xxx netmask 255.255.254.0"
ifconfig_rue0="24.85.9x.xxx netmask 255.255.252.0"
ifconfig_rue0_alias0="24.85.9x.xxx netmask 255.255.255.255"
natd_enable="NO"
firewall_enable="YES"
firewall_script="/usr/local/etc/firewall.telus+shaw-test"
-------------------------------------------------
bsdbox# cat /usr/local/etc/firewall.telus+shaw-test
##### firewall.telus+shaw 0.9.8
# Aquire variables from /etc/rc.conf
if [ -r /etc/rc.conf ]; then
. /etc/rc.conf
fi
fwcmd="/sbin/ipfw -q"
##########
## THIS SCRIPT REQUIRES THE FOLLOWING VARIABLES
## TO BE CORRECTLY DEFINED!
##########
##########
# PRIMARY external interface (Telus)
ext1="vr0" # Device name
ext1_ip="142.179.109.xxx" # IP Address
ext1_gw="142.179.104.254" # IP Gateway
ext1_bc="142.179.111.255" # Broadcast Address
ext1_srv="192.168.1.10" # Server IP Address
##########
##########
# SECONDARY external interface (Telus)
ext2="vr1" # Device name
ext2_ip="216.232.85.xxx" # IP Address
ext2_nm="255.255.254.0" # Network Mask
ext2_bc="216.232.85.255" # Broadcast Address
ext2_gw="216.232.84.254" # IP Gateway
ext2_srv="192.168.1.11" # IP Address of internal server
##########
##########
# Shaw Cable Interface(s)
# PRIMARY IP
shaw="rue0" # Device Name
shaw_ip="24.85.93.xxx" # IP Address
shaw_nm="255.255.252.0" # Network Mask
shaw_bc="24.85.95.255" # Broadcast Address
shaw_gw="24.85.92.1" # IP Gateway
srv1_int="192.168.1.10" # Internal IP of server
srv1_ext="24.85.93.xxx" # External IP of server (Same as ${shaw_ip})
# SECONDARY IP
srv2_int="192.168.1.11" # Internal IP of server
srv2_ext="24.85.93.xxx" # External IP of server
##########
##########
# INTERNAL interface
int="rl0" # Device name
int_ip="192.168.1.1" # IP Address
int_nm="255.255.255.0" # Network Mask
int_bc="192.168.1.255" # Broadcast Address
##########
## I have to handle NATd manually from this script because it's got
## too many connection specific options.
nat_in="8667"
nat_out="8669"
## Kill any running instance
if [ -r /var/run/natd.pid ]; then
kill -9 `cat /var/run/natd.pid`
sleep 1
fi
# And run our new NATd
/sbin/natd -log_ipfw_denied -i ${nat_in} -o ${nat_out} -s -m -u -n
${shaw} -punch_fw 36000:100 -redirect_port tcp ${ext1_srv}:22
${ext1_ip}:xxxx -redirect_port tcp ${ext1_srv}:53 ${ext1_ip}:53
-redirect_port tcp ${ext1_srv}:80 ${ext1_ip}:80 -redirect_port tcp
${ext1_srv}:443 ${ext1_ip}:443 -redirect_port udp ${ext1_srv}:53
${ext1_ip}:53 -redirect_port tcp ${ext2_srv}:80 ${ext2_ip}:80
-redirect_port tcp ${ext2_srv}:443 ${ext2_ip}:443 -redirect_port tcp
${srv1_int}:22 ${shaw_ip}:xxxx -redirect_port tcp ${srv1_int}:53
${shaw_ip}:53 -redirect_port udp ${srv1_int}:53 ${shaw_ip}:53
-redirect_port tcp ${srv1_int}:80 ${shaw_ip}:80 -redirect_port tcp
${srv1_int}:443 ${shaw_ip}:443 -redirect_port tcp ${srv2_int}:80
${srv2_ext}:80 -redirect_port tcp ${srv2_int}:443 ${srv2_ext}:443
# Blow away the existing firewall ruleset
${fwcmd} flush
##########
# Now I start defining rules to handle traffic
##########
# Divide and Conquer based on interface and direction
## Inbound Internal
${fwcmd} add 0500 skipto 5000 all from any to any in recv ${int}
## Outbound Internal
${fwcmd} add skipto 10000 all from any to any out xmit ${int}
# Inbound Primary External (Telus)
${fwcmd} add skipto 15000 all from any to any in recv ${ext1}
# Outbound Primary External (Telus)
${fwcmd} add skipto 20000 all from any to any out xmit ${ext1}
# Inbound Secondary External (Telus)
${fwcmd} add skipto 25000 all from any to any in recv ${ext2}
# Outbound Secondary External (Telus)
${fwcmd} add skipto 30000 all from any to any out xmit ${ext2}
# Inbound Shaw
${fwcmd} add skipto 35000 all from any to any in recv ${shaw}
# Outbound Shaw
${fwcmd} add skipto 40000 all from any to any out xmit ${shaw}
# Local Loopback
${fwcmd} add allow all from any to any via lo0
# If I still haven't matched it, it's a damn weirdo. Drop it.
${fwcmd} add reset all from any to any
### *****************
### Inbound Internal Traffic
### *****************
# Drop all packets not originating on my network
${fwcmd} add 5000 drop all from not ${int_ip}:${int_nm} to any //
Inbound Internal
# Skip the next rule for all packets destined to the firewall itself ( &
Broadcast )
${fwcmd} add 5010 skipto 5040 all from any to \( ${int_ip} or ${int_bc} \)
# Drop all packets destined to my internal network
${fwcmd} add 5020 drop all from any to ${int_ip}:${int_nm}
# Allow existing conversations, and new ICMP and SSH connections
${fwcmd} add 5040 check-state
${fwcmd} add allow icmp from any to \( ${int_ip} or ${int_bc} \) keep-state
${fwcmd} add allow tcp from any to ${int_ip} 22 setup keep-state
# Pass packets to NATd for possible translation
${fwcmd} add divert ${nat_out} all from any to any
# Redirect to gateway for ext1 if it's an existing connection on ext1
${fwcmd} add forward ${ext1_gw} all from ${ext1_ip} to any
# Redirect to gateway for ext2 if it's an existing connection on ext2
${fwcmd} add forward ${ext2_gw} all from ${ext2_ip} to any
# Allow all packets adjusted by NATd
${fwcmd} add allow ip from \( ${shaw_ip} or ${srv2_ext} \) to any
# Reject this packet
${fwcmd} add reset log all from any to any
### *****************
### Outbound Internal Traffic
### *****************
# Allow Existing Conversations
${fwcmd} add 10000 check-state // Outbound Internal
# Allow me to start ICMP, SSH and DNS sessions
${fwcmd} add allow icmp from ${int_ip} to ${int_ip}:${int_nm} keep-state
${fwcmd} add allow tcp from ${int_ip} to ${int_ip}:${int_nm} 22 keep-state
${fwcmd} add allow udp from ${int_ip} to ${srv1_int} 53 keep-state
# Allow NATd translated traffic through
${fwcmd} add allow all from not ${int_ip}:${int_nm} to ${int_ip}:${int_nm}
# Reject this packet
${fwcmd} add reset log all from any to any
### *****************
### Inbound External (primary) Traffic
### *****************
# Ensure this packet did not originate from a private network
${fwcmd} add 15000 drop all from 192.168.0.0/16 to any // Inbound Telus
Primary
${fwcmd} add 15020 drop all from 172.16.0.0/12 to any
${fwcmd} add 15030 drop all from 10.0.0.0/8 to any
${fwcmd} add 15040 drop all from 127.0.0.0/8 to any
${fwcmd} add 15050 drop all from 0.0.0.0/8 to any
${fwcmd} add 15060 drop all from 169.254.0.0/16 to any
${fwcmd} add 15070 drop all from 192.0.2.0/24 to any
${fwcmd} add 15080 drop all from 204.152.64.0/23 to any
${fwcmd} add 15090 drop all from 224.0.0.0/3 to any
# Drop the packet if it's not broadcast or destined to us
${fwcmd} add 15500 skipto 15600 all from any to \( ${ext1_ip} or
${ext1_bc} \)
${fwcmd} add 15520 deny all from any to any
# Pass to NATd for possible reverse-translation
${fwcmd} add 15600 divert ${nat_in} all from any to any
# Pass anything NATd has translated
${fwcmd} add allow all from any to ${int_ip}:${int_nm}
# Drop whatever's left
${fwcmd} add deny all from any to any
### *****************
### Outbound External (primary) Traffic
### *****************
# Deny all traffic to private nets
${fwcmd} add 20000 drop all from any to 192.168.0.0/16 // Outbound Telus
Primary
${fwcmd} add drop all from any to 172.16.0.0/12
${fwcmd} add drop all from any to 10.0.0.0/8
${fwcmd} add drop all from any to 127.0.0.0/8
${fwcmd} add drop all from any to 0.0.0.0/8
${fwcmd} add drop all from any to 169.254.0.0/16
${fwcmd} add drop all from any to 192.0.2.0/24
${fwcmd} add drop all from any to 204.152.64.0/23
${fwcmd} add drop all from any to 224.0.0.0/3
# Allow outbound traffic from this machine (NATd translated)
${fwcmd} add allow all from ${ext1_ip} to any
# Drop whatever's left
${fwcmd} add deny all from any to any
### *****************
### Inbound External (secondary) Traffic
### *****************
# Ensure this packet did not originate from a private network
${fwcmd} add 25000 drop all from 192.168.0.0/16 to any // Inbound Telus
Secondary
${fwcmd} add 25020 drop all from 172.16.0.0/12 to any
${fwcmd} add 25030 drop all from 10.0.0.0/8 to any
${fwcmd} add 25040 drop all from 127.0.0.0/8 to any
${fwcmd} add 25050 drop all from 0.0.0.0/8 to any
${fwcmd} add 25060 drop all from 169.254.0.0/16 to any
${fwcmd} add 25070 drop all from 192.0.2.0/24 to any
${fwcmd} add 25080 drop all from 204.152.64.0/23 to any
${fwcmd} add 25090 drop all from 224.0.0.0/3 to any
# Drop the packet if it's not broadcast or destined to us
${fwcmd} add 25100 skipto 25500 all from any to \( ${ext2_ip} or
${ext2_bc} \)
${fwcmd} add 25120 deny all from any to any
# Pass to NATd for possible reverse-translation
${fwcmd} add 25500 divert ${nat_in} all from any to any
# Allow all packets translated by NATd
${fwcmd} add allow all from any to ${int_ip}:${int_nm}
# Drop whatever's left
${fwcmd} add reject all from any to any
### *****************
### outbound External (secondary) Traffic
### *****************
# Deny all traffic to private nets
${fwcmd} add 30000 drop all from any to 192.168.0.0/16 // Outbound Telus
Secondary
${fwcmd} add drop all from any to 172.16.0.0/12
${fwcmd} add drop all from any to 10.0.0.0/8
${fwcmd} add drop all from any to 127.0.0.0/8
${fwcmd} add drop all from any to 0.0.0.0/8
${fwcmd} add drop all from any to 169.254.0.0/16
${fwcmd} add drop all from any to 192.0.2.0/24
${fwcmd} add drop all from any to 204.152.64.0/23
${fwcmd} add drop all from any to 224.0.0.0/3
# Pass all packets from ext2_ip
${fwcmd} add pass all from ${ext2_ip} to any
# Drop whatever's left
${fwcmd} add deny all from any to any
### *****************
### Inbound Shaw Traffic
### *****************
# Ensure this packet did not originate on a private network
${fwcmd} add 35000 drop all from 192.168.0.0/16 to any // Inbound Shaw
${fwcmd} add 35010 drop all from 172.16.0.0/12 to any
${fwcmd} add 35020 drop all from 10.0.0.0/8 to any
${fwcmd} add 35030 drop all from 127.0.0.0/8 to any
${fwcmd} add 35040 drop all from 0.0.0.0/8 to any
${fwcmd} add 35050 drop all from 169.254.0.0/16 to any
${fwcmd} add 35060 drop all from 192.0.2.0/24 to any
${fwcmd} add 35070 drop all from 204.152.64.0/23 to any
${fwcmd} add 35080 drop all from 224.0.0.0/3 to any
# Drop the packet if it's not broadcast or destined to us
${fwcmd} add 35100 skipto 35500 all from any to \( ${shaw_ip} or
${srv2_ext} or ${shaw_bc} \)
${fwcmd} add 35200 deny all from any to any
# Allow Inbound ICMP and SSH (To me)
${fwcmd} add 35510 allow icmp from any to me
${fwcmd} add 35520 allow tcp from any to me 22
# Pass to NATd for possible reverse translation
${fwcmd} add 35550 divert ${nat_in} all from any to any
# NATd punches holes from 36000 - 36100
# Pass anything NATd has translated
${fwcmd} add 36200 allow all from any to ${int_ip}:${int_nm}
# Deny whatever's left
${fwcmd} add deny all from any to any
### *****************
### Outbound Shaw Traffic
### *****************
# Deny all traffic to private nets
${fwcmd} add 40000 drop all from any to 192.168.0.0/16 // Outbound Shaw
${fwcmd} add drop all from any to 172.16.0.0/12
${fwcmd} add drop all from any to 10.0.0.0/8
${fwcmd} add drop all from any to 127.0.0.0/8
${fwcmd} add drop all from any to 0.0.0.0/8
${fwcmd} add drop all from any to 169.254.0.0/16
${fwcmd} add drop all from any to 192.0.2.0/24
${fwcmd} add drop all from any to 204.152.64.0/23
${fwcmd} add drop all from any to 224.0.0.0/3
# Allow all outbound traffic from me (includes NATted stuff)
${fwcmd} add allow all from \( ${shaw_ip} or ${srv2_ext} or ${shaw_bc}
\) to any
# Drop whatever's left
${fwcmd} add 42000 deny log all from any to any
######
######
# Log all packets falling through the firewall
${fwcmd} add 65534 deny log all from any to any
------------------------------------------
Excerpts from /var/log/security
Jan 9 09:11:55 bsdbox kernel: ipfw: 40900 Deny TCP 142.179.109.xxx:443
207.216.1
81.74:1111 out via rue0
Jan 9 09:12:07 bsdbox kernel: ipfw: 40900 Deny TCP 142.179.109.xxx:443
154.20.34
.158:61974 out via rue0
Jan 9 09:14:40 bsdbox kernel: ipfw: 40900 Deny TCP 142.179.109.xxx:443
64.180.16
4.232:44707 out via rue0
------------------------------------------
All questions, comments, suggestions and flames (Well, okay maybe not
the flames) are welcome!
My apologies for the extremely long post.
- Andrew
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?43C3896A.7090704>