From owner-freebsd-ipfw@FreeBSD.ORG Fri May 20 22:52:58 2005 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8082116A4D1; Fri, 20 May 2005 22:52:58 +0000 (GMT) Received: from sonic.ux6.net (sonic.ux6.net [64.62.252.147]) by mx1.FreeBSD.org (Postfix) with ESMTP id 42B9943D88; Fri, 20 May 2005 22:52:58 +0000 (GMT) (envelope-from miha@sonic.ux6.net) Received: from miha by sonic.ux6.net with local (Exim 4.51 (FreeBSD)) id 1DZGMU-000MDn-4p; Fri, 20 May 2005 15:52:58 -0700 Date: Fri, 20 May 2005 15:52:58 -0700 From: Mikhail To: freebsd-net@freebsd.org, freebsd-ipfw@freebsd.org Message-ID: <20050520225258.GA77121@sonic.ux6.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="/9DWx/yDrRhgMJTb" Content-Disposition: inline X-Operating-System: FreeBSD X-PGP-Key: http://sonic.ux6.net/~miha/gpg_miha.asc User-Agent: Mutt/1.5.9i Subject: Weired routing issues X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 May 2005 22:52:58 -0000 --/9DWx/yDrRhgMJTb Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello lists, (I apologize for cross-posting to freebsd-ipfw). I came across very weird issue with routing today - somehow, FreeBSD box ro= utes packets it shouldn't be routing. Here it goes: One of our customers has VPN connection to one of the remote servers. We ar= e using OpenVPN-2 tunnel for this purpose because it is relatively easy to = setup and maintain. The VPN tunnel is used for customer to access internet through remote VPN s= erver to which he connects, so it basically looks as: [customer]--->[secure vpn tun via internet]--->[vpn server]--->[internet fo= r customer] The whole setup is pretty straightforward: On customer's end, customer connects to the Net through DSL modem, and has = a small DMZ locally. There is also a FreeBSD box, which is used to setup Op= enVPN tunnel. Both FreeBSD box and customer are in one switch, within one s= ubnet, and see the Net through the DSL modem (which is plugged into the sam= e switch for uplink). On servers' end, there is a standalone FreeBSD box connected to the Net. Since customer's FreeBSD box is NAT'ed behind DSL modem, OpenVPN tunnel was= setup the way both sides see each other internally (within tunnel) using l= ocal IPs: 192.168.10.3, iface tun0 - the IP remote VPN server has once VPN connection= gets established 192.168.10.4, iface tun0 - the IP customer's FreeBSD has once VPN connectio= n gets established The subnet customer uses inside his DMZ is 10.0.0.0/24. The following IPs a= re used: 10.0.0.1 - DSL modem 10.0.0.2 - FreeBSD VPN box 10.0.0.3 - customer's computer So my aim was to route 10.0.0.3 to the Net through remote VPN server. This = was accomplished with the following steps: 1) run natd on remote VPN server, with divert rules as: divert 8668 ip from 10.0.0.0/24 to any out xmit rl0 2) add routing entry on remote VPN server for 10.0.0.0/24 net: /sbin/route add 10.0.0.0/24 192.168.10.4 3) turn on interface forwarding on both sides 4) on customer's FreeBSD box setup packet forwarding to remote VPN server a= s: ipfw add fwd 192.168.10.3 ip from 10.0.0.3 to any So the final step remains is to change default gateway on 10.0.0.3 to FreeB= SD VPN box (10.0.0.2) instead of DSL modem. And voila - 10.0.0.3 sees inter= net through this VPN tunnel and all works beautifully. So far all seems great, however now I need to route this customer back thro= ugh his DSL modem. To do that, customer simply changes his default gateway = back to DSL modem (10.0.0.1). All works beautifully, but one serious proble= m just occured -=20 customer did not change his route back to 10.0.0.1 (he still had FreeBSD's = 10.0.0.2 in his settings), and I removed "fwd" route from ipfw by mistake, = and what we saw was that customer was still able to surf the Net! However, = his was routed through his normal path (not VPN). - ipfw was flushed - there's no natd, nor any other nat on FreeBSD box - no routing or anything but it still acted as a gateway, routing packets it shouldn't be. So after such a long thread (which I could probably cut a lot), my question= is - how is it possible that local FreeBSD box passed packets that it shou= ldn't be passing? I bashed my head against the wall for few hours now, and still couldn't fig= ure - every time I remove "fwd" rule from ipfw, customer goes normal route = with no other changes. I tweaked all sysctls I could think of, and came to if I disable interface = forwarding - customer gets cut, and he does not see the Net; enable interfa= ce forwarding back and he sees the Net again. I'm totally lost at this poin= t. Here's what was used to setup VPN tunnel: - OpenVPN-2.0 - remote VPN server runs FreeBSD-4.10 - local VPN server runs FreeBSD-5.2 - customer's PC runs Windows XP - 10/100mbit network switch - DSL modem local FreeBSD /etc/sysctl & network: ############################## kern.corefile=3D/tmp/%N.code kern.logsigexit=3D0 net.inet.ip.forwarding=3D1 net.inet.ip.fastforwarding=3D1 net.inet.ip.fw.one_pass=3D0 net.inet.tcp.inflight_enable=3D1 ############################## $ ifconfig sk0: flags=3D8843 mtu 1500 inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255 inet6 fe80::211:2fff:fe87:919d%sk0 prefixlen 64 scopeid 0x1 ether 00:11:2f:87:91:9d media: Ethernet autoselect (100baseTX ) status: active plip0: flags=3D8810 mtu 1500 lo0: flags=3D8049 mtu 16384 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 tun0: flags=3D8051 mtu 1500 inet6 fe80::211:2fff:fe87:919d%tun0 prefixlen 64 scopeid 0x4 inet 192.168.10.4 --> 192.168.10.3 netmask 0xffffffff Opened by PID 454 $ netstat -nr Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 10.0.0.1 UGS 0 77 sk0 10/24 link#1 UC 0 0 sk0 10.0.0.1 00:a0:c5:9a:d7:49 UHLW 1 0 sk0 841 127.0.0.1 127.0.0.1 UH 0 0 lo0 192.168.10.3 192.168.10.4 UH 0 65 tun0 Anything else I need to specify? I would highly appreciate any help/tips on how to find and remedy this issu= e. Sincerely, --/9DWx/yDrRhgMJTb Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCjmpJoHRayxAsyDsRAq2QAKCsG4pzpKamKABPoUOxjflDSTAxzACfbAH2 L1Z831+tYEthHqqnkJEprts= =LW0g -----END PGP SIGNATURE----- --/9DWx/yDrRhgMJTb--