Date: Fri, 20 May 2005 15:52:58 -0700 From: Mikhail <miha@sonic.ux6.net> To: freebsd-net@freebsd.org, freebsd-ipfw@freebsd.org Subject: Weired routing issues Message-ID: <20050520225258.GA77121@sonic.ux6.net>
next in thread | raw e-mail | index | archive | help
--/9DWx/yDrRhgMJTb
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Hello lists,
(I apologize for cross-posting to freebsd-ipfw).
I came across very weird issue with routing today - somehow, FreeBSD box ro=
utes packets it shouldn't be routing.
Here it goes:
One of our customers has VPN connection to one of the remote servers. We ar=
e using OpenVPN-2 tunnel for this purpose because it is relatively easy to =
setup and maintain.
The VPN tunnel is used for customer to access internet through remote VPN s=
erver to which he connects, so it basically looks as:
[customer]--->[secure vpn tun via internet]--->[vpn server]--->[internet fo=
r customer]
The whole setup is pretty straightforward:
On customer's end, customer connects to the Net through DSL modem, and has =
a small DMZ locally. There is also a FreeBSD box, which is used to setup Op=
enVPN tunnel. Both FreeBSD box and customer are in one switch, within one s=
ubnet, and see the Net through the DSL modem (which is plugged into the sam=
e switch for uplink).
On servers' end, there is a standalone FreeBSD box connected to the Net.
Since customer's FreeBSD box is NAT'ed behind DSL modem, OpenVPN tunnel was=
setup the way both sides see each other internally (within tunnel) using l=
ocal IPs:
192.168.10.3, iface tun0 - the IP remote VPN server has once VPN connection=
gets established
192.168.10.4, iface tun0 - the IP customer's FreeBSD has once VPN connectio=
n gets established
The subnet customer uses inside his DMZ is 10.0.0.0/24. The following IPs a=
re used:
10.0.0.1 - DSL modem
10.0.0.2 - FreeBSD VPN box
10.0.0.3 - customer's computer
So my aim was to route 10.0.0.3 to the Net through remote VPN server. This =
was accomplished with the following steps:
1) run natd on remote VPN server, with divert rules as:
divert 8668 ip from 10.0.0.0/24 to any out xmit rl0
2) add routing entry on remote VPN server for 10.0.0.0/24 net:
/sbin/route add 10.0.0.0/24 192.168.10.4
3) turn on interface forwarding on both sides
4) on customer's FreeBSD box setup packet forwarding to remote VPN server a=
s:
ipfw add fwd 192.168.10.3 ip from 10.0.0.3 to any
So the final step remains is to change default gateway on 10.0.0.3 to FreeB=
SD VPN box (10.0.0.2) instead of DSL modem. And voila - 10.0.0.3 sees inter=
net through this VPN tunnel and all works beautifully.
So far all seems great, however now I need to route this customer back thro=
ugh his DSL modem. To do that, customer simply changes his default gateway =
back to DSL modem (10.0.0.1). All works beautifully, but one serious proble=
m just occured -=20
customer did not change his route back to 10.0.0.1 (he still had FreeBSD's =
10.0.0.2 in his settings), and I removed "fwd" route from ipfw by mistake, =
and what we saw was that customer was still able to surf the Net! However, =
his was routed through his normal path (not VPN).
- ipfw was flushed
- there's no natd, nor any other nat on FreeBSD box
- no routing or anything
but it still acted as a gateway, routing packets it shouldn't be.
So after such a long thread (which I could probably cut a lot), my question=
is - how is it possible that local FreeBSD box passed packets that it shou=
ldn't be passing?
I bashed my head against the wall for few hours now, and still couldn't fig=
ure - every time I remove "fwd" rule from ipfw, customer goes normal route =
with no other changes.
I tweaked all sysctls I could think of, and came to if I disable interface =
forwarding - customer gets cut, and he does not see the Net; enable interfa=
ce forwarding back and he sees the Net again. I'm totally lost at this poin=
t.
Here's what was used to setup VPN tunnel:
- OpenVPN-2.0
- remote VPN server runs FreeBSD-4.10
- local VPN server runs FreeBSD-5.2
- customer's PC runs Windows XP
- 10/100mbit network switch
- DSL modem
local FreeBSD /etc/sysctl & network:
##############################
kern.corefile=3D/tmp/%N.code
kern.logsigexit=3D0
net.inet.ip.forwarding=3D1
net.inet.ip.fastforwarding=3D1
net.inet.ip.fw.one_pass=3D0
net.inet.tcp.inflight_enable=3D1
##############################
$ ifconfig
sk0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
inet6 fe80::211:2fff:fe87:919d%sk0 prefixlen 64 scopeid 0x1
ether 00:11:2f:87:91:9d
media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>)
status: active
plip0: flags=3D8810<POINTOPOINT,SIMPLEX,MULTICAST> mtu 1500
lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
tun0: flags=3D8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
inet6 fe80::211:2fff:fe87:919d%tun0 prefixlen 64 scopeid 0x4
inet 192.168.10.4 --> 192.168.10.3 netmask 0xffffffff
Opened by PID 454
$ netstat -nr
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.0.0.1 UGS 0 77 sk0
10/24 link#1 UC 0 0 sk0
10.0.0.1 00:a0:c5:9a:d7:49 UHLW 1 0 sk0 841
127.0.0.1 127.0.0.1 UH 0 0 lo0
192.168.10.3 192.168.10.4 UH 0 65 tun0
Anything else I need to specify?
I would highly appreciate any help/tips on how to find and remedy this issu=
e.
Sincerely,
--/9DWx/yDrRhgMJTb
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (FreeBSD)
iD8DBQFCjmpJoHRayxAsyDsRAq2QAKCsG4pzpKamKABPoUOxjflDSTAxzACfbAH2
L1Z831+tYEthHqqnkJEprts=
=LW0g
-----END PGP SIGNATURE-----
--/9DWx/yDrRhgMJTb--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050520225258.GA77121>