Date: Mon, 27 Jul 2009 10:26:46 -0700 From: Julian Elischer <julian@elischer.org> To: Oleg Sharoyko <os@sfedu.ru> Cc: freebsd-net@freebsd.org Subject: Re: Wrong outgoing interface with multiple routing tables Message-ID: <4A6DE356.6040006@elischer.org> In-Reply-To: <1248704237.96833.127.camel@brain.cc.rsu.ru> References: <1248704237.96833.127.camel@brain.cc.rsu.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
Oleg Sharoyko wrote: > Hello! > > I'm having a trouble with multiple routing tables (FreeBSD 7.2 release). > Either I'm missing something in my setup or packets for daemons started > with setfib are being sent out via the wrong interface. > > What I'd like to implement: > > em0 - internal management network with ip address 10.2.5.2/24 and > default route 10.2.5.1 > > em1 - public interface to be used in jail with ip address > 195.208.245.229/27 and default route 195.208.245.225 > > Here are my routing tables: > > r61net-fbsdhost-1, / # setfib -0 netstat -rn > Routing tables > > Internet: > Destination Gateway Flags Refs Use Netif Expire > default 10.2.5.1 UGS 0 350 em0 > 10.2.5.0/24 link#1 UC 0 0 em0 > 10.2.5.1 00:1e:4a:b4:ea:c0 UHLW 2 0 em0 1182 > 127.0.0.1 127.0.0.1 UH 0 30 lo0 > > r61net-fbsdhost-1, / # setfib -1 netstat -rn > Routing tables > > Internet: > Destination Gateway Flags Refs Use Netif Expire > default 195.208.245.225 UGS 0 0 em1 > 195.208.245.224/27 link#2 UC 0 0 em1 > 195.208.245.225 link#2 UHLW 2 0 em1 so far, all looks correct. > > Firewall: > > r61net-fbsdhost-1, / # ipfw show > 00001 0 0 setfib 1 ip from any to any in recv em1 good > 00010 0 0 count ip from any to any dst-port 2222 fib 0 > 00011 0 0 count ip from any 2222 to any fib 0 > 00012 0 0 count ip from any to any dst-port 2222 fib 1 > 00013 0 0 count ip from any 2222 to any fib 1 > 00100 0 0 allow ip from any to any via lo0 > 00200 0 0 deny ip from any to 127.0.0.0/8 > 00300 0 0 deny ip from 127.0.0.0/8 to any > 65000 30 2648 allow ip from any to any > 65535 0 0 deny ip from any to any > > > With this setup almost everything works as I expect. For example ICMP > echo requests and responses are being received and sent via em1. Both > when ping runs on this host as "setfib 1 ping other_host" and when other > host pings ip address of em1. Connection attempts (setfib 1 telnet > other_host) are also being sent out of the right interface. But when it > comes to the daemons I run into troubles. > > I use sshd for tests (have also tried other daemons with no luck): > > r61net-fbsdhost-1, / # setfib 1 /usr/sbin/sshd -o 'ListenAddress 195.208.245.229:2222' -D Are you running this from inetd?. (doesnt look like it) btw is it 1 or -1? or -F 1? I can't remember if I supported just '1'. > > sshd is bound only to ip address of em1: > > r61net-fbsdhost-1, / # sockstat | grep 2222 > root sshd 839 3 tcp4 195.208.245.229:2222 *:* > > While doing telnet 195.208.249.229 2222 from another host I got following packet traces: > > r61net-fbsdhost-1, / # tcpdump -i em0 port 2222 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on em0, link-type EN10MB (Ethernet), capture size 96 bytes > 17:39:34.872475 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: S 2590499299:2590499299(0) ack 3939022576 win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 3248254533 147282318> > 17:39:34.902622 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: P 1:41(40) ack 1 win 8326 <nop,nop,timestamp 3248254533 147282318> > 17:39:37.572271 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: P 41:60(19) ack 7 win 8326 <nop,nop,timestamp 3248254593 147282585> > 17:39:37.572293 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: F 60:60(0) ack 7 win 8326 <nop,nop,timestamp 3248254593 147282585> > 17:39:37.572986 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: . ack 8 win 8325 <nop,nop,timestamp 3248254593 147282585> > > r61net-fbsdhost-1, / # tcpdump -i em1 port 2222 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes > 17:39:34.872370 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: S 3939022575:3939022575(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 147282318 0> > 17:39:34.872803 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 2590499300 win 8326 <nop,nop,timestamp 147282318 3248254533> > 17:39:35.002882 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 41 win 8326 <nop,nop,timestamp 147282331 3248254533> > 17:39:37.571659 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: P 0:6(6) ack 41 win 8326 <nop,nop,timestamp 147282585 3248254533> > 17:39:37.572923 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 61 win 8323 <nop,nop,timestamp 147282585 3248254593> > 17:39:37.572945 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: F 6:6(0) ack 61 win 8326 <nop,nop,timestamp 147282585 3248254593> > > And firewall counters: > > r61net-fbsdhost-1, / # ipfw show > 00001 6 326 setfib 1 ip from any to any in recv em1 > 00010 0 0 count ip from any to any dst-port 2222 fib 0 > 00011 5 327 count ip from any 2222 to any fib 0 > 00012 6 326 count ip from any to any dst-port 2222 fib 1 > 00013 0 0 count ip from any 2222 to any fib 1 > 00100 0 0 allow ip from any to any via lo0 > 00200 0 0 deny ip from any to 127.0.0.0/8 > 00300 0 0 deny ip from 127.0.0.0/8 to any > 65000 60 5057 allow ip from any to any > 65535 0 0 deny ip from any to any > > So the packets, generated by sshd are being sent out via em0 instead of > em1. > > With > > ipfw add 2 setfib 1 ip from 195.208.245.229 to any > > outgoing packets are being tagged with correct fib, but still sent via > em0. yes becasue on outgoing packets the firewall is too late to influence that. > > With > > ipfw add 60003 fwd 195.208.245.225 src-ip me src-ip 195.208.245.224/27 not dst-ip 195.208.245.224/27 > > first SYN packet from 195.208.245.229 is being sent correctly via em1, > but I cannot see any further packets at all: > > r61net-fbsdhost-1, / # tcpdump -i em1 port 2222 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes > 18:01:56.665341 IP brain.cc.rsu.ru.50435 > stat.r61.net.2222: S 2484180116:2484180116(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 147415433 0> > 18:01:56.665463 IP stat.r61.net.2222 > brain.cc.rsu.ru.50435: S 3905497961:3905497961(0) ack 2484180117 win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 3376909218 147415433> > 18:01:56.665798 IP brain.cc.rsu.ru.50435 > stat.r61.net.2222: . ack 1 win 8326 <nop,nop,timestamp 147415433 3376909218> > > and no packets at em0. TCP connection establishes but no data packets > come from daemon which is rather weird. > > I would appreciate any help with this issue. try adding a '-' on the command and get back to me. >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4A6DE356.6040006>