From owner-freebsd-net@FreeBSD.ORG Mon Jul 27 17:26:43 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2D02C1065688 for ; Mon, 27 Jul 2009 17:26:43 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outO.internet-mail-service.net (outo.internet-mail-service.net [216.240.47.238]) by mx1.freebsd.org (Postfix) with ESMTP id 0D6D78FC19 for ; Mon, 27 Jul 2009 17:26:42 +0000 (UTC) (envelope-from julian@elischer.org) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id CBF05CD35; Mon, 27 Jul 2009 10:26:42 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 56B722D6018; Mon, 27 Jul 2009 10:26:42 -0700 (PDT) Message-ID: <4A6DE356.6040006@elischer.org> Date: Mon, 27 Jul 2009 10:26:46 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.22 (Macintosh/20090605) MIME-Version: 1.0 To: Oleg Sharoyko References: <1248704237.96833.127.camel@brain.cc.rsu.ru> In-Reply-To: <1248704237.96833.127.camel@brain.cc.rsu.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: Wrong outgoing interface with multiple routing tables X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jul 2009 17:26:48 -0000 Oleg Sharoyko wrote: > Hello! > > I'm having a trouble with multiple routing tables (FreeBSD 7.2 release). > Either I'm missing something in my setup or packets for daemons started > with setfib are being sent out via the wrong interface. > > What I'd like to implement: > > em0 - internal management network with ip address 10.2.5.2/24 and > default route 10.2.5.1 > > em1 - public interface to be used in jail with ip address > 195.208.245.229/27 and default route 195.208.245.225 > > Here are my routing tables: > > r61net-fbsdhost-1, / # setfib -0 netstat -rn > Routing tables > > Internet: > Destination Gateway Flags Refs Use Netif Expire > default 10.2.5.1 UGS 0 350 em0 > 10.2.5.0/24 link#1 UC 0 0 em0 > 10.2.5.1 00:1e:4a:b4:ea:c0 UHLW 2 0 em0 1182 > 127.0.0.1 127.0.0.1 UH 0 30 lo0 > > r61net-fbsdhost-1, / # setfib -1 netstat -rn > Routing tables > > Internet: > Destination Gateway Flags Refs Use Netif Expire > default 195.208.245.225 UGS 0 0 em1 > 195.208.245.224/27 link#2 UC 0 0 em1 > 195.208.245.225 link#2 UHLW 2 0 em1 so far, all looks correct. > > Firewall: > > r61net-fbsdhost-1, / # ipfw show > 00001 0 0 setfib 1 ip from any to any in recv em1 good > 00010 0 0 count ip from any to any dst-port 2222 fib 0 > 00011 0 0 count ip from any 2222 to any fib 0 > 00012 0 0 count ip from any to any dst-port 2222 fib 1 > 00013 0 0 count ip from any 2222 to any fib 1 > 00100 0 0 allow ip from any to any via lo0 > 00200 0 0 deny ip from any to 127.0.0.0/8 > 00300 0 0 deny ip from 127.0.0.0/8 to any > 65000 30 2648 allow ip from any to any > 65535 0 0 deny ip from any to any > > > With this setup almost everything works as I expect. For example ICMP > echo requests and responses are being received and sent via em1. Both > when ping runs on this host as "setfib 1 ping other_host" and when other > host pings ip address of em1. Connection attempts (setfib 1 telnet > other_host) are also being sent out of the right interface. But when it > comes to the daemons I run into troubles. > > I use sshd for tests (have also tried other daemons with no luck): > > r61net-fbsdhost-1, / # setfib 1 /usr/sbin/sshd -o 'ListenAddress 195.208.245.229:2222' -D Are you running this from inetd?. (doesnt look like it) btw is it 1 or -1? or -F 1? I can't remember if I supported just '1'. > > sshd is bound only to ip address of em1: > > r61net-fbsdhost-1, / # sockstat | grep 2222 > root sshd 839 3 tcp4 195.208.245.229:2222 *:* > > While doing telnet 195.208.249.229 2222 from another host I got following packet traces: > > r61net-fbsdhost-1, / # tcpdump -i em0 port 2222 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on em0, link-type EN10MB (Ethernet), capture size 96 bytes > 17:39:34.872475 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: S 2590499299:2590499299(0) ack 3939022576 win 65535 > 17:39:34.902622 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: P 1:41(40) ack 1 win 8326 > 17:39:37.572271 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: P 41:60(19) ack 7 win 8326 > 17:39:37.572293 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: F 60:60(0) ack 7 win 8326 > 17:39:37.572986 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: . ack 8 win 8325 > > r61net-fbsdhost-1, / # tcpdump -i em1 port 2222 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes > 17:39:34.872370 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: S 3939022575:3939022575(0) win 65535 > 17:39:34.872803 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 2590499300 win 8326 > 17:39:35.002882 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 41 win 8326 > 17:39:37.571659 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: P 0:6(6) ack 41 win 8326 > 17:39:37.572923 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 61 win 8323 > 17:39:37.572945 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: F 6:6(0) ack 61 win 8326 > > And firewall counters: > > r61net-fbsdhost-1, / # ipfw show > 00001 6 326 setfib 1 ip from any to any in recv em1 > 00010 0 0 count ip from any to any dst-port 2222 fib 0 > 00011 5 327 count ip from any 2222 to any fib 0 > 00012 6 326 count ip from any to any dst-port 2222 fib 1 > 00013 0 0 count ip from any 2222 to any fib 1 > 00100 0 0 allow ip from any to any via lo0 > 00200 0 0 deny ip from any to 127.0.0.0/8 > 00300 0 0 deny ip from 127.0.0.0/8 to any > 65000 60 5057 allow ip from any to any > 65535 0 0 deny ip from any to any > > So the packets, generated by sshd are being sent out via em0 instead of > em1. > > With > > ipfw add 2 setfib 1 ip from 195.208.245.229 to any > > outgoing packets are being tagged with correct fib, but still sent via > em0. yes becasue on outgoing packets the firewall is too late to influence that. > > With > > ipfw add 60003 fwd 195.208.245.225 src-ip me src-ip 195.208.245.224/27 not dst-ip 195.208.245.224/27 > > first SYN packet from 195.208.245.229 is being sent correctly via em1, > but I cannot see any further packets at all: > > r61net-fbsdhost-1, / # tcpdump -i em1 port 2222 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes > 18:01:56.665341 IP brain.cc.rsu.ru.50435 > stat.r61.net.2222: S 2484180116:2484180116(0) win 65535 > 18:01:56.665463 IP stat.r61.net.2222 > brain.cc.rsu.ru.50435: S 3905497961:3905497961(0) ack 2484180117 win 65535 > 18:01:56.665798 IP brain.cc.rsu.ru.50435 > stat.r61.net.2222: . ack 1 win 8326 > > and no packets at em0. TCP connection establishes but no data packets > come from daemon which is rather weird. > > I would appreciate any help with this issue. try adding a '-' on the command and get back to me. >