From owner-freebsd-net@FreeBSD.ORG Mon Jul 27 14:34:50 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 652EB106564A for ; Mon, 27 Jul 2009 14:34:50 +0000 (UTC) (envelope-from os@sfedu.ru) Received: from mail.r61.net (mail.r61.net [195.208.245.249]) by mx1.freebsd.org (Postfix) with ESMTP id B11958FC1E for ; Mon, 27 Jul 2009 14:34:49 +0000 (UTC) (envelope-from os@sfedu.ru) Received: from [195.208.252.154] (brain.cc.rsu.ru [195.208.252.154]) (authenticated bits=0) by mail.r61.net (8.14.3/8.14.1) with ESMTP id n6REHIbM097053 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Mon, 27 Jul 2009 18:17:18 +0400 (MSD) (envelope-from os@sfedu.ru) X-Authentication-Warning: asterix.r61.net: Host brain.cc.rsu.ru [195.208.252.154] claimed to be [195.208.252.154] From: Oleg Sharoyko To: freebsd-net@freebsd.org Content-Type: text/plain Date: Mon, 27 Jul 2009 18:17:17 +0400 Message-Id: <1248704237.96833.127.camel@brain.cc.rsu.ru> Mime-Version: 1.0 X-Mailer: Evolution 2.24.5 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Subject: Wrong outgoing interface with multiple routing tables X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jul 2009 14:34:50 -0000 Hello! I'm having a trouble with multiple routing tables (FreeBSD 7.2 release). Either I'm missing something in my setup or packets for daemons started with setfib are being sent out via the wrong interface. What I'd like to implement: em0 - internal management network with ip address 10.2.5.2/24 and default route 10.2.5.1 em1 - public interface to be used in jail with ip address 195.208.245.229/27 and default route 195.208.245.225 Here are my routing tables: r61net-fbsdhost-1, / # setfib -0 netstat -rn Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 10.2.5.1 UGS 0 350 em0 10.2.5.0/24 link#1 UC 0 0 em0 10.2.5.1 00:1e:4a:b4:ea:c0 UHLW 2 0 em0 1182 127.0.0.1 127.0.0.1 UH 0 30 lo0 r61net-fbsdhost-1, / # setfib -1 netstat -rn Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 195.208.245.225 UGS 0 0 em1 195.208.245.224/27 link#2 UC 0 0 em1 195.208.245.225 link#2 UHLW 2 0 em1 Firewall: r61net-fbsdhost-1, / # ipfw show 00001 0 0 setfib 1 ip from any to any in recv em1 00010 0 0 count ip from any to any dst-port 2222 fib 0 00011 0 0 count ip from any 2222 to any fib 0 00012 0 0 count ip from any to any dst-port 2222 fib 1 00013 0 0 count ip from any 2222 to any fib 1 00100 0 0 allow ip from any to any via lo0 00200 0 0 deny ip from any to 127.0.0.0/8 00300 0 0 deny ip from 127.0.0.0/8 to any 65000 30 2648 allow ip from any to any 65535 0 0 deny ip from any to any With this setup almost everything works as I expect. For example ICMP echo requests and responses are being received and sent via em1. Both when ping runs on this host as "setfib 1 ping other_host" and when other host pings ip address of em1. Connection attempts (setfib 1 telnet other_host) are also being sent out of the right interface. But when it comes to the daemons I run into troubles. I use sshd for tests (have also tried other daemons with no luck): r61net-fbsdhost-1, / # setfib 1 /usr/sbin/sshd -o 'ListenAddress 195.208.245.229:2222' -D sshd is bound only to ip address of em1: r61net-fbsdhost-1, / # sockstat | grep 2222 root sshd 839 3 tcp4 195.208.245.229:2222 *:* While doing telnet 195.208.249.229 2222 from another host I got following packet traces: r61net-fbsdhost-1, / # tcpdump -i em0 port 2222 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on em0, link-type EN10MB (Ethernet), capture size 96 bytes 17:39:34.872475 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: S 2590499299:2590499299(0) ack 3939022576 win 65535 17:39:34.902622 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: P 1:41(40) ack 1 win 8326 17:39:37.572271 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: P 41:60(19) ack 7 win 8326 17:39:37.572293 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: F 60:60(0) ack 7 win 8326 17:39:37.572986 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: . ack 8 win 8325 r61net-fbsdhost-1, / # tcpdump -i em1 port 2222 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes 17:39:34.872370 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: S 3939022575:3939022575(0) win 65535 17:39:34.872803 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 2590499300 win 8326 17:39:35.002882 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 41 win 8326 17:39:37.571659 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: P 0:6(6) ack 41 win 8326 17:39:37.572923 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 61 win 8323 17:39:37.572945 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: F 6:6(0) ack 61 win 8326 And firewall counters: r61net-fbsdhost-1, / # ipfw show 00001 6 326 setfib 1 ip from any to any in recv em1 00010 0 0 count ip from any to any dst-port 2222 fib 0 00011 5 327 count ip from any 2222 to any fib 0 00012 6 326 count ip from any to any dst-port 2222 fib 1 00013 0 0 count ip from any 2222 to any fib 1 00100 0 0 allow ip from any to any via lo0 00200 0 0 deny ip from any to 127.0.0.0/8 00300 0 0 deny ip from 127.0.0.0/8 to any 65000 60 5057 allow ip from any to any 65535 0 0 deny ip from any to any So the packets, generated by sshd are being sent out via em0 instead of em1. With ipfw add 2 setfib 1 ip from 195.208.245.229 to any outgoing packets are being tagged with correct fib, but still sent via em0. With ipfw add 60003 fwd 195.208.245.225 src-ip me src-ip 195.208.245.224/27 not dst-ip 195.208.245.224/27 first SYN packet from 195.208.245.229 is being sent correctly via em1, but I cannot see any further packets at all: r61net-fbsdhost-1, / # tcpdump -i em1 port 2222 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes 18:01:56.665341 IP brain.cc.rsu.ru.50435 > stat.r61.net.2222: S 2484180116:2484180116(0) win 65535 18:01:56.665463 IP stat.r61.net.2222 > brain.cc.rsu.ru.50435: S 3905497961:3905497961(0) ack 2484180117 win 65535 18:01:56.665798 IP brain.cc.rsu.ru.50435 > stat.r61.net.2222: . ack 1 win 8326 and no packets at em0. TCP connection establishes but no data packets come from daemon which is rather weird. I would appreciate any help with this issue. -- Oleg Sharoyko. Software and Network Engineer Computer Center of Rostov State University.