From owner-freebsd-net@freebsd.org Thu Mar 3 10:38:30 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 79B08A93ECB for ; Thu, 3 Mar 2016 10:38:30 +0000 (UTC) (envelope-from pg@pakhom.spb.ru) Received: from mail.pakhom.spb.ru (mail.pakhom.spb.ru [5.19.176.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 410DB21D for ; Thu, 3 Mar 2016 10:38:30 +0000 (UTC) (envelope-from pg@pakhom.spb.ru) Received: from [78.25.122.220] (helo=[10.160.235.239]) by mail.pakhom.spb.ru with esmtpsa (TLSv1.2:DHE-RSA-AES128-SHA:128) (Exim 4.85 (FreeBSD)) (envelope-from ) id 1abQeD-00021Q-Fs for freebsd-net@FreeBSD.org; Thu, 03 Mar 2016 13:38:21 +0300 To: freebsd-net@FreeBSD.org From: Pakhom Golynga Subject: routing issue Message-ID: <56D81418.9020500@pakhom.spb.ru> Date: Thu, 3 Mar 2016 13:38:16 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 78.25.122.220 X-SA-Exim-Mail-From: pg@pakhom.spb.ru X-SA-Exim-Scanned: No (on mail.pakhom.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Mar 2016 10:38:30 -0000 Hello all! Please help me to investigate this issue. I have problem on 10.2-RELEASE-p12 with multiple network interfaces and PF (rules, NAT) # ifconfig <--cut--> em0: flags=8843 metric 0 mtu 1500 options=4209b ether 00:25:90:64:14:50 inet 172.27.27.254 netmask 0xffffff00 broadcast 172.27.27.255 inet 172.27.27.240 netmask 0xffffffff broadcast 172.27.27.240 inet 172.27.27.252 netmask 0xffffffff broadcast 172.27.27.252 inet 172.27.27.251 netmask 0xffffffff broadcast 172.27.27.251 inet 172.27.27.250 netmask 0xffffffff broadcast 172.27.27.250 inet 172.27.27.249 netmask 0xffffffff broadcast 172.27.27.249 inet 172.27.27.248 netmask 0xffffffff broadcast 172.27.27.248 inet 172.27.27.247 netmask 0xffffffff broadcast 172.27.27.247 inet 172.27.27.246 netmask 0xffffffff broadcast 172.27.27.246 inet 172.27.27.245 netmask 0xffffffff broadcast 172.27.27.245 inet 172.27.27.244 netmask 0xffffffff broadcast 172.27.27.244 inet 172.27.27.243 netmask 0xffffffff broadcast 172.27.27.243 inet 172.27.27.242 netmask 0xffffffff broadcast 172.27.27.242 inet 172.27.27.241 netmask 0xffffffff broadcast 172.27.27.241 inet 172.27.27.239 netmask 0xffffffff broadcast 172.27.27.239 inet 172.27.27.238 netmask 0xffffffff broadcast 172.27.27.238 inet 172.27.27.237 netmask 0xffffffff broadcast 172.27.27.237 inet 172.27.27.236 netmask 0xffffffff broadcast 172.27.27.236 inet 172.27.27.235 netmask 0xffffffff broadcast 172.27.27.235 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8943 metric 0 mtu 1500 options=4209b ether 00:25:90:64:14:51 inet 10.24.44.50 netmask 0xfffffc00 broadcast 10.24.47.255 media: Ethernet autoselect (100baseTX ) status: active <--cut--> em0 - local network. Aliases on em0 for Jail's em1 - connected to ISP This configuration successfully operate a long time but recently (for no apparent reason) suddenly became manifest bug with routing. Regardless from uptime (sometimes 12 hours, and maybe a couple of days) after a certain time the traffic running between jails begins to route via em1 (instead lo0 (net.link.ether.inet.useloopback=1)): # tcpdump -n -e -ttt -i em1 host 172.27.27.252 or host 172.27.27.247 or host 172.27.27.248 00:00:00.054989 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.24868 > 172.27.27.252.25: Flags [S], seq 2075254199, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161568673 ecr 0], length 0 00:00:00.318036 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 66: 172.27.27.247.80 > 172.27.27.248.60176: Flags [.], ack 772660872, win 1275, options [nop,nop,TS val 1737287913 ecr 161568991], length 0 00:00:00.036971 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 66: 172.27.27.247.80 > 172.27.27.248.33080: Flags [.], ack 1, win 1275, options [nop,nop,TS val 1701623130 ecr 161568238], length 0 00:00:00.050001 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 66: 172.27.27.247.80 > 172.27.27.248.40200: Flags [.], ack 1, win 1275, options [nop,nop,TS val 1477098721 ecr 161568288], length 0 00:00:02.794992 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.24868 > 172.27.27.252.25: Flags [S], seq 2075254199, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161571873 ecr 0], length 0 00:00:03.312026 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.29581 > 172.27.27.252.25: Flags [S], seq 1217253021, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161575185 ecr 0], length 0 00:00:02.887982 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.24868 > 172.27.27.252.25: Flags [S], seq 2075254199, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161578073 ecr 0], length 0 00:00:00.111989 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.29581 > 172.27.27.252.25: Flags [S], seq 1217253021, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161578185 ecr 0], length 0 00:00:03.471007 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 108: 172.27.27.247.80 > 172.27.27.248.26056: Flags [FP.], seq 1448:1490, ack 1, win 1275, options [nop,nop,TS val 2142270801 ecr 161575507], length 42 In the same time: # route -vn get -host RTA_DST: inet 172.27.27.247; RTA_IFP: link ; RTM_GET: Report Metrics: len 224, pid: 0, seq 1, errno 0, flags: locks: inits: sockaddrs: 172.27.27.247 link#0 route to: 172.27.27.247 destination: 172.27.27.247 fib: 0 interface: lo0 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 16384 1 0 locks: inits: sockaddrs: 172.27.27.247 link#4 lo0 127.0.0.1 RTA_DST: inet 172.27.27.250; RTA_IFP: link ; RTM_GET: Report Metrics: len 224, pid: 0, seq 1, errno 0, flags: locks: inits: sockaddrs: 172.27.27.250 link#0 route to: 172.27.27.250 destination: 172.27.27.250 fib: 0 interface: lo0 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 16384 1 0 locks: inits: sockaddrs: 172.27.27.250 link#4 lo0 127.0.0.1 RTA_DST: inet 172.27.27.252; RTA_IFP: link ; RTM_GET: Report Metrics: len 224, pid: 0, seq 1, errno 0, flags: locks: inits: sockaddrs: 172.27.27.252 link#0 route to: 172.27.27.252 destination: 172.27.27.252 fib: 0 interface: lo0 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 16384 1 0 locks: inits: sockaddrs: 172.27.27.252 link#4 lo0 127.0.0.1 # netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 10.24.44.1 UGS em1 10.24.44.0/22 link#5 U em1 10.24.44.50 link#5 UHS lo0 127.0.0.1 link#7 UH lo0 172.27.27.0/24 link#4 U em0 172.27.27.235 link#4 UHS lo0 172.27.27.235/32 link#4 U em0 172.27.27.236 link#4 UHS lo0 172.27.27.236/32 link#4 U em0 172.27.27.237 link#4 UHS lo0 172.27.27.237/32 link#4 U em0 172.27.27.238 link#4 UHS lo0 172.27.27.238/32 link#4 U em0 172.27.27.239 link#4 UHS lo0 172.27.27.239/32 link#4 U em0 172.27.27.240 link#4 UHS lo0 172.27.27.240/32 link#4 U em0 172.27.27.241 link#4 UHS lo0 172.27.27.241/32 link#4 U em0 172.27.27.242 link#4 UHS lo0 172.27.27.242/32 link#4 U em0 172.27.27.243 link#4 UHS lo0 172.27.27.243/32 link#4 U em0 172.27.27.244 link#4 UHS lo0 172.27.27.244/32 link#4 U em0 172.27.27.245 link#4 UHS lo0 172.27.27.245/32 link#4 U em0 172.27.27.246 link#4 UHS lo0 172.27.27.246/32 link#4 U em0 172.27.27.247 link#4 UHS lo0 172.27.27.247/32 link#4 U em0 172.27.27.248 link#4 UHS lo0 172.27.27.248/32 link#4 U em0 172.27.27.249 link#4 UHS lo0 172.27.27.249/32 link#4 U em0 172.27.27.250 link#4 UHS lo0 172.27.27.250/32 link#4 U em0 172.27.27.251 link#4 UHS lo0 172.27.27.251/32 link#4 U em0 172.27.27.252 link#4 UHS lo0 172.27.27.252/32 link#4 U em0 172.27.27.254 link#4 UHS lo0 172.27.28.0/24 link#8 U bridge0 172.27.28.254 link#8 UHS lo0 172.27.30.0/24 172.27.30.2 UGS tun0 172.27.30.1 link#12 UHS lo0 172.27.30.2 link#12 UH tun0 After the reboot everything works well for some time. Thanks!