From owner-freebsd-net@freebsd.org Fri Mar 4 19:17:04 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 50C759DAACF for ; Fri, 4 Mar 2016 19:17:04 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 38A9CFAC for ; Fri, 4 Mar 2016 19:17:03 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from julian-mbp3.pixel8networks.com (50-196-156-133-static.hfc.comcastbusiness.net [50.196.156.133]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id u24JGtkn058161 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Fri, 4 Mar 2016 11:16:56 -0800 (PST) (envelope-from julian@freebsd.org) Subject: Re: routing issue To: freebsd-net@freebsd.org, Pakhom Golynga References: <56D81418.9020500@pakhom.spb.ru> From: Julian Elischer Message-ID: <56D9DF22.2080504@freebsd.org> Date: Fri, 4 Mar 2016 11:16:50 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <56D81418.9020500@pakhom.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Mar 2016 19:17:04 -0000 On 3/03/2016 2:38 AM, Pakhom Golynga wrote: > Hello all! > > Please help me to investigate this issue. > I have problem on 10.2-RELEASE-p12 with multiple network interfaces > and PF (rules, NAT) > # ifconfig > <--cut--> > em0: flags=8843 metric 0 mtu > 1500 > options=4209b > > ether 00:25:90:64:14:50 > inet 172.27.27.254 netmask 0xffffff00 broadcast 172.27.27.255 > inet 172.27.27.240 netmask 0xffffffff broadcast 172.27.27.240 > inet 172.27.27.252 netmask 0xffffffff broadcast 172.27.27.252 > inet 172.27.27.251 netmask 0xffffffff broadcast 172.27.27.251 > inet 172.27.27.250 netmask 0xffffffff broadcast 172.27.27.250 > inet 172.27.27.249 netmask 0xffffffff broadcast 172.27.27.249 > inet 172.27.27.248 netmask 0xffffffff broadcast 172.27.27.248 > inet 172.27.27.247 netmask 0xffffffff broadcast 172.27.27.247 > inet 172.27.27.246 netmask 0xffffffff broadcast 172.27.27.246 > inet 172.27.27.245 netmask 0xffffffff broadcast 172.27.27.245 > inet 172.27.27.244 netmask 0xffffffff broadcast 172.27.27.244 > inet 172.27.27.243 netmask 0xffffffff broadcast 172.27.27.243 > inet 172.27.27.242 netmask 0xffffffff broadcast 172.27.27.242 > inet 172.27.27.241 netmask 0xffffffff broadcast 172.27.27.241 > inet 172.27.27.239 netmask 0xffffffff broadcast 172.27.27.239 > inet 172.27.27.238 netmask 0xffffffff broadcast 172.27.27.238 > inet 172.27.27.237 netmask 0xffffffff broadcast 172.27.27.237 > inet 172.27.27.236 netmask 0xffffffff broadcast 172.27.27.236 > inet 172.27.27.235 netmask 0xffffffff broadcast 172.27.27.235 > media: Ethernet autoselect (1000baseT ) > status: active > em1: flags=8943 > metric 0 mtu 1500 > options=4209b > > ether 00:25:90:64:14:51 > inet 10.24.44.50 netmask 0xfffffc00 broadcast 10.24.47.255 > media: Ethernet autoselect (100baseTX ) > status: active > <--cut--> > > em0 - local network. Aliases on em0 for Jail's > em1 - connected to ISP > > This configuration successfully operate a long time but recently > (for no apparent reason) suddenly became manifest bug with routing. > Regardless from uptime (sometimes 12 hours, and maybe a couple of > days) after a certain time the traffic running between jails begins > to route via em1 (instead lo0 (net.link.ether.inet.useloopback=1)): this sounds much like another report we have seen. you may be able to fix it by re-adding your routes.. The report I have seen suggested that some routes seem to stop being effective after some period. It's still not understood. > # tcpdump -n -e -ttt -i em1 host 172.27.27.252 or host 172.27.27.247 > or host 172.27.27.248 > 00:00:00.054989 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype > IPv4 (0x0800), length 74: 172.27.27.247.24868 > 172.27.27.252.25: > Flags [S], seq 2075254199, win 65535, options [mss 16344,nop,wscale > 6,sackOK,TS val 161568673 ecr 0], length 0 > 00:00:00.318036 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype > IPv4 (0x0800), length 66: 172.27.27.247.80 > 172.27.27.248.60176: > Flags [.], ack 772660872, win 1275, options [nop,nop,TS val > 1737287913 ecr 161568991], length 0 > 00:00:00.036971 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype > IPv4 (0x0800), length 66: 172.27.27.247.80 > 172.27.27.248.33080: > Flags [.], ack 1, win 1275, options [nop,nop,TS val 1701623130 ecr > 161568238], length 0 > 00:00:00.050001 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype > IPv4 (0x0800), length 66: 172.27.27.247.80 > 172.27.27.248.40200: > Flags [.], ack 1, win 1275, options [nop,nop,TS val 1477098721 ecr > 161568288], length 0 > 00:00:02.794992 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype > IPv4 (0x0800), length 74: 172.27.27.247.24868 > 172.27.27.252.25: > Flags [S], seq 2075254199, win 65535, options [mss 16344,nop,wscale > 6,sackOK,TS val 161571873 ecr 0], length 0 > 00:00:03.312026 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype > IPv4 (0x0800), length 74: 172.27.27.247.29581 > 172.27.27.252.25: > Flags [S], seq 1217253021, win 65535, options [mss 16344,nop,wscale > 6,sackOK,TS val 161575185 ecr 0], length 0 > 00:00:02.887982 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype > IPv4 (0x0800), length 74: 172.27.27.247.24868 > 172.27.27.252.25: > Flags [S], seq 2075254199, win 65535, options [mss 16344,nop,wscale > 6,sackOK,TS val 161578073 ecr 0], length 0 > 00:00:00.111989 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype > IPv4 (0x0800), length 74: 172.27.27.247.29581 > 172.27.27.252.25: > Flags [S], seq 1217253021, win 65535, options [mss 16344,nop,wscale > 6,sackOK,TS val 161578185 ecr 0], length 0 > 00:00:03.471007 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype > IPv4 (0x0800), length 108: 172.27.27.247.80 > 172.27.27.248.26056: > Flags [FP.], seq 1448:1490, ack 1, win 1275, options [nop,nop,TS val > 2142270801 ecr 161575507], length 42 > > In the same time: > # route -vn get -host > RTA_DST: inet 172.27.27.247; RTA_IFP: link ; RTM_GET: Report > Metrics: len 224, pid: 0, seq 1, errno 0, > flags: > locks: inits: > sockaddrs: > 172.27.27.247 link#0 > route to: 172.27.27.247 > destination: 172.27.27.247 > fib: 0 > interface: lo0 > flags: > recvpipe sendpipe ssthresh rtt,msec mtu weight expire > 0 0 0 0 16384 1 0 > > locks: inits: > sockaddrs: > 172.27.27.247 link#4 lo0 127.0.0.1 > RTA_DST: inet 172.27.27.250; RTA_IFP: link ; RTM_GET: Report > Metrics: len 224, pid: 0, seq 1, errno 0, > flags: > locks: inits: > sockaddrs: > 172.27.27.250 link#0 > route to: 172.27.27.250 > destination: 172.27.27.250 > fib: 0 > interface: lo0 > flags: > recvpipe sendpipe ssthresh rtt,msec mtu weight expire > 0 0 0 0 16384 1 0 > > locks: inits: > sockaddrs: > 172.27.27.250 link#4 lo0 127.0.0.1 > RTA_DST: inet 172.27.27.252; RTA_IFP: link ; RTM_GET: Report > Metrics: len 224, pid: 0, seq 1, errno 0, > flags: > locks: inits: > sockaddrs: > 172.27.27.252 link#0 > route to: 172.27.27.252 > destination: 172.27.27.252 > fib: 0 > interface: lo0 > flags: > recvpipe sendpipe ssthresh rtt,msec mtu weight expire > 0 0 0 0 16384 1 0 > > locks: inits: > sockaddrs: > 172.27.27.252 link#4 lo0 127.0.0.1 > > > # netstat -rn > Routing tables > > Internet: > Destination Gateway Flags Netif Expire > default 10.24.44.1 UGS em1 > 10.24.44.0/22 link#5 U em1 > 10.24.44.50 link#5 UHS lo0 > 127.0.0.1 link#7 UH lo0 > 172.27.27.0/24 link#4 U em0 > 172.27.27.235 link#4 UHS lo0 > 172.27.27.235/32 link#4 U em0 > 172.27.27.236 link#4 UHS lo0 > 172.27.27.236/32 link#4 U em0 > 172.27.27.237 link#4 UHS lo0 > 172.27.27.237/32 link#4 U em0 > 172.27.27.238 link#4 UHS lo0 > 172.27.27.238/32 link#4 U em0 > 172.27.27.239 link#4 UHS lo0 > 172.27.27.239/32 link#4 U em0 > 172.27.27.240 link#4 UHS lo0 > 172.27.27.240/32 link#4 U em0 > 172.27.27.241 link#4 UHS lo0 > 172.27.27.241/32 link#4 U em0 > 172.27.27.242 link#4 UHS lo0 > 172.27.27.242/32 link#4 U em0 > 172.27.27.243 link#4 UHS lo0 > 172.27.27.243/32 link#4 U em0 > 172.27.27.244 link#4 UHS lo0 > 172.27.27.244/32 link#4 U em0 > 172.27.27.245 link#4 UHS lo0 > 172.27.27.245/32 link#4 U em0 > 172.27.27.246 link#4 UHS lo0 > 172.27.27.246/32 link#4 U em0 > 172.27.27.247 link#4 UHS lo0 > 172.27.27.247/32 link#4 U em0 > 172.27.27.248 link#4 UHS lo0 > 172.27.27.248/32 link#4 U em0 > 172.27.27.249 link#4 UHS lo0 > 172.27.27.249/32 link#4 U em0 > 172.27.27.250 link#4 UHS lo0 > 172.27.27.250/32 link#4 U em0 > 172.27.27.251 link#4 UHS lo0 > 172.27.27.251/32 link#4 U em0 > 172.27.27.252 link#4 UHS lo0 > 172.27.27.252/32 link#4 U em0 > 172.27.27.254 link#4 UHS lo0 > 172.27.28.0/24 link#8 U bridge0 > 172.27.28.254 link#8 UHS lo0 > 172.27.30.0/24 172.27.30.2 UGS tun0 > 172.27.30.1 link#12 UHS lo0 > 172.27.30.2 link#12 UH tun0 > > After the reboot everything works well for some time. > > Thanks! > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >