From owner-freebsd-net@FreeBSD.ORG Thu Nov 13 16:20:51 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 25D0ECDE for ; Thu, 13 Nov 2014 16:20:51 +0000 (UTC) Received: from mail.ismobile.com (mail.ismobile.com [IPv6:2a00:f680:101:11::4]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mail.ismobile.com", Issuer "GlobalSign Domain Validation CA - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 72FF9137 for ; Thu, 13 Nov 2014 16:20:50 +0000 (UTC) Received: from mail.ismobile.com (localhost [127.0.0.1]) by dkim.mail.ismobile.com (Postfix) with ESMTP id 8B0042B54A3 for ; Thu, 13 Nov 2014 16:20:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=ismobile.com; h=date:from :to:subject:message-id:mime-version:content-type; s=selector1; bh=CidSAR4tHMvWiJ/sbHWNdMqNJc8=; b=PRNezs3XS81Mm/5ZPPhY7SCzYG81 RnEFKZrUa5eCGnQQY5amCpUKcNkDSskgAyHFRoVMHqcZ9S9rztcC5Jwb96SDhvxQ zcqc00NycvoiXldPef2tbNaw3ZYttpydg60XZEfi3x3PKlkwZseLN1dA4oIV3bnl 6LU2LIxKczAecyM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=ismobile.com; h=date:from:to :subject:message-id:mime-version:content-type; q=dns; s= selector1; b=gY/KOZXExGr7Qzdnx5fCMdzAxpY1gZwq3qp1hAIt9/3oPK05e8h usQciYtb/zk7yHnFATjjngk+5S2YGFV429RUCDjGR1DTCb0kew8wGEBWXXtKJ7Rw PwLASenEt9xx+ErafCSvbL8vy+aMnrL2lnpgu0tEp6hf5CMaUuFbm5Ns= Received: from [172.16.2.31] (glz-macbookpro.hq.ismobile.com [172.16.2.31]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.ismobile.com (Postfix) with ESMTPSA id 825C52B54A0 for ; Thu, 13 Nov 2014 16:20:47 +0000 (UTC) Date: Thu, 13 Nov 2014 17:20:47 +0100 From: =?UTF-8?Q?G=C3=B6ran_L=C3=B6wkrantz?= To: freebsd-net@freebsd.org Subject: Unexpected traffic flow Message-ID: <15969A88725905DA6D022093@[172.16.2.31]> X-Mailer: Mulberry/4.1.0a3 (Mac OS X) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="==========D6490C6FE1812265AB51==========" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Nov 2014 16:20:51 -0000 --==========D6490C6FE1812265AB51========== Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline; size=10146 One of our gateways is behaving oddly after updating to 10.1-PRERELEASE r274192 and I just can not understand what is happening. Internet | | | +------+-------+ | FW | +------+-------+ | DMZ | +------+-------+ | GW | OpenVPN +--+--+--+--+--+ | | | | 2 3 4 9 Net 2 - em2 - 192.168.2.0/24 - servers, server-net switches. Net 3 - em1 - 192.168.3.0/24 - workstations, ws-net switches Net 4 - em0 - 192.168.4.0/24 - WiFi access points + VLAN switch Net 9 - em4 - 192.168.9.0/24 - monitor net w. a few switches. DMZ - em5 - XXX.XXX.XXX.128/27 - DMZ and transfer net to outside. At the bottom of the mail I have included netstat -rn and ifconfig -au output. After the upgrade a few hosts got intermittent internet connection and a few hosts lost all connectivity. Specifically, I have one host on the 2-net, 192.168.2.22, that refuses to connect to internet or lookup names via the gw, which is also the local DNS. 1: tcpdump on em0, em1, em2, em4 and em5 while sending ping to 192.168.2.1 i.e. from the problem host to the gw. - incoming traffic on em2: 16:54:36.220053 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id 9814, seq 1, length 64 16:54:37.219705 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id 9814, seq 2, length 64 16:54:38.219732 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id 9814, seq 3, length 64 16:54:39.219759 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id 9814, seq 4, length 64 16:54:40.219783 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id 9814, seq 5, length 64 16:54:41.230678 ARP, Request who-has 192.168.2.1 tell 192.168.2.22, length 46 16:54:41.230682 ARP, Reply 192.168.2.1 is-at 00:1b:21:24:62:4a, length 28 - outgoing traffic on em5: 16:54:36.220066 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, seq 1, length 64 ...... + 62 repeates ......... 16:54:36.224436 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, seq 1, length 64 16:54:37.219712 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, seq 2, length 64 ...... + 62 repeates ......... 16:54:37.223711 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, seq 2, length 64 16:54:38.219738 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, seq 3, length 64 ...... + 62 repeates ......... 16:54:38.224984 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, seq 3, length 64 16:54:39.219766 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, seq 4, length 64 ...... + 62 repeates ......... 16:54:39.225757 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, seq 4, length 64 16:54:40.219789 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, seq 5, length 64 ...... + 62 repeates ......... 16:54:40.224282 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, seq 5, length 64 Checking on em2, I see root@gw01:/usr/local/etc # arp 192.168.2.22 dbserver-2.lulea.arcticgroup.se (192.168.2.22) at 00:19:99:21:87:62 on em2 expires in 1116 seconds [ethernet] So we have route and arp at em2 but the ping replies are sent out via the port connecting to the default gateway. If I try to ping 192.168.2.22 from the GW, I get this response and only traffic going out via em5. root@gw01:/usr/local/etc # ping 192.168.2.22 PING 192.168.2.22 (192.168.2.22): 56 data bytes 36 bytes from dmz.xxx.com (XXX.XXX.XXX.129): Redirect Host(New addr: XXX.XXX.XXX.158) Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 39e4 0 0000 40 01 0c2f XXX.XXX.XXX.158 192.168.2.22 For some hosts, 192.168.6 for example, this happens intermittently like 3-10 pings go correct, returning on em2, then 1, 2 or more are sent via em5. None of the hosts are using RIP or other active routing, only static routes. GW host is built with nanobsd and attached kernel configuration. I have never seen anything like this. Can anyone help? Any more info needed? Routing tables (only IPv4 as that's what's on the inside). Internet: Destination Gateway Flags Netif Expire default XXX.XXX.XXX.129 UGS em5 FW DMZ IF 10.191.251.0/24 10.191.251.2 UGS tun0 OpenVPN net 1 10.191.251.1 link#14 UHS lo0 OpenVPN local endpoint 1 10.191.251.2 link#14 UH tun0 OpenVPN remote endpoint 1 10.191.252.0/24 10.191.252.2 UGS tun1 OpenVPN net 2 10.191.252.1 link#15 UHS lo0 OpenVPN local endpoint 2 10.191.252.2 link#15 UH tun1 OpenVPN remote endpoint 2 10.191.253.0/24 10.191.253.2 UGS tun2 OpenVPN net 3 10.191.253.1 link#16 UHS lo0 OpenVPN local endpoint 3 10.191.253.2 link#16 UH tun2 OpenVPN remote endpoint 3 127.0.0.1 link#11 UH lo0 localhost XXX.XXX.XXX.128/27 link#6 U em5 DMZ network XXX.XXX.XXX.157 link#13 UHS lo0 XXX.XXX.XXX.157/32 link#13 U tap0 OpenVPN Server XXX.XXX.XXX.158 link#6 UHS lo0 GW extern IP, em5 192.168.2.0/24 link#3 U em2 server net 192.168.2.1 link#3 UHS lo0 GW em2 192.168.3.0/24 link#2 U em1 ws net 192.168.3.1 link#2 UHS lo0 GW em1 192.168.4.0/24 link#1 U em0 wifi net 192.168.4.254 link#1 UHS lo0 server em0 192.168.9.0/24 link#5 U em4 monitor net 192.168.9.254 link#5 UHS lo0 server em4 em0: flags=8843 metric 0 mtu 1500 options=209b ether 00:1b:21:24:62:48 inet 192.168.4.254 netmask 0xffffff00 broadcast 192.168.4.255 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8843 metric 0 mtu 1500 options=9b ether 00:1b:21:24:62:49 inet 192.168.3.1 netmask 0xffffff00 broadcast 192.168.3.255 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em2: flags=8943 metric 0 mtu 1500 options=9b ether 00:1b:21:24:62:4a inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em4: flags=8843 metric 0 mtu 1500 options=4219b ether 00:30:48:b9:99:c8 inet 192.168.9.254 netmask 0xffffff00 broadcast 192.168.9.255 nd6 options=29 media: Ethernet autoselect (100baseTX ) status: active em5: flags=8943 metric 0 mtu 1500 options=42098 ether 00:30:48:b9:99:c9 inet XXX.XXX.XXX.158 netmask 0xffffffe0 broadcast XXX.XXX.XXX.159 inet6 fe80::230:48ff:feb9:99c9%em5 prefixlen 64 scopeid 0x6 inet6 xxxx:xxxx:101:1::fffe prefixlen 64 nd6 options=21 media: Ethernet autoselect (1000baseT ) status: active lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb inet 127.0.0.1 netmask 0xff000000 nd6 options=21 bridge0: flags=8843 metric 0 mtu 1500 ether 02:33:00:4d:00:00 nd6 options=9 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: em5 flags=143 ifmaxaddr 0 port 6 priority 128 path cost 2000000 member: tap0 flags=143 ifmaxaddr 0 port 13 priority 128 path cost 2000000 tap0: flags=8943 metric 0 mtu 1500 options=80000 ether 00:bd:50:63:00:00 inet XXX.XXX.XXX.157 netmask 0xffffffff broadcast XXX.XXX.XXX.157 inet6 fe80::2bd:50ff:fe63:0%tap0 prefixlen 64 scopeid 0xd inet6 xxxx:xxxx:101:1::fffd prefixlen 64 nd6 options=21 media: Ethernet autoselect status: no carrier tun0: flags=8051 metric 0 mtu 1500 options=80000 inet6 fe80::21b:21ff:fe24:6248%tun0 prefixlen 64 scopeid 0xe inet 10.191.251.1 --> 10.191.251.2 netmask 0xffffffff nd6 options=21 Opened by PID 1565 tun1: flags=8051 metric 0 mtu 1500 options=80000 inet6 fe80::21b:21ff:fe24:6248%tun1 prefixlen 64 scopeid 0xf inet 10.191.252.1 --> 10.191.252.2 netmask 0xffffffff nd6 options=21 Opened by PID 1580 tun2: flags=8051 metric 0 mtu 1500 options=80000 inet6 fe80::21b:21ff:fe24:6248%tun2 prefixlen 64 scopeid 0x10 inet 10.191.253.1 --> 10.191.253.2 netmask 0xffffffff nd6 options=21 Opened by PID 1595 root@gw01:/usr/local/etc # route get 192.168.2.22 route to: dbserver-2.lulea.arcticgroup.se destination: 192.168.2.0 mask: 255.255.255.0 fib: 0 interface: em2 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 --==========D6490C6FE1812265AB51========== Content-Type: application/octet-stream; name=ROUTER Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=ROUTER; size=2438 IwojIFJPVVRFUiAtLSBCYXNpYyByb3V0ZXIgc2V0dXAKIwoKaW5jbHVkZSBHRU5FUklDCgppZGVu dAkJUk9VVEVSCgojIFJlbW92ZSBrZXJuZWwgZGVidWdnZXIKbm9vcHRpb25zICAgICAgIEtEQgpu b29wdGlvbnMgICAgICAgS0RCX1RSQUNFCgojIEFkZCBudWxsZnMgdG8gYWxsb3cgbWFwcGluZyBv ZiBkYXRhCm9wdGlvbnMgCU5VTExGUwoKIyBSb3V0ZXIgY29uZmlnCm9wdGlvbnMgICBERVZJQ0Vf UE9MTElORwpvcHRpb25zICAgRkxPV1RBQkxFICAgICAgICAgICAgICAgIyBwZXItY3B1IHJvdXRp bmcgY2FjaGUKCiMgSVBTRUMgZXRjCm9wdGlvbnMgSVBTRUMKb3B0aW9ucyAgIElQU0VDX05BVF9U CmRldmljZSBjcnlwdG8KZGV2aWNlIGNyeXB0b2RldgpkZXZpY2UgaGlmbgoKIyBmaXJld2FsbCBh bmQgc3R1ZmYKZGV2aWNlIHBmCmRldmljZSBwZmxvZwpkZXZpY2UgcGZzeW5jCgpvcHRpb25zIEFM VFEKCm9wdGlvbnMgQUxUUV9DQlEKb3B0aW9ucyBBTFRRX1JFRApvcHRpb25zIEFMVFFfUklPCm9w dGlvbnMgQUxUUV9IRlNDCm9wdGlvbnMgQUxUUV9DRE5SCm9wdGlvbnMgQUxUUV9QUklRCm9wdGlv bnMgQUxUUV9OT1BDQwoKIyBSZW1vdmUgdW51c2VkIHN0dWZmCiMgU0NTSSBDb250cm9sbGVycwpu b2RldmljZSBhaGMgICAgICAgICAgICAgIyBBSEEyOTQwIGFuZCBvbmJvYXJkIEFJQzd4eHggZGV2 aWNlcwpub2RldmljZSBhaGQgICAgICAgICAgICAgIyBBSEEzOTMyMC8yOTMyMCBhbmQgb25ib2Fy ZCBBSUM3OXh4IGRldmljZXMKbm9kZXZpY2UgZXNwICAgICAgICAgICAgICMgQU1EIEFtNTNDOTc0 IChUZWtyYW0gREMtMzkwKFQpKQpub2RldmljZSBocHRpb3AgICAgICAgICAgIyBIaWdocG9pbnQg Um9ja2V0UmFpZCAzeHh4IHNlcmllcwpub2RldmljZSBpc3AgICAgICAgICAgICAgIyBRbG9naWMg ZmFtaWx5Cm5vZGV2aWNlIG1wdCAgICAgICAgICAgICAjIExTSS1Mb2dpYyBNUFQtRnVzaW9uCm5v ZGV2aWNlIG1wcyAgICAgICAgICAgICAjIExTSS1Mb2dpYyBNUFQtRnVzaW9uIDIKbm9kZXZpY2Ug c3ltICAgICAgICAgICAgICMgTkNSL1N5bWJpb3MgTG9naWMgKG5ld2VyIGNoaXBzZXRzICsgdGhv c2Ugb2YgYG5jcicpCm5vZGV2aWNlIHRybSAgICAgICAgICAgICAjIFRla3JhbSBEQzM5NVUvVVcv RiBEQzMxNVUgYWRhcHRlcnMKbm9kZXZpY2UgYWR2ICAgICAgICAgICAgICMgQWR2YW5zeXMgU0NT SSBhZGFwdGVycwpub2RldmljZSBhZHcgICAgICAgICAgICAgIyBBZHZhbnN5cyB3aWRlIFNDU0kg YWRhcHRlcnMKbm9kZXZpY2UgYWljICAgICAgICAgICAgICMgQWRhcHRlYyAxNVswMTJdeCBTQ1NJ IGFkYXB0ZXJzLCBBSUMtNlsyM102MC4Kbm9kZXZpY2UgYnQgICAgICAgICAgICAgICMgQnVzbG9n aWMvTXlsZXggTXVsdGlNYXN0ZXIgU0NTSSBhZGFwdGVycwpub2RldmljZSBpc2NpICAgICAgICAg ICAgIyBJbnRlbCBDNjAwIFNBUyBjb250cm9sbGVyCiMgUkFJRCBjb250cm9sbGVycyBpbnRlcmZh Y2VkIHRvIHRoZSBTQ1NJIHN1YnN5c3RlbQpub2RldmljZSBhbXIgICAgICAgICAgICAgIyBBTUkg TWVnYVJBSUQKbm9kZXZpY2UgYXJjbXNyICAgICAgICAgICMgQXJlY2EgU0FUQSBJSSBSQUlECm5v ZGV2aWNlIGNpc3MgICAgICAgICAgICAjIENvbXBhcSBTbWFydCBSQUlEIDUqCm5vZGV2aWNlIGRw dCAgICAgICAgICAgICAjIERQVCBTbWFydGNhY2hlIElJSSwgSVYgLSBTZWUgTk9URVMgZm9yIG9w dGlvbnMKbm9kZXZpY2UgaHB0bXYgICAgICAgICAgICMgSGlnaHBvaW50IFJvY2tldFJBSUQgMTgy eApub2RldmljZSBocHRyciAgICAgICAgICAgIyBIaWdocG9pbnQgUm9ja2V0UkFJRCAxN3h4LCAy Mnh4LCAyM3h4LCAyNXh4Cm5vZGV2aWNlIGlpciAgICAgICAgICAgICAjIEludGVsIEludGVncmF0 ZWQgUkFJRApub2RldmljZSBpcHMgICAgICAgICAgICAgIyBJQk0gKEFkYXB0ZWMpIFNlcnZlUkFJ RApub2RldmljZSBtbHkgICAgICAgICAgICAgIyBNeWxleCBBY2NlbGVSQUlEL2VYdHJlbWVSQUlE Cm5vZGV2aWNlIHR3YSAgICAgICAgICAgICAjIDN3YXJlIDkwMDAgc2VyaWVzIFBBVEEvU0FUQSBS QUlECiMgUkFJRCBjb250cm9sbGVycwpub2RldmljZSBhYWMgICAgICAgICAgICAgIyBBZGFwdGVj IEZTQSBSQUlECm5vZGV2aWNlIGFhY3AgICAgICAgICAgICAjIFNDU0kgcGFzc3Rocm91Z2ggZm9y IGFhYyAocmVxdWlyZXMgQ0FNKQpub2RldmljZSBpZGEgICAgICAgICAgICAgIyBDb21wYXEgU21h cnQgUkFJRApub2RldmljZSBtZmkgICAgICAgICAgICAgIyBMU0kgTWVnYVJBSUQgU0FTCm5vZGV2 aWNlIG1seCAgICAgICAgICAgICAjIE15bGV4IERBQzk2MCBmYW1pbHkKbm9kZXZpY2UgdHdlICAg ICAgICAgICAgICMgM3dhcmUgQVRBIFJBSUQKbm9kZXZpY2UgdHdzICAgICAgICAgICAgICMgTFNJ IDN3YXJlIDk3NTAgU0FUQStTQVMgNkdiL3MgUkFJRCBjb250cm9sbGVyCgo= --==========D6490C6FE1812265AB51==========--