From owner-freebsd-net@FreeBSD.ORG Fri Nov 14 07:28:18 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4F8B4E25 for ; Fri, 14 Nov 2014 07:28:18 +0000 (UTC) Received: from mail.ismobile.com (mail.ismobile.com [176.57.193.164]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mail.ismobile.com", Issuer "GlobalSign Domain Validation CA - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AA919C4D for ; Fri, 14 Nov 2014 07:28:17 +0000 (UTC) Received: from mail.ismobile.com (localhost [127.0.0.1]) by dkim.mail.ismobile.com (Postfix) with ESMTP id C844C2B54A1 for ; Fri, 14 Nov 2014 07:28:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=ismobile.com; h=date:from :to:subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=selector1; bh=p+gUcs6 PN3jc1XHxUWtTZEg3h2I=; b=MPIzMi7axY66ZEQQ7Fdcot2WNmvsrpS3v8rQDk5 FfypQA/1h0ax7HZ6PvTq1MHaSeY9Uo6r6HirM7nKekjibDmLYySSLEIPy4oXilQ3 qpd9RWygNemlaAdOuo6l+Ij3JxQPL7ewGQddLwckMRg/5EjX1NqwzbRpQEFts8hQ YYP0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=ismobile.com; h=date:from:to :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=selector1; b=I hext5S3qxMeHTjpuwEMUOaDpXIIq3+uc3T9lVF7cqhd7LUSddGTy0QiDtcOxXJUc 94Ph3LwBNECj6yvsEeIaszk/zk4z9UqlgudLIPltXnwy3LJZ1Q6njc2NrSWPFxbK CLibJFKAJILdtjDM4eGjIjLECoXklSURuMV3SsZsfY= Received: from [172.16.2.31] (glz-macbookpro.hq.ismobile.com [172.16.2.31]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.ismobile.com (Postfix) with ESMTPSA id C05492B54A0 for ; Fri, 14 Nov 2014 07:28:06 +0000 (UTC) Date: Fri, 14 Nov 2014 08:28:06 +0100 From: =?UTF-8?Q?G=C3=B6ran_L=C3=B6wkrantz?= To: freebsd-net@freebsd.org Subject: Re: Unexpected traffic flow Message-ID: <69AF2F05DAD895C2D6810B2D@[172.16.2.31]> In-Reply-To: <15969A88725905DA6D022093@[172.16.2.31]> References: <15969A88725905DA6D022093@[172.16.2.31]> X-Mailer: Mulberry/4.1.0a3 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline; size=11116 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Nov 2014 07:28:18 -0000 Sorry for top posting but I found the problem. If I remove the bridge=20 interface everything returns to normal. Anyone hava any idea? Could there be bad RSTP messages tricking the=20 forwarding? Bug in if_bridge? Will do some more network snooping but as this was not a problem with the=20 previous FBSD version used here, 8.2, regression? /glz --On 13 Nov 2014 17:20:47 +0100 G=C3=B6ran L=C3=B6wkrantz=20 wrote: > > One of our gateways is behaving oddly after updating to 10.1-PRERELEASE > r274192 and I just can not understand what is happening. > > Internet > | > | > | > +------+-------+ >| FW | > +------+-------+ > | > DMZ > | > +------+-------+ >| GW | OpenVPN > +--+--+--+--+--+ > | | | | > 2 3 4 9 > > Net 2 - em2 - 192.168.2.0/24 - servers, server-net switches. > Net 3 - em1 - 192.168.3.0/24 - workstations, ws-net switches > Net 4 - em0 - 192.168.4.0/24 - WiFi access points + VLAN switch > Net 9 - em4 - 192.168.9.0/24 - monitor net w. a few switches. > DMZ - em5 - XXX.XXX.XXX.128/27 - DMZ and transfer net to outside. > > At the bottom of the mail I have included netstat -rn and ifconfig -au > output. > > > After the upgrade a few hosts got intermittent internet connection and a > few hosts lost all connectivity. > > Specifically, I have one host on the 2-net, 192.168.2.22, that refuses to > connect to internet or > lookup names via the gw, which is also the local DNS. > > 1: tcpdump on em0, em1, em2, em4 and em5 while sending ping to > 192.168.2.1 i.e. from the problem host to the gw. > - incoming traffic on em2: > 16:54:36.220053 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id > 9814, seq 1, length 64 > 16:54:37.219705 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id > 9814, seq 2, length 64 > 16:54:38.219732 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id > 9814, seq 3, length 64 > 16:54:39.219759 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id > 9814, seq 4, length 64 > 16:54:40.219783 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id > 9814, seq 5, length 64 > 16:54:41.230678 ARP, Request who-has 192.168.2.1 tell 192.168.2.22, > length 46 > 16:54:41.230682 ARP, Reply 192.168.2.1 is-at 00:1b:21:24:62:4a, length 28 > > - outgoing traffic on em5: > 16:54:36.220066 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, > seq 1, length 64 > ...... + 62 repeates ......... > 16:54:36.224436 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, > seq 1, length 64 > 16:54:37.219712 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, > seq 2, length 64 > ...... + 62 repeates ......... > 16:54:37.223711 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, > seq 2, length 64 > 16:54:38.219738 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, > seq 3, length 64 > ...... + 62 repeates ......... > 16:54:38.224984 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, > seq 3, length 64 > 16:54:39.219766 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, > seq 4, length 64 > ...... + 62 repeates ......... > 16:54:39.225757 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, > seq 4, length 64 > 16:54:40.219789 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, > seq 5, length 64 > ...... + 62 repeates ......... > 16:54:40.224282 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814, > seq 5, length 64 > > Checking on em2, I see > root@gw01:/usr/local/etc # arp 192.168.2.22 > dbserver-2.lulea.arcticgroup.se (192.168.2.22) at 00:19:99:21:87:62 on > em2 expires in 1116 seconds [ethernet] > > So we have route and arp at em2 but the ping replies are sent out via the > port connecting to the default gateway. > > If I try to ping 192.168.2.22 from the GW, I get this response and only > traffic going out via em5. > root@gw01:/usr/local/etc # ping 192.168.2.22 > PING 192.168.2.22 (192.168.2.22): 56 data bytes > 36 bytes from dmz.xxx.com (XXX.XXX.XXX.129): Redirect Host(New addr: > XXX.XXX.XXX.158) > Vr HL TOS Len ID Flg off TTL Pro cks Src Dst > 4 5 00 0054 39e4 0 0000 40 01 0c2f XXX.XXX.XXX.158 192.168.2.22 > > For some hosts, 192.168.6 for example, this happens intermittently like > 3-10 pings go correct, returning on em2, then 1, 2 or more are sent via > em5. > > None of the hosts are using RIP or other active routing, only static > routes. > GW host is built with nanobsd and attached kernel configuration. > > I have never seen anything like this. Can anyone help? Any more info > needed? > > Routing tables (only IPv4 as that's what's on the inside). > > Internet: > Destination Gateway Flags Netif Expire > default XXX.XXX.XXX.129 UGS em5 FW DMZ IF > 10.191.251.0/24 10.191.251.2 UGS tun0 OpenVPN > net 1 > 10.191.251.1 link#14 UHS lo0 OpenVPN > local endpoint 1 > 10.191.251.2 link#14 UH tun0 OpenVPN > remote endpoint 1 > 10.191.252.0/24 10.191.252.2 UGS tun1 OpenVPN > net 2 > 10.191.252.1 link#15 UHS lo0 OpenVPN > local endpoint 2 > 10.191.252.2 link#15 UH tun1 OpenVPN > remote endpoint 2 > 10.191.253.0/24 10.191.253.2 UGS tun2 OpenVPN > net 3 > 10.191.253.1 link#16 UHS lo0 OpenVPN > local endpoint 3 > 10.191.253.2 link#16 UH tun2 OpenVPN > remote endpoint 3 > 127.0.0.1 link#11 UH lo0 localhost > XXX.XXX.XXX.128/27 link#6 U em5 DMZ = network > XXX.XXX.XXX.157 link#13 UHS lo0 > XXX.XXX.XXX.157/32 link#13 U tap0 OpenVPN > Server > XXX.XXX.XXX.158 link#6 UHS lo0 GW extern > IP, em5 > 192.168.2.0/24 link#3 U em2 server net > 192.168.2.1 link#3 UHS lo0 GW em2 > 192.168.3.0/24 link#2 U em1 ws net > 192.168.3.1 link#2 UHS lo0 GW em1 > 192.168.4.0/24 link#1 U em0 wifi net > 192.168.4.254 link#1 UHS lo0 server em0 > 192.168.9.0/24 link#5 U em4 monitor = net > 192.168.9.254 link#5 UHS lo0 server em4 > > > em0: flags=3D8843 metric 0 mtu = 1500 > = options=3D209b= > ether 00:1b:21:24:62:48 > inet 192.168.4.254 netmask 0xffffff00 broadcast 192.168.4.255 > nd6 options=3D29 > media: Ethernet autoselect (1000baseT ) > status: active > em1: flags=3D8843 metric 0 mtu = 1500 > options=3D9b > ether 00:1b:21:24:62:49 > inet 192.168.3.1 netmask 0xffffff00 broadcast 192.168.3.255 > nd6 options=3D29 > media: Ethernet autoselect (1000baseT ) > status: active > em2: flags=3D8943 metric = 0 > mtu 1500 > options=3D9b > ether 00:1b:21:24:62:4a > inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255 > nd6 options=3D29 > media: Ethernet autoselect (1000baseT ) > status: active > em4: flags=3D8843 metric 0 mtu = 1500 > = options=3D4219b _MAGIC,VLAN_HWTSO> > ether 00:30:48:b9:99:c8 > inet 192.168.9.254 netmask 0xffffff00 broadcast 192.168.9.255 > nd6 options=3D29 > media: Ethernet autoselect (100baseTX ) > status: active > em5: flags=3D8943 metric = 0 > mtu 1500 > = options=3D42098 > ether 00:30:48:b9:99:c9 > inet XXX.XXX.XXX.158 netmask 0xffffffe0 broadcast XXX.XXX.XXX.159 > inet6 fe80::230:48ff:feb9:99c9%em5 prefixlen 64 scopeid 0x6 > inet6 xxxx:xxxx:101:1::fffe prefixlen 64 > nd6 options=3D21 > media: Ethernet autoselect (1000baseT ) > status: active > lo0: flags=3D8049 metric 0 mtu 16384 > options=3D600003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb > inet 127.0.0.1 netmask 0xff000000 > nd6 options=3D21 > bridge0: flags=3D8843 metric 0 = mtu > 1500 > ether 02:33:00:4d:00:00 > nd6 options=3D9 > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: em5 flags=3D143 > ifmaxaddr 0 port 6 priority 128 path cost 2000000 > member: tap0 flags=3D143 > ifmaxaddr 0 port 13 priority 128 path cost 2000000 > tap0: flags=3D8943 metric = 0 > mtu 1500 > options=3D80000 > ether 00:bd:50:63:00:00 > inet XXX.XXX.XXX.157 netmask 0xffffffff broadcast XXX.XXX.XXX.157 > inet6 fe80::2bd:50ff:fe63:0%tap0 prefixlen 64 scopeid 0xd > inet6 xxxx:xxxx:101:1::fffd prefixlen 64 > nd6 options=3D21 > media: Ethernet autoselect > status: no carrier > tun0: flags=3D8051 metric 0 mtu 1500 > options=3D80000 > inet6 fe80::21b:21ff:fe24:6248%tun0 prefixlen 64 scopeid 0xe > inet 10.191.251.1 --> 10.191.251.2 netmask 0xffffffff > nd6 options=3D21 > Opened by PID 1565 > tun1: flags=3D8051 metric 0 mtu 1500 > options=3D80000 > inet6 fe80::21b:21ff:fe24:6248%tun1 prefixlen 64 scopeid 0xf > inet 10.191.252.1 --> 10.191.252.2 netmask 0xffffffff > nd6 options=3D21 > Opened by PID 1580 > tun2: flags=3D8051 metric 0 mtu 1500 > options=3D80000 > inet6 fe80::21b:21ff:fe24:6248%tun2 prefixlen 64 scopeid 0x10 > inet 10.191.253.1 --> 10.191.253.2 netmask 0xffffffff > nd6 options=3D21 > Opened by PID 1595 > > root@gw01:/usr/local/etc # route get 192.168.2.22 > route to: dbserver-2.lulea.arcticgroup.se > destination: 192.168.2.0 > mask: 255.255.255.0 > fib: 0 > interface: em2 > flags: > recvpipe sendpipe ssthresh rtt,msec mtu weight expire > 0 0 0 0 1500 1 0 > > "There are no solved problems; there are only problems that are more or less solved" -- Henri Poincare