Date: Thu, 13 Nov 2014 17:20:47 +0100 From: =?UTF-8?Q?G=C3=B6ran_L=C3=B6wkrantz?= <goran.lowkrantz@ismobile.com> To: freebsd-net@freebsd.org Subject: Unexpected traffic flow Message-ID: <15969A88725905DA6D022093@[172.16.2.31]>
next in thread | raw e-mail | index | archive | help
--==========D6490C6FE1812265AB51==========
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline; size=10146
One of our gateways is behaving oddly after updating to 10.1-PRERELEASE
r274192 and I just can not understand what is happening.
Internet
|
|
|
+------+-------+
| FW |
+------+-------+
|
DMZ
|
+------+-------+
| GW | OpenVPN
+--+--+--+--+--+
| | | |
2 3 4 9
Net 2 - em2 - 192.168.2.0/24 - servers, server-net switches.
Net 3 - em1 - 192.168.3.0/24 - workstations, ws-net switches
Net 4 - em0 - 192.168.4.0/24 - WiFi access points + VLAN switch
Net 9 - em4 - 192.168.9.0/24 - monitor net w. a few switches.
DMZ - em5 - XXX.XXX.XXX.128/27 - DMZ and transfer net to outside.
At the bottom of the mail I have included netstat -rn and ifconfig -au
output.
After the upgrade a few hosts got intermittent internet connection and a
few hosts lost all connectivity.
Specifically, I have one host on the 2-net, 192.168.2.22, that refuses to
connect to internet or
lookup names via the gw, which is also the local DNS.
1: tcpdump on em0, em1, em2, em4 and em5 while sending ping to 192.168.2.1
i.e. from the problem host to the gw.
- incoming traffic on em2:
16:54:36.220053 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id 9814,
seq 1, length 64
16:54:37.219705 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id 9814,
seq 2, length 64
16:54:38.219732 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id 9814,
seq 3, length 64
16:54:39.219759 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id 9814,
seq 4, length 64
16:54:40.219783 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id 9814,
seq 5, length 64
16:54:41.230678 ARP, Request who-has 192.168.2.1 tell 192.168.2.22, length
46
16:54:41.230682 ARP, Reply 192.168.2.1 is-at 00:1b:21:24:62:4a, length 28
- outgoing traffic on em5:
16:54:36.220066 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
seq 1, length 64
...... + 62 repeates .........
16:54:36.224436 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
seq 1, length 64
16:54:37.219712 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
seq 2, length 64
...... + 62 repeates .........
16:54:37.223711 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
seq 2, length 64
16:54:38.219738 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
seq 3, length 64
...... + 62 repeates .........
16:54:38.224984 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
seq 3, length 64
16:54:39.219766 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
seq 4, length 64
...... + 62 repeates .........
16:54:39.225757 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
seq 4, length 64
16:54:40.219789 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
seq 5, length 64
...... + 62 repeates .........
16:54:40.224282 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
seq 5, length 64
Checking on em2, I see
root@gw01:/usr/local/etc # arp 192.168.2.22
dbserver-2.lulea.arcticgroup.se (192.168.2.22) at 00:19:99:21:87:62 on em2
expires in 1116 seconds [ethernet]
So we have route and arp at em2 but the ping replies are sent out via the
port connecting to the default gateway.
If I try to ping 192.168.2.22 from the GW, I get this response and only
traffic going out via em5.
root@gw01:/usr/local/etc # ping 192.168.2.22
PING 192.168.2.22 (192.168.2.22): 56 data bytes
36 bytes from dmz.xxx.com (XXX.XXX.XXX.129): Redirect Host(New addr:
XXX.XXX.XXX.158)
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 0054 39e4 0 0000 40 01 0c2f XXX.XXX.XXX.158 192.168.2.22
For some hosts, 192.168.6 for example, this happens intermittently like
3-10 pings go correct, returning on em2, then 1, 2 or more are sent via em5.
None of the hosts are using RIP or other active routing, only static routes.
GW host is built with nanobsd and attached kernel configuration.
I have never seen anything like this. Can anyone help? Any more info needed?
Routing tables (only IPv4 as that's what's on the inside).
Internet:
Destination Gateway Flags Netif Expire
default XXX.XXX.XXX.129 UGS em5 FW DMZ IF
10.191.251.0/24 10.191.251.2 UGS tun0 OpenVPN net 1
10.191.251.1 link#14 UHS lo0 OpenVPN
local endpoint 1
10.191.251.2 link#14 UH tun0 OpenVPN
remote endpoint 1
10.191.252.0/24 10.191.252.2 UGS tun1 OpenVPN net 2
10.191.252.1 link#15 UHS lo0 OpenVPN
local endpoint 2
10.191.252.2 link#15 UH tun1 OpenVPN
remote endpoint 2
10.191.253.0/24 10.191.253.2 UGS tun2 OpenVPN net 3
10.191.253.1 link#16 UHS lo0 OpenVPN
local endpoint 3
10.191.253.2 link#16 UH tun2 OpenVPN
remote endpoint 3
127.0.0.1 link#11 UH lo0 localhost
XXX.XXX.XXX.128/27 link#6 U em5 DMZ network
XXX.XXX.XXX.157 link#13 UHS lo0
XXX.XXX.XXX.157/32 link#13 U tap0 OpenVPN
Server
XXX.XXX.XXX.158 link#6 UHS lo0 GW extern
IP, em5
192.168.2.0/24 link#3 U em2 server net
192.168.2.1 link#3 UHS lo0 GW em2
192.168.3.0/24 link#2 U em1 ws net
192.168.3.1 link#2 UHS lo0 GW em1
192.168.4.0/24 link#1 U em0 wifi net
192.168.4.254 link#1 UHS lo0 server em0
192.168.9.0/24 link#5 U em4 monitor net
192.168.9.254 link#5 UHS lo0 server em4
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 00:1b:21:24:62:48
inet 192.168.4.254 netmask 0xffffff00 broadcast 192.168.4.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:1b:21:24:62:49
inet 192.168.3.1 netmask 0xffffff00 broadcast 192.168.3.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0
mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:1b:21:24:62:4a
inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
ether 00:30:48:b9:99:c8
inet 192.168.9.254 netmask 0xffffff00 broadcast 192.168.9.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
em5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0
mtu 1500
options=42098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO>
ether 00:30:48:b9:99:c9
inet XXX.XXX.XXX.158 netmask 0xffffffe0 broadcast XXX.XXX.XXX.159
inet6 fe80::230:48ff:feb9:99c9%em5 prefixlen 64 scopeid 0x6
inet6 xxxx:xxxx:101:1::fffe prefixlen 64
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
ether 02:33:00:4d:00:00
nd6 options=9<PERFORMNUD,IFDISABLED>
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: em5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000000
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 13 priority 128 path cost 2000000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0
mtu 1500
options=80000<LINKSTATE>
ether 00:bd:50:63:00:00
inet XXX.XXX.XXX.157 netmask 0xffffffff broadcast XXX.XXX.XXX.157
inet6 fe80::2bd:50ff:fe63:0%tap0 prefixlen 64 scopeid 0xd
inet6 xxxx:xxxx:101:1::fffd prefixlen 64
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
inet6 fe80::21b:21ff:fe24:6248%tun0 prefixlen 64 scopeid 0xe
inet 10.191.251.1 --> 10.191.251.2 netmask 0xffffffff
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Opened by PID 1565
tun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
inet6 fe80::21b:21ff:fe24:6248%tun1 prefixlen 64 scopeid 0xf
inet 10.191.252.1 --> 10.191.252.2 netmask 0xffffffff
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Opened by PID 1580
tun2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
inet6 fe80::21b:21ff:fe24:6248%tun2 prefixlen 64 scopeid 0x10
inet 10.191.253.1 --> 10.191.253.2 netmask 0xffffffff
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Opened by PID 1595
root@gw01:/usr/local/etc # route get 192.168.2.22
route to: dbserver-2.lulea.arcticgroup.se
destination: 192.168.2.0
mask: 255.255.255.0
fib: 0
interface: em2
flags: <UP,DONE,PINNED>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0
--==========D6490C6FE1812265AB51==========
Content-Type: application/octet-stream; name=ROUTER
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename=ROUTER; size=2438
IwojIFJPVVRFUiAtLSBCYXNpYyByb3V0ZXIgc2V0dXAKIwoKaW5jbHVkZSBHRU5FUklDCgppZGVu
dAkJUk9VVEVSCgojIFJlbW92ZSBrZXJuZWwgZGVidWdnZXIKbm9vcHRpb25zICAgICAgIEtEQgpu
b29wdGlvbnMgICAgICAgS0RCX1RSQUNFCgojIEFkZCBudWxsZnMgdG8gYWxsb3cgbWFwcGluZyBv
ZiBkYXRhCm9wdGlvbnMgCU5VTExGUwoKIyBSb3V0ZXIgY29uZmlnCm9wdGlvbnMgICBERVZJQ0Vf
UE9MTElORwpvcHRpb25zICAgRkxPV1RBQkxFICAgICAgICAgICAgICAgIyBwZXItY3B1IHJvdXRp
bmcgY2FjaGUKCiMgSVBTRUMgZXRjCm9wdGlvbnMgSVBTRUMKb3B0aW9ucyAgIElQU0VDX05BVF9U
CmRldmljZSBjcnlwdG8KZGV2aWNlIGNyeXB0b2RldgpkZXZpY2UgaGlmbgoKIyBmaXJld2FsbCBh
bmQgc3R1ZmYKZGV2aWNlIHBmCmRldmljZSBwZmxvZwpkZXZpY2UgcGZzeW5jCgpvcHRpb25zIEFM
VFEKCm9wdGlvbnMgQUxUUV9DQlEKb3B0aW9ucyBBTFRRX1JFRApvcHRpb25zIEFMVFFfUklPCm9w
dGlvbnMgQUxUUV9IRlNDCm9wdGlvbnMgQUxUUV9DRE5SCm9wdGlvbnMgQUxUUV9QUklRCm9wdGlv
bnMgQUxUUV9OT1BDQwoKIyBSZW1vdmUgdW51c2VkIHN0dWZmCiMgU0NTSSBDb250cm9sbGVycwpu
b2RldmljZSBhaGMgICAgICAgICAgICAgIyBBSEEyOTQwIGFuZCBvbmJvYXJkIEFJQzd4eHggZGV2
aWNlcwpub2RldmljZSBhaGQgICAgICAgICAgICAgIyBBSEEzOTMyMC8yOTMyMCBhbmQgb25ib2Fy
ZCBBSUM3OXh4IGRldmljZXMKbm9kZXZpY2UgZXNwICAgICAgICAgICAgICMgQU1EIEFtNTNDOTc0
IChUZWtyYW0gREMtMzkwKFQpKQpub2RldmljZSBocHRpb3AgICAgICAgICAgIyBIaWdocG9pbnQg
Um9ja2V0UmFpZCAzeHh4IHNlcmllcwpub2RldmljZSBpc3AgICAgICAgICAgICAgIyBRbG9naWMg
ZmFtaWx5Cm5vZGV2aWNlIG1wdCAgICAgICAgICAgICAjIExTSS1Mb2dpYyBNUFQtRnVzaW9uCm5v
ZGV2aWNlIG1wcyAgICAgICAgICAgICAjIExTSS1Mb2dpYyBNUFQtRnVzaW9uIDIKbm9kZXZpY2Ug
c3ltICAgICAgICAgICAgICMgTkNSL1N5bWJpb3MgTG9naWMgKG5ld2VyIGNoaXBzZXRzICsgdGhv
c2Ugb2YgYG5jcicpCm5vZGV2aWNlIHRybSAgICAgICAgICAgICAjIFRla3JhbSBEQzM5NVUvVVcv
RiBEQzMxNVUgYWRhcHRlcnMKbm9kZXZpY2UgYWR2ICAgICAgICAgICAgICMgQWR2YW5zeXMgU0NT
SSBhZGFwdGVycwpub2RldmljZSBhZHcgICAgICAgICAgICAgIyBBZHZhbnN5cyB3aWRlIFNDU0kg
YWRhcHRlcnMKbm9kZXZpY2UgYWljICAgICAgICAgICAgICMgQWRhcHRlYyAxNVswMTJdeCBTQ1NJ
IGFkYXB0ZXJzLCBBSUMtNlsyM102MC4Kbm9kZXZpY2UgYnQgICAgICAgICAgICAgICMgQnVzbG9n
aWMvTXlsZXggTXVsdGlNYXN0ZXIgU0NTSSBhZGFwdGVycwpub2RldmljZSBpc2NpICAgICAgICAg
ICAgIyBJbnRlbCBDNjAwIFNBUyBjb250cm9sbGVyCiMgUkFJRCBjb250cm9sbGVycyBpbnRlcmZh
Y2VkIHRvIHRoZSBTQ1NJIHN1YnN5c3RlbQpub2RldmljZSBhbXIgICAgICAgICAgICAgIyBBTUkg
TWVnYVJBSUQKbm9kZXZpY2UgYXJjbXNyICAgICAgICAgICMgQXJlY2EgU0FUQSBJSSBSQUlECm5v
ZGV2aWNlIGNpc3MgICAgICAgICAgICAjIENvbXBhcSBTbWFydCBSQUlEIDUqCm5vZGV2aWNlIGRw
dCAgICAgICAgICAgICAjIERQVCBTbWFydGNhY2hlIElJSSwgSVYgLSBTZWUgTk9URVMgZm9yIG9w
dGlvbnMKbm9kZXZpY2UgaHB0bXYgICAgICAgICAgICMgSGlnaHBvaW50IFJvY2tldFJBSUQgMTgy
eApub2RldmljZSBocHRyciAgICAgICAgICAgIyBIaWdocG9pbnQgUm9ja2V0UkFJRCAxN3h4LCAy
Mnh4LCAyM3h4LCAyNXh4Cm5vZGV2aWNlIGlpciAgICAgICAgICAgICAjIEludGVsIEludGVncmF0
ZWQgUkFJRApub2RldmljZSBpcHMgICAgICAgICAgICAgIyBJQk0gKEFkYXB0ZWMpIFNlcnZlUkFJ
RApub2RldmljZSBtbHkgICAgICAgICAgICAgIyBNeWxleCBBY2NlbGVSQUlEL2VYdHJlbWVSQUlE
Cm5vZGV2aWNlIHR3YSAgICAgICAgICAgICAjIDN3YXJlIDkwMDAgc2VyaWVzIFBBVEEvU0FUQSBS
QUlECiMgUkFJRCBjb250cm9sbGVycwpub2RldmljZSBhYWMgICAgICAgICAgICAgIyBBZGFwdGVj
IEZTQSBSQUlECm5vZGV2aWNlIGFhY3AgICAgICAgICAgICAjIFNDU0kgcGFzc3Rocm91Z2ggZm9y
IGFhYyAocmVxdWlyZXMgQ0FNKQpub2RldmljZSBpZGEgICAgICAgICAgICAgIyBDb21wYXEgU21h
cnQgUkFJRApub2RldmljZSBtZmkgICAgICAgICAgICAgIyBMU0kgTWVnYVJBSUQgU0FTCm5vZGV2
aWNlIG1seCAgICAgICAgICAgICAjIE15bGV4IERBQzk2MCBmYW1pbHkKbm9kZXZpY2UgdHdlICAg
ICAgICAgICAgICMgM3dhcmUgQVRBIFJBSUQKbm9kZXZpY2UgdHdzICAgICAgICAgICAgICMgTFNJ
IDN3YXJlIDk3NTAgU0FUQStTQVMgNkdiL3MgUkFJRCBjb250cm9sbGVyCgo=
--==========D6490C6FE1812265AB51==========--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?15969A88725905DA6D022093>