Date: Wed, 16 Mar 2011 14:28:34 GMT From: Steve Polyack <spolyack@collaborativefusion.com> To: freebsd-gnats-submit@FreeBSD.org Subject: amd64/155604: Flowtable excessively caches dest MAC addresses for outgoing UDP traffic Message-ID: <201103161428.p2GESYfl090159@red.freebsd.org> Resent-Message-ID: <201103161430.p2GEUKH9094918@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 155604
>Category: amd64
>Synopsis: Flowtable excessively caches dest MAC addresses for outgoing UDP traffic
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: freebsd-amd64
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Wed Mar 16 14:30:19 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator: Steve Polyack
>Release: 8.1-RELEASE
>Organization:
Collaborative Fusion, Inc.
>Environment:
FreeBSD web00 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 #1: Wed Dec 8 11:56:34 UTC 2010 root@web00:/usr/obj/usr/src/sys/WEB-1850-AMD64 amd64
>Description:
The flowtable facility added in 8.x and enabled by default can map new outgoing UDP packets to old gateway MAC addresses even after the system ARP tables have been updated. This is particularly troubling for DNS requests on a busy server if the DNS requests must pass through a router before reaching the DNS server.
The problem goes away after the flow expiration times, provided that you can prevent new UDP traffic that would match existing flows and prevent them from being expired.
>How-To-Repeat:
Setup a FreeBSD system to use a DNS server behind the default gateway or another router. Now, replace the default gateway/router with something else (or simply change the MAC address on the gateway/router). Attempt to make some DNS requests; tcpdump and you will see that the UDP DNS requests are still headed to the old MAC address of the router:
Current ARP table after the router replacement, the default gateway is 10.0.1.254, which it has the correct new MAC address for already, and the DNS server we are trying to reach is 10.0.2.80:
[spolyack@web01 ~]$ arp -an
? (10.0.1.17) at 00:0c:29:47:74:3a on em2 permanent [ethernet]
? (10.0.1.130) at 00:0c:29:47:74:26 on em0 permanent [ethernet]
? (10.0.1.254) at 00:a0:c9:00:01:01 on em0 expires in 915 seconds [ethernet]
? (10.0.0.17) at 00:0c:29:47:74:30 on em1 permanent [ethernet]
? (10.0.0.15) at 00:0c:29:47:74:30 on em1 permanent [ethernet]
? (10.0.0.11) at 00:0c:29:47:74:30 on em1 permanent [ethernet]
? (10.0.0.2) at 00:1f:a0:10:28:70 on em1 expires in 1162 seconds [ethernet]
? (10.0.0.1) at 00:1f:a0:10:28:70 on em1 expires in 943 seconds [ethernet]
tcpdump shows the DNS requests heading to the *old* router's MAC address despite the contents of the ARP table:
[spolyack@web01 ~]$ sudo tcpdump -i em0 -s 256 -vvv -e -n -ttt 'port 53'
tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 256 bytes
00:00:00.000000 00:0c:29:47:74:26 > 54:75:d0:a3:7c:8c, ethertype IPv4 (0x0800), length 86: (tos 0x0, ttl 64, id 55590, offset 0, flags [none], proto UDP (17), length 72)
10.0.1.130.52419 > 10.0.2.80.53: [bad udp cksum fdc5!] 52051+ A? db-testing-lab. (44)
Checking the ARP table again still shows no matches for the old MAC address:
[spolyack@web01 ~]$ arp -an | grep 54:75:d0
[spolyack@web01 ~]$
Interestingly enough, new TCP connections are not affected (presumably they do not match the old UDP flows):
[spolyack@web01 ~]$ telnet 10.0.2.80 53
Trying 10.0.2.80...
Connected to 10.0.2.80.
Escape character is '^]'.
00:03:43.272134 00:0c:29:47:74:26 > 00:a0:c9:00:01:01, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 24383, offset 0, flags [DF], proto TCP (6), length 60)
10.0.2.130.20130 > 10.0.2.80.53: Flags [S], cksum 0x0353 (incorrect -> 0xf74d), seq 2674341615, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 60433610 ecr 0], length
..
tcpdumping and looking at ARP requests/responses doesn't show any traces of the old MAC address, nor does the switch connecting the server and router have any entries referencing the old router's MAC address.
Disabling the flowtable eliminates the problem completely, even on a running system that is currently experiencing the above behavior:
[spolyack@web01 ~]$ time host web00.lab00 ; sudo sysctl net.inet.flowtable.enable=0 ; time host web00.lab00
;; connection timed out; no servers could be reached
real 0m10.017s
user 0m0.000s
sys 0m0.008s
net.inet.flowtable.enable: 1 -> 0
web00.lab00 has address 10.0.1.129
real 0m0.069s
user 0m0.000s
sys 0m0.003s
>Fix:
I'm not familiar with the flowtable code, so these are just general suggestions:
* If there is an ARP change (arp: xxx.xxx.xxx.xxx moved from <mac> to <mac> on em0), force expiration of flows in the flowtable which reference the old destination MAC
* Ensure that the flowtable is not matching new flows against existing flows too loosely. If all of these new UDP DNS requests are coming from new source ports, and a flow is comprised of a 4-tuple of src port, src addr, dst port, and dst addr, then old flows should not be matched for new DNS requests.
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201103161428.p2GESYfl090159>