From owner-freebsd-net@FreeBSD.ORG Thu May 19 22:36:31 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF20C1065673 for ; Thu, 19 May 2011 22:36:30 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: from mail-px0-f176.google.com (mail-px0-f176.google.com [209.85.212.176]) by mx1.freebsd.org (Postfix) with ESMTP id B71098FC14 for ; Thu, 19 May 2011 22:36:30 +0000 (UTC) Received: by pxi11 with SMTP id 11so2526563pxi.7 for ; Thu, 19 May 2011 15:36:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:reply-to:date:message-id:subject :from:to:content-type; bh=devV4CUpqR/uRmmEgd4It3QVRyTBJspku2ez8vU15WA=; b=dYQQ8zAS6mW/W6TsYWgUAYk/8IKTu+/CHbAvGDNiFesy8B6lR+X5DsMRV40/nywsHh 1BEkIWJ78WXs6EYx5IkBq1wj2zbcnMYYEMGmf3aKIj6mCwlkCRczOGr0jdQE9HqmQQOd E2FlLZo7Bjg3zP8M33Fov5RlXCTP3iJP5b+x4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:reply-to:date:message-id:subject:from:to:content-type; b=CErLIWtGc8KRVRCKeJ5Ie/Z6rGXCuG5c72SBq+46CeQrx9NRq0V4ETqAWbgmq4eU7I 6DjDqhf8gD+TgMGoW+PJ6ejgvv1JUA+7Lhcz4CgmDQzvXC5asqhVsbUlzT8wm9RFEsew TN6em8pBOXTEfN7fkYiG86wql0KGnF+KFROKc= MIME-Version: 1.0 Received: by 10.68.50.9 with SMTP id y9mr5932611pbn.24.1305843169086; Thu, 19 May 2011 15:12:49 -0700 (PDT) Received: by 10.68.63.104 with HTTP; Thu, 19 May 2011 15:12:49 -0700 (PDT) Date: Thu, 19 May 2011 15:12:49 -0700 Message-ID: From: Josh Carroll To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: aliases on em(4) do not work properly X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: josh.carroll@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 May 2011 22:36:31 -0000 After upgrading my hardware, I now have two new em(4) in this box running FreeBSD 8.2-RELEASE/amd64. One NIC is the onboard NIC on the Asus P8Z68-V Pro board, the other is the Intel EXPI9301CTBLK PCI-Express card. em0 is the internet-facing card and I have 4 aliases in addition to the primary address (I have a /29 IP allocation from Comcast Business Class). After this hardware upgrade, any connection with a source address of one of the aliases on em0 does not work properly, it just hangs. Here is my rc.conf entry for this interface: ipv4_addrs_em0="XYZ.14.73.185-189/29" I also tried with the deprecated way (ifconfig_em0_alias0="..."). So connections using the default IP (XYZ.14.73.185) work fine, for example I can do the following: ping -S XYZ.14.73.185 www.google.com But if I try: ping -S XYZ.14.73.186 www.google.com It drops all packets. If I modify rc.conf to use .186 as the primary IP address and make .185 an alias, then I am able to use .186 to initiate connections. What's even more interesting is I can then set the primary IP back to .185 and have .186 as an alias again, and I can still use that IP. I thought it might be a strange arp problem, but even after putting .185 back as the default IP, and doing an arp -a -d to wipe the arp table, the .186 alias still works. I haven't rebooted, but I suspect as soon as I reboot the problem will reoccur. I can confirm that later this evening when I'm physically with the hardware. I saw a thread a couple of years ago with a similar issue: http://lists.freebsd.org/pipermail/freebsd-stable/2009-January/047839.html So I don't know if this is the same problem or just the same symptoms. Here is a tcpdump on em0 (tcpdump -n -vvv -i em0 host XYZ.14.73.188) during a ping -S XYZ.14.73.188 (I had to use .188 because .186 and .187 work now after making them the default IP as I mentioned above): 13:52:00.794099 IP (tos 0x0, ttl 64, id 20808, offset 0, flags [none], proto ICMP (1), length 84) XYZ.14.73.188 > 74.125.224.114: ICMP echo request, id 14907, seq 0, length 64 13:52:00.817650 IP (tos 0x20, ttl 53, id 43491, offset 0, flags [none], proto ICMP (1), length 84) 74.125.224.114 > XYZ.14.73.188: ICMP echo reply, id 14907, seq 0, length 64 So traffic seems to pass and responses come back, but I cannot complete a TCP handshake. Here's a tcpdump of an attempt to telnet to www.google.com on port 80: 13:54:12.662323 IP (tos 0x20, ttl 53, id 61451, offset 0, flags [none], proto TCP (6), length 60) 74.125.224.113.80 > XYZ.14.73.188.21613: Flags [S.], cksum 0x6796 (correct), seq 1796974652, ack 1835043040, win 5672, options [mss 1430,sackOK,TS val 3210048141 ecr 14620543,nop,wscale 6], length 0 13:54:13.049625 IP (tos 0x20, ttl 53, id 61451, offset 0, flags [none], proto TCP (6), length 60) 74.125.224.113.80 > XYZ.14.73.188.21613: Flags [S.], cksum 0x6613 (correct), seq 1796974652, ack 1835043040, win 5672, options [mss 1430,sackOK,TS val 3210048528 ecr 14620543,nop,wscale 6], length 0 13:54:13.650624 IP (tos 0x20, ttl 53, id 61451, offset 0, flags [none], proto TCP (6), length 60) 74.125.224.113.80 > XYZ.14.73.188.21613: Flags [S.], cksum 0x63ba (correct), seq 1796974652, ack 1835043040, win 5672, options [mss 1430,sackOK,TS val 3210049129 ecr 14620543,nop,wscale 6], length 0 13:54:14.849538 IP (tos 0x20, ttl 53, id 61451, offset 0, flags [none], proto TCP (6), length 60) 74.125.224.113.80 > XYZ.14.73.188.21613: Flags [S.], cksum 0x5f0a (correct), seq 1796974652, ack 1835043040, win 5672, options [mss 1430,sackOK,TS val 3210050329 ecr 14620543,nop,wscale 6], length 0 13:54:15.646261 IP (tos 0x10, ttl 64, id 60781, offset 0, flags [none], proto TCP (6), length 60) XYZ.14.73.188.21613 > 74.125.224.113.80: Flags [S], cksum 0xb735 (correct), seq 1835043039, win 65535, options [mss 1460,nop,wscale 5,sackOK,TS val 14623543 ecr 0], length 0 13:54:15.662281 IP (tos 0x20, ttl 53, id 61451, offset 0, flags [none], proto TCP (6), length 60) 74.125.224.113.80 > XYZ.14.73.188.21613: Flags [S.], cksum 0x5bde (correct), seq 1796974652, ack 1835043040, win 5672, options [mss 1430,sackOK,TS val 3210051141 ecr 14620543,nop,wscale 6], length 0 13:54:17.253367 IP (tos 0x20, ttl 53, id 61451, offset 0, flags [none], proto TCP (6), length 60) 74.125.224.113.80 > XYZ.14.73.188.21613: Flags [S.], cksum 0x55a6 (correct), seq 1796974652, ack 1835043040, win 5672, options [mss 1430,sackOK,TS val 3210052733 ecr 14620543,nop,wscale 6], length 0 ^C I tried adding an alias on em1 and I could ping things on my LAN with ping -S , so it seems particular to em0. I think em0 is the onboard NIC and em1 is the PCI-Express card. Here is the relevant pciconf output: em0@pci0:0:25:0: class=0x020000 card=0x849c1043 chip=0x15038086 rev=0x05 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xfbe00000, size 131072, enabled bar [14] = type Memory, range 32, base 0xfbe28000, size 4096, enabled bar [18] = type I/O Port, range 32, base 0xf080, size 32, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 13[e0] = PCI Advanced Features: FLR TP em1@pci0:4:0:0: class=0x020000 card=0xa01f8086 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xfbcc0000, size 131072, enabled bar [14] = type Memory, range 32, base 0xfbc00000, size 524288, enabled bar [18] = type I/O Port, range 32, base 0xe000, size 32, enabled bar [1c] = type Memory, range 32, base 0xfbce0000, size 16384, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled And here is the relevant dmesg output: em0: port 0xf080-0xf09f mem 0xfbe00000-0xfbe1ffff,0xfbe28000-0xfbe28fff irq 18 at device 25.0 on pci0 em0: Using an MSI interrupt em0: [FILTER] em0: Ethernet address: AA:BB:CC:DD:EE:FF em1: port 0xe000-0xe01f mem 0xfbcc0000-0xfbcdffff,0xfbc00000-0xfbc7ffff,0xfbce0000-0xfbce3fff irq 18 at device 0.0 on pci4 em1: Using MSIX interrupts with 3 vectors em1: [ITHREAD] em1: [ITHREAD] em1: [ITHREAD] em1: Ethernet address: FF:EE:DD:CC:BB:AA ifconfig em0 output: em0: flags=8843 metric 0 mtu 1500 options=219b ether AA:BB:CC:DD:EE:FF inet XYZ.14.73.185 netmask 0xfffffff8 broadcast XYZ.14.73.191 inet XYZ.14.73.186 netmask 0xffffffff broadcast XYZ.14.73.186 inet XYZ.14.73.187 netmask 0xffffffff broadcast XYZ.14.73.187 inet XYZ.14.73.188 netmask 0xffffffff broadcast XYZ.14.73.188 inet XYZ.14.73.189 netmask 0xffffffff broadcast XYZ.14.73.189 media: Ethernet autoselect (1000baseT ) status: active arp -na output (LAN IPs masked out): ? (XYZ.14.73.186) at AA:BB:CC:DD:EE:FF on em0 permanent [ethernet] ? (XYZ.14.73.187) at AA:BB:CC:DD:EE:FF on em0 permanent [ethernet] ? (XYZ.14.73.185) at AA:BB:CC:DD:EE:FF on em0 permanent [ethernet] ? (XYZ.14.73.190) at 00:22:2d:6c:e0:3e on em0 expires in 113 seconds [ethernet] ? (XYZ.14.73.188) at AA:BB:CC:DD:EE:FF on em0 permanent [ethernet] ? (XYZ.14.73.189) at AA:BB:CC:DD:EE:FF on em0 permanent [ethernet] netstat -nr output: Internet: Destination Gateway Flags Refs Use Netif Expire default XYZ.14.73.190 UGS 34 27763 em0 10.0.0.0/24 link#2 U 2 96232 em1 => 10.0.0.0/8 link#2 U 1 28 em1 10.0.0.1 link#2 UHS 0 837 lo0 10.0.0.100 link#2 UHS 0 0 lo0 10.10.10.1 link#2 UHS 0 0 lo0 => 10.10.10.1/32 link#2 U 0 0 em1 10.10.10.2 link#2 UHS 0 0 lo0 => 10.10.10.2/32 link#2 U 0 0 em1 127.0.0.1 link#5 UH 0 486 lo0 XYZ.14.73.184/29 link#1 U 0 0 em0 XYZ.14.73.185 link#1 UHS 0 137 lo0 XYZ.14.73.186 link#1 UHS 0 0 lo0 => XYZ.14.73.186/32 link#1 U 0 0 em0 XYZ.14.73.187 link#1 UHS 0 0 lo0 => XYZ.14.73.187/32 link#1 U 0 0 em0 XYZ.14.73.188 link#1 UHS 0 0 lo0 => XYZ.14.73.188/32 link#1 U 0 0 em0 XYZ.14.73.189 link#1 UHS 0 0 lo0 => XYZ.14.73.189/32 link#1 U 0 0 em0 I can try building an 8-STABLE kernel to see if it works ok, but ideally I'd like to remain on 8.2-RELEASE. Please let me know what other information is needed to help debug this. Thanks, Josh