From owner-freebsd-net@FreeBSD.ORG Fri Nov 2 00:08:38 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CEE063D0 for ; Fri, 2 Nov 2012 00:08:38 +0000 (UTC) (envelope-from rand@meridian-enviro.com) Received: from zimbra.meridian-enviro.com (zimbra.meridian-enviro.com [12.192.92.32]) by mx1.freebsd.org (Postfix) with ESMTP id 770908FC08 for ; Fri, 2 Nov 2012 00:08:38 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by zimbra.meridian-enviro.com (Postfix) with ESMTP id CFA07307009B for ; Thu, 1 Nov 2012 18:59:05 -0500 (CDT) X-Virus-Scanned: amavisd-new at meridian-enviro.com Received: from zimbra.meridian-enviro.com ([127.0.0.1]) by localhost (zimbra.meridian-enviro.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PHJ-FjkQgOCZ; Thu, 1 Nov 2012 18:59:04 -0500 (CDT) Received: from delta.meridian-enviro.com (delta.meridian-enviro.com [10.10.10.43]) by zimbra.meridian-enviro.com (Postfix) with ESMTPSA id E72C1307009A; Thu, 1 Nov 2012 18:59:04 -0500 (CDT) Message-ID: <50930CC8.4050406@meridian-enviro.com> Date: Thu, 01 Nov 2012 18:59:04 -0500 From: "Douglas K. Rand" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:12.0) Gecko/20120522 Thunderbird/12.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Trouble with TCP/UDP picking source addresses Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Ryan Langseth X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2012 00:08:38 -0000 We have an 8.3 system that picks the wrong, or at least inconvenient, source IP address for UDP and TCP packets. This *only* happens when sending packets to itself, never when sending packets to other hosts. And not when sending packets to 127.0.0.1. I *think* the problem might be related to this system being a CARP backup system. The local system has as its "internal" IP address 10.100.2.11. This is on a VLAN interface called internal0. It also has a carp0 interface in backup state with 10.100.2.1 as an address. When I send TCP or UDP packets from the system back to itself using 10.100.2.11, tcpdump (watching lo0) shows the source address as 10.100.2.1: 18:47:44.742063 IP 10.100.2.1.45061 > 10.100.2.11.53: 31845+ A? puppet.r2.ivr.meridian-enviro.com. (51) And of course I see named trying to reply to this request on the internal0 interface, but the packet gets routed out to the other CARP host that is currently master: 18:47:44.742245 IP 10.100.2.11.53 > 10.100.2.1.45061: 31845* 2/3/3 CNAME front0-vpn.r2.ivr.meridian-enviro.com., A 10.100.2.10 (201) I can "fix" this by destroying and re-creating the carp0 interface. What I'm thinking is that somehow the kernel is latching onto the IP address of carp0 instead of internal0. Perhaps because carp0 is created before internal0 during boot? If I destroy and re-create carp0, then internal0 is earlier in the list of interfaces. At least how I seem them with ifconfig. I've verified that both UDP and TCP do the same thing. But ICMP doesn't for some reason, it picks the "right" address of 10.100.2.11 as the source address when I use ping. We have another 8.3 based firewall with a similar, but not quite identical, configuration. But it doesn't exhibit the problem, it picks a source from the VLAN interface, not the CARP interface. I was wondering if anybody has any ideas. Here is the output from ifconfig: bce0: flags=8843 \ metric 0 mtu 1500 options=c01bb ether 84:8f:69:e3:a1:51 inet 65.101.96.19 netmask 0xfffffff8 broadcast 65.101.96.23 media: Ethernet autoselect (1000baseT ) status: active bce1: flags=8943 \ metric 0 mtu 1500 options=c01bb ether 84:8f:69:e3:a1:53 media: Ethernet autoselect (1000baseT ) status: active bce2: flags=8802 metric 0 mtu 1500 options=c01bb ether 84:8f:69:e3:a1:55 media: Ethernet autoselect bce3: flags=8843 \ metric 0 mtu 1500 options=c01bb ether 84:8f:69:e3:a1:57 inet 10.254.3.11 netmask 0xffffff00 broadcast 10.254.3.255 media: Ethernet autoselect (1000baseT ) status: active ipfw0: flags=8801 metric 0 mtu 65536 lo0: flags=8149 \ metric 0 mtu 16384 options=3 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xc inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 nd6 options=3 pfsync0: flags=41 metric 0 mtu 1460 pfsync: syncdev: bce3 syncpeer: 10.254.3.10 maxupd: 128 pflog0: flags=141 metric 0 mtu 33152 carp0: flags=49 metric 0 mtu 1500 inet 10.100.2.1 netmask 0xffffff00 carp: BACKUP vhid 12 advbase 1 advskew 150 internal0: flags=8943 \ metric 0 mtu 1500 options=103 ether 84:8f:69:e3:a1:53 inet 10.100.2.11 netmask 0xffffff00 broadcast 10.100.2.255 media: Ethernet autoselect (1000baseT ) status: active vlan: 1 parent interface: bce1 management0: flags=8843 \ metric 0 mtu 1500 options=103 ether 84:8f:69:e3:a1:53 inet 10.253.0.11 netmask 0xffffff00 broadcast 10.253.0.255 media: Ethernet autoselect (1000baseT ) status: active vlan: 410 parent interface: bce1