From owner-freebsd-net@FreeBSD.ORG Wed Mar 5 22:18:42 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 62E5D1065697 for ; Wed, 5 Mar 2008 22:18:42 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from smtp.sd73.bc.ca (smtp.sd73.bc.ca [142.24.13.140]) by mx1.freebsd.org (Postfix) with ESMTP id 7BEC98FC26 for ; Wed, 5 Mar 2008 22:18:41 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from localhost (localhost [127.0.0.1]) by localhost.sd73.bc.ca (Postfix) with ESMTP id 123151A000B0E; Wed, 5 Mar 2008 14:18:40 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at smtp.sd73.bc.ca Received: from smtp.sd73.bc.ca ([127.0.0.1]) by localhost (smtp.sd73.bc.ca [127.0.0.1]) (amavisd-new, port 10024) with LMTP id uIMdH3YsPFNl; Wed, 5 Mar 2008 14:18:33 -0800 (PST) Received: from coal.local (s10.sbo [192.168.0.10]) by smtp.sd73.bc.ca (Postfix) with ESMTP id 472511A000B0B; Wed, 5 Mar 2008 14:18:33 -0800 (PST) From: Freddie Cash Organization: School District 73 To: "Max Laier" Date: Wed, 5 Mar 2008 14:18:32 -0800 User-Agent: KMail/1.9.7 References: <200803041351.46053.fjwcash@gmail.com> <200803051139.01547.fjwcash@gmail.com> <41303.192.168.4.151.1204747767.squirrel@router> In-Reply-To: <41303.192.168.4.151.1204747767.squirrel@router> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200803051418.32940.fjwcash@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: Understanding the interplay of ipfw, vlan, and carp X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Mar 2008 22:18:42 -0000 On March 5, 2008 12:09 pm you wrote: > Am Mi, 5.03.2008, 20:39, schrieb Freddie Cash: > > On March 4, 2008 03:25 pm Freddie Cash wrote: > > Patch applied cleanly to RELENG_7.0. However, there are a few > > strange things happening now. > > > > If there are IPs on the physical devices (em0|em1) things only seem > > to work if my ipfw rules allow traffic over em0|em1. If there are no > > IPs on em0|em1, then the ipfw rules work fine using carp0|carp1. But > > it's not consistent. Sometimes the counters for the em rules > > increment and sometimes the counters for the carp rules increment. > > I'll look into this ... it would help if you could qualify "it's not > consistent" a bit, so that I can reproduce. I'll have to run some more tests on this to try and narrow things down, and make sure I'm actually seeing what I think I'm seeing. This might just be me misunderstanding how the network stack works, and how a packet travels through the physical interfaces, through the virtual interfaces, and through the packet filter. > > The rc.conf entries are: > > cloned_interfaces="carp0 carp2" > > ifconfig_em0="up" > > ifconfig_em2="up" > > ifconfig_carp0="carpdev em0 vhid 100 pass whatever > > 192.168.0.11/24" > > ifconfig_carp0_alias0="192.168.0.10/32" > > ifconfig_carp2="carpdev em2 vhid 102 pass whatever2 172.20.0/1/24" > > > > I only upgraded one of my test boxes to RELENG_7_0. The other is > > still RELENG_6_3. They no longer stay in sync. Even though > > net.inet.carp.preempt=1 is set on both boxes, only the interface that > > I pull the plug on or manually down will fail-over to the other box. > > > > The ifconfig ouput on the 6.3 box will show (unplug em2 on the 6.3 > > box): carp0: flags=49 mtu 1500 > > inet 192.168.0.11 netmask 0xffffff00 > > inet 192.168.0.10 netmask 0xffffffff > > carp: MASTER vhid 100 advbase 1 advskew 150 > > carp2: flags=49 mtu 1500 > > inet 172.20.0.1 netmask 0xffffff00 > > carp: BACKUP vhid 102 advbase 1 advskew 150 > > > > And the ifconfig output on the 7.0 box will show: > > carp0: flags=8843 metric 0 > > mtu 1500 > > ether 00:00:5e:00:01:64 > > inet 192.168.0.10 netmask 0xffffffff > > inet 192.168.0.11 netmask 0xffffff00 > > carp: MASTER carpdev em0 vhid 100 advbase 1 advskew 0 > > carp2: flags=8843 metric 0 > > mtu 1500 > > ether 00:00:5e:00:01:66 > > inet 172.20.0.1 netmask 0xffffff00 > > carp: MASTER carpdev em2 vhid 102 advbase 1 advskew 0 > > What does "netstat -ssp carp" say? It seems that vhid 100 doesn't sync > at all. Might be a problem with the order of the address list. FreeBSD 6.3 box: carp: 1649 packets received (IPv4) 1649 discarded for bad authentication 6871 packets sent (IPv4) FreeBSD 7.0 box: carp: 1138 packets received (IPv4) 1138 discarded for bad authentication 1797 packets sent (IPv4) The rc.conf entries from the 6.3 box: ifconfig_carp0="vhid 100 pass nexus-carp-pass advskew 150 192.168.0.11/24" "ifconfig carp0" lists 192.168.0.11/24 first and 192.168.0.10/32 second. The rc.conf entry from the 7.0 box: ifconfig_carp0="carpdev em0 vhid 100 pass nexus-carp-pass 192.168.0.11/24" "ifconfig carp0" lists 192.168.0.10/32 first and 192.168.0.11/24 second. If I create the carp devices in the exact same order on each box, using the exact same commands (but with carpdev added on the 7.0 box), with only 1 IP on each carp interface, then things almost work. If I down carp2 on the 7.0 box, carp2 on the 6.3 box becomes the master, but carp0 remains as BACKUP on the 6.3 box. And vice versa when I down carp0 on the 7.0 box. Changing the advskew option on the 7.0 box to be 200 causes both carp devices switch. 6.3 becomes master and 7.0 becomes backup. BUT, downing one interface still only causes that one to failover. net.inet.carp_preempt is still set to 1 on both boxes. If I create two IPs on the carp interface, even if created in the exact same order on box boxes, then they won't failover at all. Both boxes show all the carp interfaces set to MASTER. And the discarded counters in "netstat -ssp carp" increment on both boxes every second. Thanks for your help on this. If needed, I can upgrade the other 6.3 box to 7.0. -- Freddie Cash fjwcash@gmail.com