From owner-freebsd-net@FreeBSD.ORG Wed Mar 5 19:39:14 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3AE221065671 for ; Wed, 5 Mar 2008 19:39:14 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from smtp.sd73.bc.ca (smtp.sd73.bc.ca [142.24.13.140]) by mx1.freebsd.org (Postfix) with ESMTP id 1483A8FC18 for ; Wed, 5 Mar 2008 19:39:14 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from localhost (localhost [127.0.0.1]) by localhost.sd73.bc.ca (Postfix) with ESMTP id 59A2A1A000B18; Wed, 5 Mar 2008 11:39:13 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at smtp.sd73.bc.ca Received: from smtp.sd73.bc.ca ([127.0.0.1]) by localhost (smtp.sd73.bc.ca [127.0.0.1]) (amavisd-new, port 10024) with LMTP id aQTKrz7tXzEM; Wed, 5 Mar 2008 11:39:02 -0800 (PST) Received: from coal.local (s10.sbo [192.168.0.10]) by smtp.sd73.bc.ca (Postfix) with ESMTP id 74BBE1A000B20; Wed, 5 Mar 2008 11:39:02 -0800 (PST) From: Freddie Cash Organization: School District 73 To: "Max Laier" Date: Wed, 5 Mar 2008 11:39:01 -0800 User-Agent: KMail/1.9.7 References: <200803041351.46053.fjwcash@gmail.com> <36735.192.168.4.151.1204669226.squirrel@router> <200803041525.42330.fjwcash@gmail.com> In-Reply-To: <200803041525.42330.fjwcash@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200803051139.01547.fjwcash@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: Understanding the interplay of ipfw, vlan, and carp X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Mar 2008 19:39:14 -0000 On March 4, 2008 03:25 pm Freddie Cash wrote: > On March 4, 2008 02:20 pm Max Laier wrote: > > Am Di, 4.03.2008, 22:51, schrieb Freddie Cash: > > ... > > > > > The lack of a "carpdev" option to directly link a carp device to an > > > interface (similar to "vlandev" for vlan(4)) is what's really > > > tripping me up. It appears the carp(4) driver looks at all the > > > interfaces in the box to find one with an IP in the same subnet as > > > the carp IP and then uses that as the physical device. > > > > You could try the attached patch. It adds carpdev support. You'll > > have to recompile ifconfig to make use of it. > > > > This patch has some shortcomings that I wanted to address for a long > > time now, but never found the time to do so. Mostly that IPv6 over > > CARP is broken with this patch. Everything else is supposed to work > > and I'd like to hear if you experience otherwise (success stories > > welcome, too). This is from back in early January, but should apply > > to RELENG_7 and HEAD w/o too much trouble. Patch applied cleanly to RELENG_7.0. However, there are a few strange things happening now. If there are IPs on the physical devices (em0|em1) things only seem to work if my ipfw rules allow traffic over em0|em1. If there are no IPs on em0|em1, then the ipfw rules work fine using carp0|carp1. But it's not consistent. Sometimes the counters for the em rules increment and sometimes the counters for the carp rules increment. If there are no IPs on the physical devices, and I configure rc.conf to put two IPs onto carp0 (one with /24, one with /32) it loses the route for the /24, can't find the default router, and traffic doesn't go through. Manually adding the route via "route add -net 192.168.0.0/24 -iface carp0" allows traffic to flow again. The rc.conf entries are: cloned_interfaces="carp0 carp2" ifconfig_em0="up" ifconfig_em2="up" ifconfig_carp0="carpdev em0 vhid 100 pass whatever 192.168.0.11/24" ifconfig_carp0_alias0="192.168.0.10/32" ifconfig_carp2="carpdev em2 vhid 102 pass whatever2 172.20.0/1/24" I only upgraded one of my test boxes to RELENG_7_0. The other is still RELENG_6_3. They no longer stay in sync. Even though net.inet.carp.preempt=1 is set on both boxes, only the interface that I pull the plug on or manually down will fail-over to the other box. The ifconfig ouput on the 6.3 box will show (unplug em2 on the 6.3 box): carp0: flags=49 mtu 1500 inet 192.168.0.11 netmask 0xffffff00 inet 192.168.0.10 netmask 0xffffffff carp: MASTER vhid 100 advbase 1 advskew 150 carp2: flags=49 mtu 1500 inet 172.20.0.1 netmask 0xffffff00 carp: BACKUP vhid 102 advbase 1 advskew 150 And the ifconfig output on the 7.0 box will show: carp0: flags=8843 metric 0 mtu 1500 ether 00:00:5e:00:01:64 inet 192.168.0.10 netmask 0xffffffff inet 192.168.0.11 netmask 0xffffff00 carp: MASTER carpdev em0 vhid 100 advbase 1 advskew 0 carp2: flags=8843 metric 0 mtu 1500 ether 00:00:5e:00:01:66 inet 172.20.0.1 netmask 0xffffff00 carp: MASTER carpdev em2 vhid 102 advbase 1 advskew 0 And, finally, if I try to create two carp devices using the same physical device, with IPs in the same subnet, the box crashes. The first time, it locked up with the kernel panic. Every other time it just locks the box. The commands to do this are reproducable: ifconfig em0 up ifconfig carp0 create ifconfig carp0 carpdev em0 vhid 1 192.168.0.1/24 ifconfig carp1 create ifconfig carp1 carpdev em0 vhid 2 192.168.0.2/24 It will complain once that it can't assign the requested address. If you try the ifconfig command again, the box locks up. Might take two or three tries if you're lucky. :) -- Freddie Cash fjwcash@gmail.com