From owner-freebsd-cluster@FreeBSD.ORG Tue Apr 21 08:37:36 2009 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D7E4106566C for ; Tue, 21 Apr 2009 08:37:36 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from mx0.deglitch.com (backbone.deglitch.com [IPv6:2001:16d8:fffb:4::abba]) by mx1.freebsd.org (Postfix) with ESMTP id C39E58FC18 for ; Tue, 21 Apr 2009 08:37:35 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from DSPAM-Daemon (localhost [127.0.0.1]) by mx0.deglitch.com (Postfix) with SMTP id C17268FC2B for ; Tue, 21 Apr 2009 12:37:33 +0400 (MSD) Received: from orion.SpringDaemons.com (unknown [77.232.3.143]) by mx0.deglitch.com (Postfix) with ESMTPA id 244EE8FC18; Tue, 21 Apr 2009 12:37:32 +0400 (MSD) Received: from orion (localhost [127.0.0.1]) by orion.SpringDaemons.com (Postfix) with SMTP id 3F17439832; Tue, 21 Apr 2009 12:37:35 +0400 (MSD) Date: Tue, 21 Apr 2009 12:37:35 +0400 From: Stanislav Sedov To: Sebastiaan van Erk Message-Id: <20090421123735.f7caf3cc.stas@FreeBSD.org> In-Reply-To: <49EC68B6.9090303@sebster.com> References: <49EC68B6.9090303@sebster.com> Organization: The FreeBSD Project X-XMPP: ssedov@jabber.ru X-Voice: +7 916 849 20 23 X-PGP-Fingerprint: F21E D6CC 5626 9609 6CE2 A385 2BF5 5993 EB26 9581 X-Mailer: carrier-pigeon Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-DSPAM-Result: Innocent X-DSPAM-Processed: Tue Apr 21 12:37:33 2009 X-DSPAM-Confidence: 1.0000 X-DSPAM-Improbability: 1 in 98689409 chance of being spam X-DSPAM-Probability: 0.0023 X-DSPAM-Signature: 49ed85cd967001477745436 Cc: freebsd-cluster@freebsd.org Subject: Re: pf and carp, BACKUP host dropping connection X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Apr 2009 08:37:36 -0000 On Mon, 20 Apr 2009 14:21:10 +0200 Sebastiaan van Erk mentioned: > Hi, > > I have 3 hosts set up with 1 virtual IP using carp. I don't yet have > pfsync (which I'm planning to do next). However, there is a strange > behavior that I cannot understand. > > The 3 machines are all gateways between two networks and have 2 VIP ips > which are used for routing (actually they have 4 networks and 4 VIPs, > but only 2 are relevant in this case). When I ssh from one network to > the other however, connections are sometimes blocked by pf. However, > they're dropped on the machine which is NOT currently master! > > That is, I have machines: > > 1) > carp1: flags=49 metric 0 mtu 1500 > inet 10.0.80.74 netmask 0xffffff00 > carp: MASTER vhid 2 advbase 1 advskew 0 > carp3: flags=49 metric 0 mtu 1500 > inet 10.0.82.74 netmask 0xffffff00 > carp: MASTER vhid 4 advbase 1 advskew 0 > > 2) > carp0: flags=49 metric 0 mtu 1500 > inet 212.61.136.74 netmask 0xfffffff0 > carp: BACKUP vhid 1 advbase 1 advskew 50 > carp2: flags=49 metric 0 mtu 1500 > inet 10.0.81.74 netmask 0xffffff00 > carp: BACKUP vhid 3 advbase 1 advskew 50 > > 3) > carp1: flags=49 metric 0 mtu 1500 > inet 10.0.80.74 netmask 0xffffff00 > carp: BACKUP vhid 2 advbase 1 advskew 100 > carp3: flags=49 metric 0 mtu 1500 > inet 10.0.82.74 netmask 0xffffff00 > carp: BACKUP vhid 4 advbase 1 advskew 100 > > > Then from the 10.0.80 network I do a ssh to the 10.0.82 network. The > router for the 10.0.82 network is 10.0.82.74 and the router for the > 10.0.80 network is 10.0.80.74 (the VIPs): > > > ssh 10.0.82.5 > sebster@10.0.82.5's password: > > Read from remote host 10.0.82.5: Connection reset by peer > Connection to 10.0.82.5 closed. > > And then I get on the backup gateways pf log: > > machine 2: > # tcpdump -nttteli pflog0 not src or dst port 6155 and not src or dst > host 224.0.0.18 and not src or dst port 68 > tcpdump: WARNING: pflog0: no IPv4 address assigned > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size > 96 bytes > 000000 rule 11/0(match): block in on em1: 10.0.80.3.58876 > > 10.0.82.5.22: [|tcp] > 001161 rule 11/0(match): block in on em1: 10.0.80.3.58876 > > 10.0.82.5.22: [|tcp] > 000018 rule 11/0(match): block in on em1: 10.0.80.3.58876 > > 10.0.82.5.22: tcp 20 [bad hdr length 0 - too short, < 20] > > machine 3: > # tcpdump -nttteli pflog0 not src or dst port 6155 and not src or dst > host 224.0.0.18 and not src or dst port 68 > tcpdump: WARNING: pflog0: no IPv4 address assigned > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size > 96 bytes > 000000 rule 11/0(match): block in on em1: 10.0.80.3.58876 > > 10.0.82.5.22: [|tcp] > 001113 rule 11/0(match): block in on em1: 10.0.80.3.58876 > > 10.0.82.5.22: [|tcp] > 000019 rule 11/0(match): block in on em1: 10.0.80.3.58876 > > 10.0.82.5.22: tcp 20 [bad hdr length 0 - too short, < 20] > > I'm wondering why these backup hosts are blocking these packets, even > though the master is still up, and why they are causing the connection > to fail. (The pf on all 3 hosts do a "block return log on devif all" > where devif is the interface with the real 10.0.80.x ip; however, why is > it returning a RST packet when it's backup?). > > I think once I have pfsync the problem will go away due to the > synchronized state (the backups won't block anymore), but it still seems > strange to me that all 3 machines will then be actively filtering the > packets... > > Does anybody know what's going on? > I'd suggest to look first why all of them're receiving this traffic. It looks like something is not right in the network itself. -- Stanislav Sedov ST4096-RIPE !DSPAM:49ed85cd967001477745436!