From owner-freebsd-pf@freebsd.org Sat Dec 3 10:48:57 2016 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48C41C64874 for ; Sat, 3 Dec 2016 10:48:57 +0000 (UTC) (envelope-from patfbsd@davenulle.org) Received: from sender163-mail.zoho.com (sender163-mail.zoho.com [74.201.84.163]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 23D57221 for ; Sat, 3 Dec 2016 10:48:56 +0000 (UTC) (envelope-from patfbsd@davenulle.org) Received: from baby-jane.lamaiziere.net (32.169.100.84.rev.sfr.net [84.100.169.32]) by mx.zohomail.com with SMTPS id 1480762123082489.0648164014061; Sat, 3 Dec 2016 02:48:43 -0800 (PST) Date: Sat, 3 Dec 2016 11:48:35 +0100 From: Patrick Lamaiziere Cc: freebsd-pf@freebsd.org Subject: Re: 10.3 pfsync large difference between number of states on two firewalls Message-ID: <20161203114835.100355c0@baby-jane.lamaiziere.net> In-Reply-To: <20161024145926.4770faf5@mr185083> References: <20161024145926.4770faf5@mr185083> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.29; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 10:48:57 -0000 Le Mon, 24 Oct 2016 14:59:26 +0200, Patrick Lamaiziere a =E9crit : > (trying freebsd-pf) >=20 > Hello, >=20 > I have a pair of firewalls with carp, pf and pfsync and I see a large > difference between the number of states (pfctl -si, current entries) > on the firewalls. The pfsync link is a 10 GB link witht around 20 > Kpps on load (don't think it's the issue). >=20 > pf1 is the master with 807598 states, > pf2 is the backup with 1696258 states=20 >=20 > There is only small traffic from / to the firewalls that can explain > this difference. >=20 > I'm looking on the states (but it's not easy on real traffic) and I've > found some states not present in pf1, but still present in pf2. >=20 > One states was in state tcp ESTABLISHED:ESTABLISHED with a expire age > around 23:55:00 (the default of a tcp timeout) and I can confirm that > the tcp session was ended (with netflow traces) and started 5 minutes > ago. >=20 > So it looks like sometimes pf2 misses (or pf1 does not send) some > state updates. >=20 > I say "sometimes" because with the rates of states inserts here, I > think that if this is always the case, the states table on pf2 would > have already exploded. Hello, For the record the problem was on the physical link between the two firewall. On ix (intel X520 here) cards there is a problem when SFP+ device are connected back to back.=20 The README says : "When 82599-based SFP+ devices are connected back to back, they should be set to the same Speed setting. Results may vary if you mix speed settings." Unfortunatly we cannot set the media type and speed on ix driver (only auto detection is available). When a firewall reboot, this affects the media detection on the other firewall (seen with ifconfig). And then the ix card starts to drop packets on output (sysctl dev.ix.N.queue.N.br_drops). Then only a ifconfig down/ up resumes normal operation. So I have moved the pfsync device to a copper interface. That works fine now. Regards.