From owner-freebsd-pf@freebsd.org Sun Aug 23 15:10:00 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CBBF79C040D; Sun, 23 Aug 2015 15:10:00 +0000 (UTC) (envelope-from kp@vega.codepro.be) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6468FCEC; Sun, 23 Aug 2015 15:10:00 +0000 (UTC) (envelope-from kp@vega.codepro.be) Received: from vega.codepro.be (unknown [172.16.1.3]) by venus.codepro.be (Postfix) with ESMTP id CB50CE295; Sun, 23 Aug 2015 17:09:57 +0200 (CEST) Received: by vega.codepro.be (Postfix, from userid 1001) id B528B25B39; Sun, 23 Aug 2015 17:09:57 +0200 (CEST) Date: Sun, 23 Aug 2015 17:09:57 +0200 From: Kristof Provost To: freebsd-net@freebsd.org, freebsd-pf@freebsd.org Subject: Near-term pf plans Message-ID: <20150823150957.GK48727@vega.codepro.be> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Checked-By-NSA: Probably User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 15:10:01 -0000 Hi, Some of you may have noticed that I fixed a couple of pf issues (or in some cases broke things. Sorry Allan.) recently. Here's a quick list of my current priorities: - PR 127042, 202178: This is a panic when an interface an ifgroup have the same name. There's a fix for the immediate problem in https://reviews.freebsd.org/D3435 I'm inclined to say that ifgroups and interfaces should share a namespace. That would certainly help pf, because it uses both interchangeably, which just doesn't work unless they're in the same namespace. That affects more in the network stack though, so I'd like opinions for people with more experience with ifgroups. - PR 202351 This is a panic after ip6 reassembly in pf. We set the rcvif to NULL when refragmenting. That seems to go OK execpt when we're refragmenting broadcast/multicast packets in the forwarding path. It's not at all clear to me how that could happen. - Removal of frag drop / frag crop support in pf. pf has two (or really three, but crop and drop mode are very similar) ways to handle fragmented packets. I've recently extended the 'reassemble' mode to also work for ip6. The crop/drop modes really only verify that fragments make sense (i.e. don't claim to deliver data beyond the last fragment, or duplicate data). I'd like to remove support for the crop/drop modes for two reasons: * the code is relatively large and complex. I've already run into a couple of problems related to it. * It doesn't do what users expect. Enabling scrub fragment crop doesn't actually mean pf will filter on the entire fragment. You still end up having to let all fragments pass through the firewall. There's a patch to do just that here: https://reviews.freebsd.org/D3466 - PR 198868, 193579 (TSO issues) pf has issues on certain network interfaces with TSO enabled. The most important of these are amazon VMs. I believe the problem is that pf tries to work with packets with full checksums. Usually output packets have pseudo-header checksums in the IP and TCP headers. pf doesn't know about those. It always tries to do updates on checksums assuming there's a full checksum. (Which is the case, pf always calculates a full checksum in pf_check_out()) I believe the fix for this issue is to have pf be aware of the pseudo-header checksums. The type of checksum can be determined from the CSUM_TCP and CSUM_TCP_IPV6 flags. Whenever pf changes a packet header it will have to check for those flags to figure out if the checksum should be updated or not. - PR 188511 divert-reply implementation for pf I need to review the use case and implementation of the work done by PiBa.NL.dev@gmail.com - PR 172648 This bug started out with an issue with TCP reassembly, but I've not been able to reproduce that. There is a problem with rdr targets for ipv6 though. With rdr to ::1 we fail the scope check in ip6_input() (right after the pfil hook) because we have a packet to localhost with a m->m_pkthdr.rcvif which is not a loopback interface. That can be fixed by having pf rewrite the rcvif, but that'd special-case rdr to ::1. We've got a similar problem for the reply. There we've got a packet from ::1 to something else. This fails the scope lookup too. In essence the problem is that we've already made the routing decision before pf gets the chance to rewrite the destination address. I'm not quite sure how to fix this though. If anyone has any comments or questions, or disagrees with any of the above, please let me know. If anyone knows of critical problems not on this list please let me know, and I'll see what I can do. Also, pf is currently an (unpaid) side project for me. I'm reasonably limited in the amount of time I can invest in it. Regards, Kristof From owner-freebsd-pf@freebsd.org Sun Aug 23 16:53:58 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E5D29C0AAF for ; Sun, 23 Aug 2015 16:53:58 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from phabric-backend.isc.freebsd.org (phabric-backend.isc.freebsd.org [IPv6:2001:4f8:3:ffe0:406a:0:50:2]) by mx1.freebsd.org (Postfix) with ESMTP id E95691729 for ; Sun, 23 Aug 2015 16:53:57 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by phabric-backend.isc.freebsd.org (Postfix, from userid 1346) id E5E9218322; Sun, 23 Aug 2015 16:53:57 +0000 (UTC) Date: Sun, 23 Aug 2015 16:53:57 +0000 To: freebsd-pf@freebsd.org From: "javier_ovi_yahoo.com (Javier Villavicencio)" Reply-to: D1944+331+90181aefda88703e@reviews.freebsd.org Subject: [Differential] [Changed Subscribers] D1944: PF and VIMAGE fixes Message-ID: X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: Thread-Topic: D1944: PF and VIMAGE fixes X-Herald-Rules: none X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: NDc2NzM0MzY4OTdiYThiNTU1MjY2ZDZmMTJiIFXZ+qU= MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="utf-8" X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 16:53:58 -0000 javier_ovi_yahoo.com added a subscriber: javier_ovi_yahoo.com. REVISION DETAIL https://reviews.freebsd.org/D1944 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: nvass-gmx.com, bz, trociny, kristof, gnn, zec, rodrigc, glebius, eri Cc: javier_ovi_yahoo.com, farrokhi, julian, robak, freebsd-virtualization-list, freebsd-pf-list, freebsd-net-list From owner-freebsd-pf@freebsd.org Mon Aug 24 15:33:14 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 63AEA9C0C88; Mon, 24 Aug 2015 15:33:14 +0000 (UTC) (envelope-from markus.gebert@hostpoint.ch) Received: from mail.adm.hostpoint.ch (mail.adm.hostpoint.ch [IPv6:2a00:d70:0:a::e0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2A09A169E; Mon, 24 Aug 2015 15:33:13 +0000 (UTC) (envelope-from markus.gebert@hostpoint.ch) Received: from [2001:1620:2013:1:b1fa:bcf5:a6e2:f2a0] (port=59328) by mail.adm.hostpoint.ch with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.84 (FreeBSD)) (envelope-from ) id 1ZTtkD-0005yj-Tw; Mon, 24 Aug 2015 17:33:09 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: Near-term pf plans From: Markus Gebert In-Reply-To: <20150823150957.GK48727@vega.codepro.be> Date: Mon, 24 Aug 2015 17:33:08 +0200 Cc: freebsd-net@freebsd.org, freebsd-pf@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <3121D8E4-A27E-475B-9771-C09347D1D793@hostpoint.ch> References: <20150823150957.GK48727@vega.codepro.be> To: Kristof Provost X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 15:33:14 -0000 Hi Kristof > On 23.08.2015, at 17:09, Kristof Provost wrote: >=20 > - PR 202351 > This is a panic after ip6 reassembly in pf. We set the rcvif to NULL > when refragmenting. That seems to go OK execpt when we're = refragmenting > broadcast/multicast packets in the forwarding path. It's not at all > clear to me how that could happen. if_bridge wants to forward ipv6 multicasts. pf refragmentation code = tries to send out the resulting packets using ip6_forward() which does = not handle multicasts, drops the packet and tries to log that fact, = which causes the panic. I=E2=80=99ve updated the PR with some more thoughts about this. Markus From owner-freebsd-pf@freebsd.org Mon Aug 24 16:16:15 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6A189C1DE0; Mon, 24 Aug 2015 16:16:15 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9C8FCCFA; Mon, 24 Aug 2015 16:16:15 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [IPv6:2a02:1811:2419:4e02:6d:6d9f:9966:b7fe] (unknown [IPv6:2a02:1811:2419:4e02:6d:6d9f:9966:b7fe]) by venus.codepro.be (Postfix) with ESMTPSA id 628EBB5A9; Mon, 24 Aug 2015 18:16:12 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.0 \(3083\)) Subject: Re: Near-term pf plans From: Kristof Provost X-Mailer: Apple Mail (2.3083) In-Reply-To: <3121D8E4-A27E-475B-9771-C09347D1D793@hostpoint.ch> Date: Mon, 24 Aug 2015 18:16:12 +0200 Cc: freebsd-net@freebsd.org, freebsd-pf@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <1DDBFAD5-9AFB-4A21-8D16-BD85AB30F448@FreeBSD.org> References: <20150823150957.GK48727@vega.codepro.be> <3121D8E4-A27E-475B-9771-C09347D1D793@hostpoint.ch> To: Markus Gebert X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 16:16:16 -0000 > On 24 Aug 2015, at 17:33, Markus Gebert = wrote: >=20 >> On 23.08.2015, at 17:09, Kristof Provost wrote: >>=20 >> - PR 202351 >> This is a panic after ip6 reassembly in pf. We set the rcvif to NULL >> when refragmenting. That seems to go OK execpt when we're = refragmenting >> broadcast/multicast packets in the forwarding path. It's not at all >> clear to me how that could happen. >=20 > if_bridge wants to forward ipv6 multicasts. pf refragmentation code = tries to send out the resulting packets using ip6_forward() which does = not handle multicasts, drops the packet and tries to log that fact, = which causes the panic. >=20 > I=E2=80=99ve updated the PR with some more thoughts about this. >=20 Yes, I saw that pass by earlier. Thanks for that, I think you did a = great analysis. Unfortunately there are other issues with pf on bridges. (See PR 185633 = for example) I wouldn=E2=80=99t expect the fragmentation and reassembly to work at = all in that scenario. I=E2=80=99ll see what I can do about at least fixing the panic in the = short term. Even if the reassembly/refragmentation doesn=E2=80=99t work (on bridges) = we should at least no panic. Regards, Kristof From owner-freebsd-pf@freebsd.org Tue Aug 25 15:56:25 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3A39C99AF80 for ; Tue, 25 Aug 2015 15:56:25 +0000 (UTC) (envelope-from Andrej.Kolontai@verwaltung.uni-muenchen.de) Received: from mailto2.verwaltung.uni-muenchen.de (mailto2.verwaltung.uni-muenchen.de [141.84.149.7]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "IronPort Appliance Demo Certificate", Issuer "IronPort Appliance Demo Certificate" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 614886E for ; Tue, 25 Aug 2015 15:56:23 +0000 (UTC) (envelope-from Andrej.Kolontai@verwaltung.uni-muenchen.de) X-IronPort-AV: E=McAfee;i="5700,7163,7903"; a="2993810" X-IronPort-AV: E=Sophos;i="5.15,746,1432591200"; d="scan'208";a="2993810" Received: from cashts1.zuv.uni-muenchen.de ([10.153.81.103]) by smtpout2.verwaltung.uni-muenchen.de with ESMTP/TLS/AES256-SHA; 25 Aug 2015 17:55:11 +0200 Received: from MXS2.zuv.uni-muenchen.de ([fe80::e8db:cdb2:9a:a69f]) by CASHTS1.zuv.uni-muenchen.de ([::1]) with mapi id 14.03.0248.002; Tue, 25 Aug 2015 17:55:08 +0200 From: Kolontai Andrej To: "'freebsd-pf@freebsd.org'" Subject: Machine freezes when loading pf ruleset Thread-Topic: Machine freezes when loading pf ruleset Thread-Index: AdDfSL0WCcADlcE1Qv6+xfswQgISow== Date: Tue, 25 Aug 2015 15:55:08 +0000 Message-ID: Accept-Language: de-DE, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.23.107.156] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2015 15:56:25 -0000 Hello, I'm new to this list and I hope it's the right place to ask.=20 We have highly utilized installation of two FreeBSD-machines running 10.1-R= ELEASE, pf and carp. There are about 50 networks (some via vlan, some ipsec= ) connected to them, usually about 50000 pf states, about 1500 rules and tr= affic sometimes hitting 1 gbit/s or more.=20 When we load the ruleset on the active machine it sometimes freezes. It occ= urs more often as our installation grows. Sometimes it freezes completely, = which is actually good as the backup machine takes over. But at other times= it's still some kind of alive but unbearably slow (ping is about 40 s inst= ead of the usual 0.2 ms), the backup machine does not take over and people = start to get angry.=20 The crashed machine usually recovers after a couple of minutes.=20 We've been trying to resolve this for some months now and are running out o= f ideas. We do believe that something is wrong with or setup but have no cl= ue what it could be.=20 When it happens, the syslogs show tons of messages like these: Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps 172.23.101.87= to 141.84.149.18 Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps 172.23.101.87= to 141.84.149.18 Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps 172.23.107.20= 6 to 141.84.149.18 Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps 172.23.101.87= to 141.84.149.18 Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps 172.23.107.20= 6 to 141.84.149.18 Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps 172.23.101.87= to 141.84.149.18 Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps 172.23.106.17= 1 to 141.84.149.18 Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps 172.23.106.17= 1 to 141.84.149.18 Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps 172.23.106.72= to 141.84.149.18 Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps 172.23.106.58= to 141.84.149.19 Aug 24 09:08:39 applej last message repeated 2 times Aug 24 09:08:39 applej kernel: pf: BAD state: TCP in wire: 10.200.108.141:4= 9202 172.23.2.92:8192 stack: - [lo=3D3482585550 high=3D3482585552 win=3D819= 2 modulator=3D0pf_map_addr: src tracking maps ]172.23.106.72 [lo=3D0 hi gh=3D1 win=3D1 modulator=3D0 to ]141.84.149.18 2:0 Aug 24 09:08:39 applej kernel: S seq=3D3503222902 (3503222902) ack=3D0 len= =3D0 ackskew=3D0 pkts=3D1:1 dir=3Din,fwd Aug 24 09:08:39 applej kernel: pf: State failure on: 1 | 5 =20 Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps 172.23.106.17= 1 to 141.84.149.18 Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps 172.23.107.20= to 141.84.149.18 Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps 172.23.101.87= to 141.84.149.18 Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps pf_map_addr: = src tracking maps pf_map_addr: src tracking maps 172.23.106.171172.23.106.1= 71 to 172.23.106.171141.84.149.18 to to=20 Aug 24 09:08:39 applej kernel: 141.84.149.18141.84.149.18 Aug 24 09:08:39 applej kernel:=20 Aug 24 09:08:40 applej kernel: pf_map_addr: src tracking maps pf_map_addr: = src tracking maps 172.23.106.171172.23.101.87 to=20 Aug 24 09:08:40 applej kernel: to 141.84.149.18141.84.149.18 Aug 24 09:08:40 applej kernel:=20 Aug 24 09:08:40 applej kernel: pf: OK ICMP 3:10 141.84.149.249 -> 10.153.10= 1.239 state: TCP in wire: 141.84.149.249:37677 10.153.101.239 Aug 24 09:08:40 applej kernel: :80 stack: - [lo=3D4171141930 high=3D4171159= 337 win=3D106 modulator=3D0 wscale=3D7] [lo=3D2653133771 high=3D2653147339 = win=3D136 modulator=3D0 wscale=3D7] 10:10 seq=3D2653133771 Aug 24 09:08:40 applej kernel: pf: OK ICMP 3:10 141.84.149.249 -> 10.153.10= 1.239 state: TCP out wire: 10.153.101.239:80 141.84.149.249 Aug 24 09:08:40 applej kernel: :37677 stack: - [lo=3D4171141930 high=3D4171= 159337 win=3D106 modulator=3D0 wscale=3D7] [lo=3D2653133771 high=3D26531473= 39 win=3D136 modulator=3D0 wscale=3D7] 10:10 seq=3D2653133771 Aug 24 09:08:40 applej kernel: pf_map_addr: src tracking maps pf_map_addr: = src tracking maps 172.23.106.184pf: wire key attach failed on vlan42: 172.2= 3.106.165pf_map_addr: src tracking maps=20 Aug 24 09:08:40 applej kernel: 112 to in141.84.149.18 wire: 172.23.106.58 Aug 24 09:08:40 applej kernel: 141.84.149.27 Aug 24 09:08:40 applej kernel: to 224.0.0.18 to 1:0141.84.149.18, existing= :=20 Aug 24 09:08:40 applej kernel: 112141.84.149.19 in Aug 24 09:08:40 applej kernel: wire:=20 Aug 24 09:08:40 applej kernel: 141.84.149.27=20 Aug 24 09:08:40 applej kernel: 224.0.0.18 stack: 141.84.149.27 224.0.0.18 1= :0 Aug 24 09:08:40 applej kernel: pf: wire key attach failed on vlan42: 112 in= wire: 141.84.149.1 224.0.0.18 1:0, existing:=20 Aug 24 09:08:40 applej kernel: 112 in wire: 141.84.149.1 224.0.0.18 stack: = 141.84.149.1 224.0.0.18 1:0 Aug 24 09:08:40 applej kernel:=20 Aug 24 09:08:40 applej kernel: pf_map_addr: src tracking maps pf_map_addr: = src tracking maps pf: wire key attach failed on vlan250: 172.23.100.108pf: = OK ICMP 3:10 112 Aug 24 09:08:40 applej kernel: 172.23.106.184 in to wire: to 10.153.101.7= 3141.84.149.18 224.0.0.18141.84.149.18 1:0141.84.149.198, existing:=20 Aug 24 09:08:40 applej kernel: 112 in Aug 24 09:08:40 applej kernel: pf_map_addr: src tracking maps=20 Aug 24 09:08:40 applej kernel:=20 Aug 24 09:08:40 applej kernel: wire: -> 10.153.101.73172.23.106.58 to 224= .0.0.18141.84.149.19 stack:=20 Aug 24 09:08:40 applej kernel: 10.153.101.73 Aug 24 09:08:40 applej kernel: 77.7.121.109 state: 224.0.0.18TCP 1:0 Aug 24 09:08:40 applej kernel: out wire: 141.84.149.198:80 77.7.121.109:525= 21 stack:=20 Aug 24 09:08:40 applej kernel: - [lo=3D2750816253 high=3D2750829820 win=3D2= 56 modulator=3D0 wscale=3D8] [lo=3D2902946358 high=3D2903011894 win=3D106 m= odulator=3D0 Aug 24 09:08:40 applej kernel: wscale=3D7] 7:9 seq=3D2750816252 Aug 24 09:08:40 applej kernel: pf: OK ICMP 3:10 141.84.149.198 -> 77.7.121.= 109 state: TCP in wire: 77.7.121.109:52521 Aug 24 09:08:40 applej kernel: 141.84.149.198:80 stack: - [lo=3D2750816253 = high=3D2750829820 win=3D256 modulator=3D0 Aug 24 09:08:40 applej kernel: wscale=3D8] [lo=3D2902946358 high=3D29030118= 94 win=3D106 modulator=3D0 wscale=3D7] 7:9 seq=3D2750816252=20 .... Aug 24 09:09:11 applej kernel: 4:7:80:1433 seq=3D1025128833 Aug 24 09:09:11 applej kernel: 141.84.149.19:47806 stack: 10.110.32.129-:5= 0143 [lo=3D3790813207 high=3D3790831127 win=3D14600 modulator=3D0 stack: w= scale=3D8-] [lo=3D3961092366 high=3D3962143087 win=3D4106 modulator=3D0 [lo= =3D155 1251521 high=3D1554989121 win=3D17898 modulator=3D0 wscale=3D8 wscale=3D8]]= [lo=3D1005669588 high=3D1006720647 win=3D4106 modulator=3D0 4:2 wscale=3D8= seq=3D1551251520 Aug 24 09:09:11 applej kernel: ] 4:4 seq=3D1005669587 Aug 24 09:09:11 applej kernel: pf: OK ICMP 3:1 172.23.254.250pf: OK ICMP 3:= 1 -> 10.110.32.220172.23.106.87 -> state: 10.110.32.193TCP state: inTCP = wire: out172.23.106.87 wire: :5017310.110.32.193 :1433141.84 .149.21 :844310.110.32.130 stack: :54454172.23.106.87 stack: :50173- [lo= =3D2704530546 high=3D2705581267 win=3D4106 modulator=3D0141.84.149.18 wscal= e=3D8:8443] [lo=3D521335749 high=3D521353924 win=3D256 modulator=3D0 [lo=3D= 19884644 01 high=3D1989515460 win=3D4106 modulator=3D0 wscale=3D8 wscale=3D8]] [lo= =3D3203135938 high=3D3203200962 win=3D71 modulator=3D0 4:4 wscale=3D8 seq= =3D1988464400 Aug 24 09:09:11 applej kernel: ] 7:4 seq=3D521335748 Aug 24 09:09:11 applej kernel: pf: OK ICMP 3:1 pf: OK ICMP 3:1 pf: OK ICMP = 3:1 pf: OK ICMP 3:1 pf: OK ICMP 3:1 pf: OK ICMP 3:1 172.23.254.250172.23.25= 4.25010.110.32.220 -> -> 172.23.106.8710.110.32.193 state: =20 state: TCPTCP in out wire: wire: 172.23.106.8710.110.32.193:50168:1433 14= 1.84.149.2110.110.32.129:8443:50175 stack: stack: 172.23.106.87-:50168 [lo= =3D1504649300 high=3D1505700021 win=3D4106 modulator=3D0 wscale=3D8141 .84.149.18]:8443 [lo=3D723475428 high=3D724526487 win=3D4106 modulator=3D0 = [lo=3D975354116 high=3D975374851 win=3D256 modulator=3D0 wscale=3D8 wscale= =3D8]] 4:4 [lo=3D4251253613 high=3D4251317869 win=3D81 modulator=3D0 seq=3D= 723475427 Aug 24 09:09:11 applej kernel: wscale=3D8] 7:4 seq=3D975354115 Aug 24 09:09:11 applej kernel: 141.84.44.211 -> 172.23.254.250217.86.168.23= 6 -> state: 172.23.3.29TCP state: inTCP wire: 141.84.44.211217.86.168.236= -> :5098310.180.10.209 state: 141.84.149.198TCP:80 in stack : wire: -10.180.10.209 [lo=3D3709359805 high=3D3709359806 win=3D8192 modul= ator=3D0 in:49223] [lo=3D0 high=3D8192 win=3D1 modulator=3D0141.84.149.244= ]:443 2:0 stack: seq=3D3709359804 Aug 24 09:09:11 applej kernel: - [lo=3D827930303 high=3D827944894 win=3D67 = modulator=3D0 wire: wscale=3D8172.23.3.29]:35196 [lo=3D1796383793 high=3D1= 796400177 win=3D114 modulator=3D0 wscale=3D710.153.101.108]:6556 10:10 sta= ck: s eq=3D827930303 Aug 24 09:09:11 applej kernel: - [lo=3D1991477221 high=3D1991477222 win=3D1= 4600 modulator=3D0 -> ]172.23.101.172 [lo=3D0 high=3D14600 win=3D1 modulato= r=3D0 state: ]TCP 2:0 in seq=3D1991477220 Aug 24 09:09:11 applej kernel: wire: 172.23.101.172:49613 141.84.149.21:808= 0 stack: 172.23.101.172:49613 141.84.149.19:8080 [lo=3D3559829126 high=3D35= 59829127 win=3D8192 modulator=3D0] [lo=3D0 high=3D8192 win=3D1 modulator=3D= 0] 2:0 seq=3D3559829125 Aug 24 09:09:11 applej kernel: pf: OK ICMP 3:1 pf: OK ICMP 3:1 10.110.32.22= 0141.84.44.211 -> pf_map_addr: src tracking maps -> 172.23.101.8710.110.32= .193 to 115.248.50.20141.84.149.19 state:=20 Aug 24 09:09:11 applej kernel: state: TCPTCP out in wire: wire: 10.110.32.= 193115.248.50.20:1433:50426 10.110.32.130141.84.149.229:54455:80 stack: s= tack: -- [lo=3D201188016 high=3D202238737 win=3D513 modulator=3D0 [lo =3D2382456191 high=3D2382456193 win=3D2048 modulator=3D0 wscale=3D8]] [lo= =3D0 high=3D1 win=3D1 modulator=3D0 [lo=3D3935828538 high=3D3935959789 win= =3D4106 modulator=3D0] wscale=3D8 2:0] seq=3D2382456191 Aug 24 09:09:11 applej kernel: 4:4 seq=3D3935828537 Sorry if this is too much but I really can' t tell which of these messages= could help. Perhaps there isn't any useful information at all. To me it se= ems it's just a side effect of the actual problem which I am unable to see. Maybe somebody has an idea about what's going wrong and what to investigate= . Or theories right now are: * issues with pfsync (messages complaining about states, but unlikely, it'= s normally working just fine) * some lack of ressources (yet, which?) * race condition inside the kernel (there are 24 CPU cores) We have not yet been able to reproduce it in a lab environment.=20 Right now, we work around this by putting the master into the backup state = (using demotion) before loading the ruleset and changing back to master sta= te after the reload. This is working fine. But it kills, of course, the IPS= ec-SAs. But that's something we can live with right now.=20 Viele Gr=FC=DFe=20 Andrej Kolontai Ludwig-Maximilians-Universitaet Muenchen Ref. VI.4 (IT-Sicherheit & Verzeichnisdienste)=20 Martiusstrasse 4 / 207 80802 Muenchen phone +49 (0)89 2180-3815 email mailto:andrej.kolontai@verwaltung.uni-muenchen.de web http://www.uni-muenchen.de/zuv/it/ From owner-freebsd-pf@freebsd.org Tue Aug 25 16:13:39 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3CEB999E6F5 for ; Tue, 25 Aug 2015 16:13:39 +0000 (UTC) (envelope-from s.tyshchenko@identika.pro) Received: from scale212.ru (scale212.ru [51.254.36.76]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A0F76B90 for ; Tue, 25 Aug 2015 16:13:38 +0000 (UTC) (envelope-from s.tyshchenko@identika.pro) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=scale212.ru; s=default; h=Content-Type:List-Unsubscribe:Message-ID:Sender:From:Date:MIME-Version:Subject:To; bh=+q18Zhx6yOah9PcbBkQRbqz5/myrOgnFDJh5TdQg0hI=; b=PJqlUctZjCoATr9SK0xVXnp1MWsGdimVfFgKdlstp2t3DQWuMI5Ywc4kJ6fnYWwvi0SzESpRQ5xB9U3ZJ7WxNF3UhKUlfnRdKXakyeSL+Vf/EfK+tISN+QVKv7fUu03cprL133P67sPNgakT4NjLDTrMYWEOqgCQj9CWZ9yNHD4=; Received: from root by scale212.ru with local (Exim 4.80) (envelope-from ) id 1ZUGqu-0002ge-Ae for freebsd-pf@freebsd.org; Tue, 25 Aug 2015 18:13:36 +0200 To: freebsd-pf@freebsd.org Subject: For you MIME-Version: 1.0 Date: Tue, 25 Aug 2015 18:13:36 +0200 From: Sergey Tyshchenko Sender: s.tyshchenko@identika.pro Message-ID: <241800456.26531@scale212.ru> X-Priority: 3 X-Mailer: scale212.ru mailer. Ver. 1.1. Precedence: bulk Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2015 16:13:39 -0000 Zm9yIHlvdQ0KCQkJDQoNCgkJCQ0KCQkJwqANCg0KCQkJwqANCg0KCQkJwqANCg0KCQkJwqANCg0K CQkJDQoJCQ0KCQkNCgkJCQ0KCQkJDQoJCQkNCgkJCUhlbGxvLCBNeSBuYW1lIGlzIFNlcmdleSwg SSBwcm9wb3NlIHlvdSBjb29wZXJhdGlvbi4gV2UgYXJlIGEgY3JlYXRpdmUgY29tcGFueSBpbiB0 aGUgZGV2ZWxvcG1lbnQgYW5kIGNyZWF0aW9uIG9mIHVuaXF1ZSBwcm9kdWN0cyBmb3IgZGVjb3Jh dGlvbiAtIHRvIGRlc2lnbiBidWlsZGluZ3MgSURFTlRJS0EuUFJPLiBXZScncmUgc3BlY2lhbGl6 ZWQgb24gZGVjb3JhdGlvbiBvZiBzaG9wcywgcGV0cm9sIHN0YXRpb24sIGNhZmUsIGZhc3QgZm9v ZCByZXN0YXVyYW50cywgYW5kIHJldGFpbCBmcmFuY2hpc2VzLldlIGFyZSBpbnRlcmVzdGVkIGF0 IGxvbmctdGVybSBjb29wZXJhdGlvbiB3aXRoIHlvdS4gV2UgcHJvcG9zZSBhIGdvb2QgZW52aXJv bm1lbnQgdG8gd29yayB3aXRoIHBhcnRuZXJzLkV4YW1wbGVzIG9mIG91ciB3b3JrIGh0dHAgOmh0 dHA6Ly9pZGVudGlrYS5wcm8vY291bnRlcl9saW5rL2NvdW50ZXIucGhwP2NsaWNrPXByZXNlbnRh dGlvbl9lbkkgcHJvcG9zZSBZb3UgdG8gYmVjb21lIG91ciBwYXJ0bmVyIGluIHlvdXIgYXJlYS5M ZXQgdXMga25vdyBhYm91dCB5b3VyIGRlY2lzaW9uDQoJCQkNCg0KCQkJDQoJCQnCoA0KDQoJCQnC oA0KDQoJCQnCoA0KDQoJCQnCoA0KDQoJCQkNCgkJDQoJCQ0KCQkJDQoJCQkNCgkJCQ0KCQkJDQoJ CQkNCg0KCQkJDQoJCQnCoA0KDQoJCQnCoA0KDQoJCQnCoA0KDQoJCQnCoA0KDQoJCQkNCgkJDQoJ CQ0KCQkJDQoJCQkNCgkJCQ0KCQkJDQoJCQkNCg0KCQkJDQoJCQnCoA0KDQoJCQnCoA0KDQoJCQnC oA0KDQoJCQnCoA0KDQoJCQkNCgkJDQoJCQ0KCQkJDQoJCQkNCgkJCQ0KCQkJIEV4YW1wbGVzIG9m IG91ciB3b3JrIGh0dHAgOmh0dHA6Ly9pZGVudGlrYS5wcm8vY291bnRlcl9saW5rL2NvdW50ZXIu cGhwP2NsaWNrPXByZXNlbnRhdGlvbl9lbg0KCQkJDQoNCgkJCQ0KCQkJwqANCg0KCQkJwqANCg0K CQkJwqANCg0KCQkJwqANCg0KCQkJDQoJCQ0KCQkNCgkJCQ0KCQkJDQoJCQkNCgkJCVNlcmdleSBU eXNoY2hlbmtvQ0VPIHwgSURFTlRJS0EuUFJPVmliZXI6ICszODA1MDU1NjY5NjUgfCBXaGF0c0Fw cDogKzM4MDUwNTU2Njk2NVNreXBlOiB0LnNlcmdleS5tcy50eXNoY2hlbmtvQGlkZW50aWthLnBy byB8IHd3dy5pZGVudGlrYS5wcm8wMzA0MCB8IEdvbG9zaWl2c2t5aSBBdmUuIDcwIHwgb2ZmaWNl IDUwMiB8IEtpZXYgDQoJCQkNCg0KCQkJDQoJCQnCoA0KDQoJCQnCoA0KDQoJCQnCoA0KDQoJCQnC oA== From owner-freebsd-pf@freebsd.org Tue Aug 25 17:57:00 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 860709C3446; Tue, 25 Aug 2015 17:57:00 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: from mail-yk0-x22e.google.com (mail-yk0-x22e.google.com [IPv6:2607:f8b0:4002:c07::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4276D2DF; Tue, 25 Aug 2015 17:57:00 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: by ykdt205 with SMTP id t205so164624176ykd.1; Tue, 25 Aug 2015 10:56:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=CvjsxtEFacw8stFLx6NKiIJjAKQRFcY6zzFmW5xSbxA=; b=ILZIBFp8BM8woAs6CYESyc2U4T7Gtql+9Nj+KqI0GHNgG3XMNECjEV8lClodpImqF1 jIf7dEFrxx0SI/7s89cxuSO/loIG2eUtue+UQPDF3W6RL+kjKzBMSQtovFHWr/qrhcaP aHAwfe0j3fDMdJ8dzJQqpO8yOGgrGTl6+wpppTscvQS5DJeG7JTlwXOpskNWpPJ7d73C Q4LE1tCdONBzQ0fYwACKFAZKaXAhji+tqSTvYhAO05zKkfx49qPhUkpzCqS736gmDASy WMLputUBiCD6z44EiwTEOyv5SpPDj+0BZbHXXN/mqc7k3PBXT2/uQwjc6zqFDNhg8NN+ pz/w== MIME-Version: 1.0 X-Received: by 10.170.70.213 with SMTP id m204mr39637810ykm.112.1440525419493; Tue, 25 Aug 2015 10:56:59 -0700 (PDT) Received: by 10.129.73.205 with HTTP; Tue, 25 Aug 2015 10:56:59 -0700 (PDT) In-Reply-To: <20150823150957.GK48727@vega.codepro.be> References: <20150823150957.GK48727@vega.codepro.be> Date: Tue, 25 Aug 2015 19:56:59 +0200 Message-ID: Subject: Re: Near-term pf plans From: =?UTF-8?Q?Ermal_Lu=C3=A7i?= To: Kristof Provost Cc: freebsd-net , "freebsd-pf@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2015 17:57:00 -0000 On Sun, Aug 23, 2015 at 5:09 PM, Kristof Provost wrote: > Hi, > > Some of you may have noticed that I fixed a couple of pf issues (or in > some cases broke things. Sorry Allan.) recently. > > Here's a quick list of my current priorities: > > - PR 127042, 202178: > This is a panic when an interface an ifgroup have the same name. > There's a fix for the immediate problem in > https://reviews.freebsd.org/D3435 > > I'm inclined to say that ifgroups and interfaces should share a > namespace. That would certainly help pf, because it uses both > interchangeably, which just doesn't work unless they're in the same > namespace. That affects more in the network stack though, so I'd like > opinions for people with more experience with ifgroups. > The solution is easy the scenario of interface name and group name should not be allowed. I do not see the use case for this to be allowed at all its just confusion in general in pf(4). > > - PR 202351 > This is a panic after ip6 reassembly in pf. We set the rcvif to NULL > when refragmenting. That seems to go OK execpt when we're refragmenting > broadcast/multicast packets in the forwarding path. It's not at all > clear to me how that could happen. > > - Removal of frag drop / frag crop support in pf. > pf has two (or really three, but crop and drop mode are very similar) > ways to handle fragmented packets. I've recently extended the > 'reassemble' mode to also work for ip6. > The crop/drop modes really only verify that fragments make sense > (i.e. don't claim to deliver data beyond the last fragment, or duplicate > data). > I'd like to remove support for the crop/drop modes for two reasons: > * the code is relatively large and complex. I've already run into a > couple of problems related to it. > * It doesn't do what users expect. Enabling scrub fragment crop > doesn't actually mean pf will filter on the entire fragment. You > still end up having to let all fragments pass through the > firewall. > There's a patch to do just that here: > https://reviews.freebsd.org/D3466 > > While those provide more security protection they do not always work as advertised so i agree on their removal. > - PR 198868, 193579 (TSO issues) > pf has issues on certain network interfaces with TSO enabled. The > most important of these are amazon VMs. > I believe the problem is that pf tries to work with packets with full > checksums. Usually output packets have pseudo-header checksums in the IP > and TCP headers. pf doesn't know about those. It always tries to do > updates on checksums assuming there's a full checksum. (Which is the > case, pf always calculates a full checksum in pf_check_out()) > > I believe the fix for this issue is to have pf be aware of the > pseudo-header checksums. The type of checksum can be determined from the > CSUM_TCP and CSUM_TCP_IPV6 flags. Whenever pf changes a packet header it > will have to check for those flags to figure out if the checksum should > be updated or not. > > I would be inclined to say that the real solution is not check packets generated from the host, otherwise you will have to go into complicated checksum handling which might not be worth it. I know Open has done some work on checksum handling altogether which might be a good reason to go and look there first. > - PR 188511 > divert-reply implementation for pf > I need to review the use case and implementation of the work done by > PiBa.NL.dev@gmail.com > > I am aware of the patch just do not think this will be a good candidate for inclusion. First divert sockets are slow and should not be advertised for use unless properly fixed. Second the better implementation in general, which is on my roadmap, is convert the route-to/reply-to keywords to the same implementation as ipfw fwd and that will reduce code complexity and provide proper functionality for the scenario the patch is written. > - PR 172648 > This bug started out with an issue with TCP reassembly, but I've not > been able to reproduce that. There is a problem with rdr targets for > ipv6 though. > > With rdr to ::1 we fail the scope check in ip6_input() (right after the > pfil hook) because we have a packet to localhost with a > m->m_pkthdr.rcvif which is not a loopback interface. > That can be fixed by having pf rewrite the rcvif, but that'd > special-case rdr to ::1. > > We've got a similar problem for the reply. There we've got a packet from > ::1 to something else. This fails the scope lookup too. > In essence the problem is that we've already made the routing decision > before pf gets the chance to rewrite the destination address. > > I'm not quite sure how to fix this though. > I will try to look back at this but as a general rule look first at Open if they have already fixed this. IPv4 code has some security belts on such packets in pf(4) code to avoid doing the wrong thing. > > If anyone has any comments or questions, or disagrees with any of the > above, please let me know. > > If anyone knows of critical problems not on this list please let me > know, and I'll see what I can do. > > Also, pf is currently an (unpaid) side project for me. I'm reasonably > limited in the amount of time I can invest in it. > > Regards, > Kristof > _______________________________________________ > freebsd-pf@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-pf > To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" > -- Ermal From owner-freebsd-pf@freebsd.org Wed Aug 26 10:09:19 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 30A009C3132 for ; Wed, 26 Aug 2015 10:09:19 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-lb0-f172.google.com (mail-lb0-f172.google.com [209.85.217.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 89B101F32 for ; Wed, 26 Aug 2015 10:09:17 +0000 (UTC) (envelope-from ml@my.gd) Received: by lbbpu9 with SMTP id pu9so116864196lbb.3 for ; Wed, 26 Aug 2015 03:09:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=hnz4o3GPPp3Rff/vOOmv2+PnCEz5memcTs4DDS84cX8=; b=btSFolqyhLBSY0FgcPjHNsG+cw1gRRtgiRAg0dNQXDfQenXm+GPV+TaJUOlJ4ZLO1I nz3u5KSeMxx1OzaEX8vPPBROb/RTfCR/d5QZ83KKUCQ8oFsn7E94MRF7XGJ921J47CRQ JZc5UuXxo0mrIWVl5v8JtpnO6xzcEeDjsiwF9lFTI5wyVxMtk4XeRi45InyRQ5P2R+Tg 0h1JdymmORiXfgDsd6msgozQOSsnFzsT7ObTlxmaGJwi86HY/8E52AImoiad9GPBNoIY 2X0iMPDSwwiPTrgcBs8GjUShHxybVuPUFeQToaX+n9fv4eJ2I6TM2bDgnSNdEcibNG2Q KB9g== X-Gm-Message-State: ALoCoQlOGecPIoCa3mBJZzzjOEhwVeNnhBCJbsT++CVCk1DlIizwe1+RPDzT1kQj/BcSIWNug4NG MIME-Version: 1.0 X-Received: by 10.152.22.4 with SMTP id z4mr30359566lae.81.1440583750166; Wed, 26 Aug 2015 03:09:10 -0700 (PDT) Received: by 10.112.60.34 with HTTP; Wed, 26 Aug 2015 03:09:10 -0700 (PDT) In-Reply-To: References: Date: Wed, 26 Aug 2015 12:09:10 +0200 Message-ID: Subject: Re: Machine freezes when loading pf ruleset From: Damien Fleuriot To: Kolontai Andrej Cc: "freebsd-pf@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 10:09:19 -0000 On 25 August 2015 at 17:55, Kolontai Andrej < Andrej.Kolontai@verwaltung.uni-muenchen.de> wrote: > Hello, > > I'm new to this list and I hope it's the right place to ask. > > We have highly utilized installation of two FreeBSD-machines running > 10.1-RELEASE, pf and carp. There are about 50 networks (some via vlan, so= me > ipsec) connected to them, usually about 50000 pf states, about 1500 rules > and traffic sometimes hitting 1 gbit/s or more. > > When we load the ruleset on the active machine it sometimes freezes. It > occurs more often as our installation grows. Sometimes it freezes > completely, which is actually good as the backup machine takes over. But = at > other times it's still some kind of alive but unbearably slow (ping is > about 40 s instead of the usual 0.2 ms), the backup machine does not take > over and people start to get angry. > The crashed machine usually recovers after a couple of minutes. > > We've been trying to resolve this for some months now and are running out > of ideas. We do believe that something is wrong with or setup but have no > clue what it could be. > > When it happens, the syslogs show tons of messages like these: > > Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps > 172.23.101.87 to 141.84.149.18 > Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps > 172.23.101.87 to 141.84.149.18 > Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps > 172.23.107.206 to 141.84.149.18 > Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps > 172.23.101.87 to 141.84.149.18 > Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps > 172.23.107.206 to 141.84.149.18 > Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps > 172.23.101.87 to 141.84.149.18 > Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps > 172.23.106.171 to 141.84.149.18 > Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps > 172.23.106.171 to 141.84.149.18 > Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps > 172.23.106.72 to 141.84.149.18 > Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps > 172.23.106.58 to 141.84.149.19 > Aug 24 09:08:39 applej last message repeated 2 times > Aug 24 09:08:39 applej kernel: pf: BAD state: TCP in wire: > 10.200.108.141:49202 172.23.2.92:8192 stack: - [lo=3D3482585550 > high=3D3482585552 win=3D8192 modulator=3D0pf_map_addr: src tracking maps > ]172.23.106.72 [lo=3D0 hi > gh=3D1 win=3D1 modulator=3D0 to ]141.84.149.18 2:0 > Aug 24 09:08:39 applej kernel: S seq=3D3503222902 (3503222902) ack=3D0 le= n=3D0 > ackskew=3D0 pkts=3D1:1 dir=3Din,fwd > Aug 24 09:08:39 applej kernel: pf: State failure on: 1 | 5 > Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps > 172.23.106.171 to 141.84.149.18 > Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps > 172.23.107.20 to 141.84.149.18 > Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps > 172.23.101.87 to 141.84.149.18 > Aug 24 09:08:39 applej kernel: pf_map_addr: src tracking maps pf_map_addr= : > src tracking maps pf_map_addr: src tracking maps > 172.23.106.171172.23.106.171 to 172.23.106.171141.84.149.18 to to > Aug 24 09:08:39 applej kernel: 141.84.149.18141.84.149.18 > Aug 24 09:08:39 applej kernel: > Aug 24 09:08:40 applej kernel: pf_map_addr: src tracking maps pf_map_addr= : > src tracking maps 172.23.106.171172.23.101.87 to > Aug 24 09:08:40 applej kernel: to 141.84.149.18141.84.149.18 > Aug 24 09:08:40 applej kernel: > Aug 24 09:08:40 applej kernel: pf: OK ICMP 3:10 141.84.149.249 -> > 10.153.101.239 state: TCP in wire: 141.84.149.249:37677 10.153.101.239 > Aug 24 09:08:40 applej kernel: :80 stack: - [lo=3D4171141930 high=3D41711= 59337 > win=3D106 modulator=3D0 wscale=3D7] [lo=3D2653133771 high=3D2653147339 wi= n=3D136 > modulator=3D0 wscale=3D7] 10:10 seq=3D2653133771 > Aug 24 09:08:40 applej kernel: pf: OK ICMP 3:10 141.84.149.249 -> > 10.153.101.239 state: TCP out wire: 10.153.101.239:80 141.84.149.249 > Aug 24 09:08:40 applej kernel: :37677 stack: - [lo=3D4171141930 > high=3D4171159337 win=3D106 modulator=3D0 wscale=3D7] [lo=3D2653133771 > high=3D2653147339 win=3D136 modulator=3D0 wscale=3D7] 10:10 seq=3D2653133= 771 > Aug 24 09:08:40 applej kernel: pf_map_addr: src tracking maps pf_map_addr= : > src tracking maps 172.23.106.184pf: wire key attach failed on vlan42: > 172.23.106.165pf_map_addr: src tracking maps > Aug 24 09:08:40 applej kernel: 112 to in141.84.149.18 wire: 172.23.106.5= 8 > Aug 24 09:08:40 applej kernel: 141.84.149.27 > Aug 24 09:08:40 applej kernel: to 224.0.0.18 to 1:0141.84.149.18, > existing: > Aug 24 09:08:40 applej kernel: 112141.84.149.19 in > Aug 24 09:08:40 applej kernel: wire: > Aug 24 09:08:40 applej kernel: 141.84.149.27 > Aug 24 09:08:40 applej kernel: 224.0.0.18 stack: 141.84.149.27 224.0.0.18 > 1:0 > Aug 24 09:08:40 applej kernel: pf: wire key attach failed on vlan42: 112 > in wire: 141.84.149.1 224.0.0.18 1:0, existing: > Aug 24 09:08:40 applej kernel: 112 in wire: 141.84.149.1 224.0.0.18 stack= : > 141.84.149.1 224.0.0.18 1:0 > Aug 24 09:08:40 applej kernel: > Aug 24 09:08:40 applej kernel: pf_map_addr: src tracking maps pf_map_addr= : > src tracking maps pf: wire key attach failed on vlan250: 172.23.100.108pf= : > OK ICMP 3:10 112 > Aug 24 09:08:40 applej kernel: 172.23.106.184 in to wire: to > 10.153.101.73141.84.149.18 224.0.0.18141.84.149.18 1:0141.84.149.198, > existing: > Aug 24 09:08:40 applej kernel: 112 in > Aug 24 09:08:40 applej kernel: pf_map_addr: src tracking maps > Aug 24 09:08:40 applej kernel: > Aug 24 09:08:40 applej kernel: wire: -> 10.153.101.73172.23.106.58 to > 224.0.0.18141.84.149.19 stack: > Aug 24 09:08:40 applej kernel: 10.153.101.73 > Aug 24 09:08:40 applej kernel: 77.7.121.109 state: 224.0.0.18TCP 1:0 > Aug 24 09:08:40 applej kernel: out wire: 141.84.149.198:80 > 77.7.121.109:52521 stack: > Aug 24 09:08:40 applej kernel: - [lo=3D2750816253 high=3D2750829820 win= =3D256 > modulator=3D0 wscale=3D8] [lo=3D2902946358 high=3D2903011894 win=3D106 mo= dulator=3D0 > Aug 24 09:08:40 applej kernel: wscale=3D7] 7:9 seq=3D2750816252 > Aug 24 09:08:40 applej kernel: pf: OK ICMP 3:10 141.84.149.198 -> > 77.7.121.109 state: TCP in wire: 77.7.121.109:52521 > Aug 24 09:08:40 applej kernel: 141.84.149.198:80 stack: - [lo=3D275081625= 3 > high=3D2750829820 win=3D256 modulator=3D0 > Aug 24 09:08:40 applej kernel: wscale=3D8] [lo=3D2902946358 high=3D290301= 1894 > win=3D106 modulator=3D0 wscale=3D7] 7:9 seq=3D2750816252 > .... > Aug 24 09:09:11 applej kernel: 4:7:80:1433 seq=3D1025128833 > Aug 24 09:09:11 applej kernel: 141.84.149.19:47806 stack: > 10.110.32.129-:50143 [lo=3D3790813207 high=3D3790831127 win=3D14600 modul= ator=3D0 > stack: wscale=3D8-] [lo=3D3961092366 high=3D3962143087 win=3D4106 modula= tor=3D0 > [lo=3D155 > 1251521 high=3D1554989121 win=3D17898 modulator=3D0 wscale=3D8 wscale=3D8= ]] > [lo=3D1005669588 high=3D1006720647 win=3D4106 modulator=3D0 4:2 wscale=3D= 8 > seq=3D1551251520 > Aug 24 09:09:11 applej kernel: ] 4:4 seq=3D1005669587 > Aug 24 09:09:11 applej kernel: pf: OK ICMP 3:1 172.23.254.250pf: OK ICMP > 3:1 -> 10.110.32.220172.23.106.87 -> state: 10.110.32.193TCP state: > inTCP wire: out172.23.106.87 wire: :5017310.110.32.193 :1433141.84 > .149.21 :844310.110.32.130 stack: :54454172.23.106.87 stack: :50173- > [lo=3D2704530546 high=3D2705581267 win=3D4106 modulator=3D0141.84.149.18 > wscale=3D8:8443] [lo=3D521335749 high=3D521353924 win=3D256 modulator=3D0= [lo=3D19884644 > 01 high=3D1989515460 win=3D4106 modulator=3D0 wscale=3D8 wscale=3D8]] [lo= =3D3203135938 > high=3D3203200962 win=3D71 modulator=3D0 4:4 wscale=3D8 seq=3D1988464400 > Aug 24 09:09:11 applej kernel: ] 7:4 seq=3D521335748 > Aug 24 09:09:11 applej kernel: pf: OK ICMP 3:1 pf: OK ICMP 3:1 pf: OK ICM= P > 3:1 pf: OK ICMP 3:1 pf: OK ICMP 3:1 pf: OK ICMP 3:1 > 172.23.254.250172.23.254.25010.110.32.220 -> -> 172.23.106.8710.110.32.1= 93 > state: > state: TCPTCP in out wire: wire: 172.23.106.8710.110.32.193:50168:1433 > 141.84.149.2110.110.32.129:8443:50175 stack: stack: 172.23.106.87-:50168 > [lo=3D1504649300 high=3D1505700021 win=3D4106 modulator=3D0 wscale=3D814= 1 > .84.149.18]:8443 [lo=3D723475428 high=3D724526487 win=3D4106 modulator=3D= 0 > [lo=3D975354116 high=3D975374851 win=3D256 modulator=3D0 wscale=3D8 wscal= e=3D8]] 4:4 > [lo=3D4251253613 high=3D4251317869 win=3D81 modulator=3D0 seq=3D723475427 > Aug 24 09:09:11 applej kernel: wscale=3D8] 7:4 seq=3D975354115 > Aug 24 09:09:11 applej kernel: 141.84.44.211 -> > 172.23.254.250217.86.168.236 -> state: 172.23.3.29TCP state: inTCP wire= : > 141.84.44.211217.86.168.236 -> :5098310.180.10.209 state: > 141.84.149.198TCP:80 in stack > : wire: -10.180.10.209 [lo=3D3709359805 high=3D3709359806 win=3D8192 > modulator=3D0 in:49223] [lo=3D0 high=3D8192 win=3D1 modulator=3D0141.84.= 149.244]:443 > 2:0 stack: seq=3D3709359804 > Aug 24 09:09:11 applej kernel: - [lo=3D827930303 high=3D827944894 win=3D6= 7 > modulator=3D0 wire: wscale=3D8172.23.3.29]:35196 [lo=3D1796383793 > high=3D1796400177 win=3D114 modulator=3D0 wscale=3D710.153.101.108]:6556= 10:10 > stack: s > eq=3D827930303 > Aug 24 09:09:11 applej kernel: - [lo=3D1991477221 high=3D1991477222 win= =3D14600 > modulator=3D0 -> ]172.23.101.172 [lo=3D0 high=3D14600 win=3D1 modulator= =3D0 state: > ]TCP 2:0 in seq=3D1991477220 > Aug 24 09:09:11 applej kernel: wire: 172.23.101.172:49613 > 141.84.149.21:8080 stack: 172.23.101.172:49613 141.84.149.19:8080 > [lo=3D3559829126 high=3D3559829127 win=3D8192 modulator=3D0] [lo=3D0 high= =3D8192 win=3D1 > modulator=3D0] > 2:0 seq=3D3559829125 > Aug 24 09:09:11 applej kernel: pf: OK ICMP 3:1 pf: OK ICMP 3:1 > 10.110.32.220141.84.44.211 -> pf_map_addr: src tracking maps -> > 172.23.101.8710.110.32.193 to 115.248.50.20141.84.149.19 state: > Aug 24 09:09:11 applej kernel: state: TCPTCP out in wire: wire: > 10.110.32.193115.248.50.20:1433:50426 10.110.32.130141.84.149.229:54455:= 80 > stack: stack: -- [lo=3D201188016 high=3D202238737 win=3D513 modulator=3D= 0 [lo > =3D2382456191 high=3D2382456193 win=3D2048 modulator=3D0 wscale=3D8]] [lo= =3D0 high=3D1 > win=3D1 modulator=3D0 [lo=3D3935828538 high=3D3935959789 win=3D4106 modul= ator=3D0] > wscale=3D8 2:0] seq=3D2382456191 > Aug 24 09:09:11 applej kernel: 4:4 seq=3D3935828537 > > Sorry if this is too much but I really can' t tell which of these > messages could help. Perhaps there isn't any useful information at all. T= o > me it seems it's just a side effect of the actual problem which I am unab= le > to see. > > Maybe somebody has an idea about what's going wrong and what to > investigate. Or theories right now are: > * issues with pfsync (messages complaining about states, but unlikely, > it's normally working just fine) > * some lack of ressources (yet, which?) > * race condition inside the kernel (there are 24 CPU cores) > > We have not yet been able to reproduce it in a lab environment. > > Right now, we work around this by putting the master into the backup stat= e > (using demotion) before loading the ruleset and changing back to master > state after the reload. This is working fine. But it kills, of course, th= e > IPSec-SAs. But that's something we can live with right now. > > > Viele Gr=C3=BC=C3=9Fe > Andrej Kolontai > > Ludwig-Maximilians-Universitaet Muenchen > Ref. VI.4 (IT-Sicherheit & Verzeichnisdienste) > Martiusstrasse 4 / 207 > 80802 Muenchen > > phone +49 (0)89 2180-3815 > email mailto:andrej.kolontai@verwaltung.uni-muenchen.de > web http://www.uni-muenchen.de/zuv/it/ > > 1.5k rules seems like a lot for PF to handle. Is that 1.5k rules you've written in the conf, or 1.5k rules from `pfctl -sr | wc -l' ? If that's merely your number of configuration lines, that may very well expand to many more rules in reality. For example "pass in quick on { $vlan2 $vlan3 } inet proto { $tcp $udp } from to port 53" will expand into 4 actual rules. I would suggest you find a way to drastically lower that. You may also wish to ensure : - you're using, if at all possible, a *dedicated* pfsync link (like a cable on a dedicated interface between the boxes) - you've got enough RAM to store your PF states As another, *unrelated* performance enhancement, make sure to use the "quick" keyword whenever applicable. From owner-freebsd-pf@freebsd.org Wed Aug 26 11:43:53 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 180509C3A37; Wed, 26 Aug 2015 11:43:53 +0000 (UTC) (envelope-from kp@vega.codepro.be) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A2D121469; Wed, 26 Aug 2015 11:43:52 +0000 (UTC) (envelope-from kp@vega.codepro.be) Received: from vega.codepro.be (unknown [172.16.1.3]) by venus.codepro.be (Postfix) with ESMTP id 3FDEBB18F; Wed, 26 Aug 2015 13:43:48 +0200 (CEST) Received: by vega.codepro.be (Postfix, from userid 1001) id 3B6DF54C6; Wed, 26 Aug 2015 13:43:48 +0200 (CEST) Date: Wed, 26 Aug 2015 13:43:48 +0200 From: Kristof Provost To: Ermal =?utf-8?B?THXDp2k=?= Cc: freebsd-net , "freebsd-pf@freebsd.org" Subject: Re: Near-term pf plans Message-ID: <20150826114348.GA3433@vega.codepro.be> References: <20150823150957.GK48727@vega.codepro.be> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Checked-By-NSA: Probably User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 11:43:53 -0000 On 2015-08-25 19:56:59 (+0200), Ermal Luçi wrote: > On Sun, Aug 23, 2015 at 5:09 PM, Kristof Provost wrote: > > > I'm inclined to say that ifgroups and interfaces should share a > > namespace. That would certainly help pf, because it uses both > > interchangeably, which just doesn't work unless they're in the same > > namespace. That affects more in the network stack though, so I'd like > > opinions for people with more experience with ifgroups. > > > > > The solution is easy the scenario of interface name and group name should > not be allowed. > I do not see the use case for this to be allowed at all its just confusion > in general in pf(4). > Agreed. I'll see about putting together a patch to do that. > > > > - Removal of frag drop / frag crop support in pf. ... > While those provide more security protection they do not always work as > advertised so i agree on their removal. > Okay, I'll get the patch committed soon. > > - PR 198868, 193579 (TSO issues) > > pf has issues on certain network interfaces with TSO enabled. The > > most important of these are amazon VMs. > > I believe the problem is that pf tries to work with packets with full > > checksums. Usually output packets have pseudo-header checksums in the IP > > and TCP headers. pf doesn't know about those. It always tries to do > > updates on checksums assuming there's a full checksum. (Which is the > > case, pf always calculates a full checksum in pf_check_out()) > > > > I believe the fix for this issue is to have pf be aware of the > > pseudo-header checksums. The type of checksum can be determined from the > > CSUM_TCP and CSUM_TCP_IPV6 flags. Whenever pf changes a packet header it > > will have to check for those flags to figure out if the checksum should > > be updated or not. > > > > > I would be inclined to say that the real solution is not check packets > generated from the host, This also applies to forwarded packets. Think of things like NAT or port rewriting. We have to change IP addresses for nat, and thus also update checksums. > otherwise you will have to go into complicated checksum handling which > might not be worth it. We should end up with slightly more complicated checksum code (because sometimes we'll update and sometimes we won't), but I expect it to actually be faster. Right now we always calculate a full checksum on outbound packets. In the new case we'll either not touch the checksum, or only calculate updates (depending on scenario). > I know Open has done some work on checksum handling altogether which might > be a good reason > to go and look there first. > Right, I'll check. > > - PR 172648 > > This bug started out with an issue with TCP reassembly, but I've not > > been able to reproduce that. There is a problem with rdr targets for > > ipv6 though. > ... > I will try to look back at this but as a general rule look first at Open if > they have already fixed this. > IPv4 code has some security belts on such packets in pf(4) code to avoid > doing the wrong thing. > I think their checks in ip6_output() are less strict than ours. Regards, Kristof From owner-freebsd-pf@freebsd.org Wed Aug 26 12:50:34 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BCB0B99AF84; Wed, 26 Aug 2015 12:50:34 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: from mail-yk0-x22c.google.com (mail-yk0-x22c.google.com [IPv6:2607:f8b0:4002:c07::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 90AB961; Wed, 26 Aug 2015 12:50:34 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: by ykbi184 with SMTP id i184so185328299ykb.2; Wed, 26 Aug 2015 05:50:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=IoOHX2RFm6A9u1Mg2d+ncn0hSoMLEMF7L8ZY+JBSTlk=; b=pfneAVQa+owfoM+AgSORBji2cQ4g2yXDI4T3HzVsEmmLvhm2i5ftyYicJS3e4CRIrR I8IUdobP8+FnGjve8bAELIQKAT9+LmblzA1R2+IQtNfRCWxJen/kJPrkD5OQ89KEg60S XWfdS9sqQr6sVLVSZxDAYITUj0lP+ldi0Xno8FDR5tR3rKH0k2A0Vfz9ltBiS2LuYVjm vO/EVD9BzDU/4J0MqAY850l2UqC3M9bhUZMTCKtGQlQWaUcJIVZ3lk8CxF0bD4rQNUmE taxwVXqJgiGr63rycApoa42xT3LuWT1iOAK9SX0F2YY+k+lK4Z4dlXpu9CsvDfMTVEX6 DHkg== MIME-Version: 1.0 X-Received: by 10.129.78.85 with SMTP id c82mr41079083ywb.162.1440593433587; Wed, 26 Aug 2015 05:50:33 -0700 (PDT) Received: by 10.129.73.205 with HTTP; Wed, 26 Aug 2015 05:50:33 -0700 (PDT) In-Reply-To: <20150826114348.GA3433@vega.codepro.be> References: <20150823150957.GK48727@vega.codepro.be> <20150826114348.GA3433@vega.codepro.be> Date: Wed, 26 Aug 2015 14:50:33 +0200 Message-ID: Subject: Re: Near-term pf plans From: =?UTF-8?Q?Ermal_Lu=C3=A7i?= To: Kristof Provost Cc: freebsd-net , "freebsd-pf@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 12:50:34 -0000 On Wed, Aug 26, 2015 at 1:43 PM, Kristof Provost wrote= : > On 2015-08-25 19:56:59 (+0200), Ermal Lu=C3=A7i wr= ote: > > On Sun, Aug 23, 2015 at 5:09 PM, Kristof Provost wrote= : > > > > > I'm inclined to say that ifgroups and interfaces should share a > > > namespace. That would certainly help pf, because it uses both > > > interchangeably, which just doesn't work unless they're in the sam= e > > > namespace. That affects more in the network stack though, so I'd > like > > > opinions for people with more experience with ifgroups. > > > > > > > > > The solution is easy the scenario of interface name and group name shou= ld > > not be allowed. > > I do not see the use case for this to be allowed at all its just > confusion > > in general in pf(4). > > > Agreed. I'll see about putting together a patch to do that. > Ok, keep me in the loop for the review. > > > > > > > - Removal of frag drop / frag crop support in pf. > ... > > While those provide more security protection they do not always work as > > advertised so i agree on their removal. > > > Okay, I'll get the patch committed soon. > > > > - PR 198868, 193579 (TSO issues) > > > pf has issues on certain network interfaces with TSO enabled. The > > > most important of these are amazon VMs. > > > I believe the problem is that pf tries to work with packets with > full > > > checksums. Usually output packets have pseudo-header checksums in > the IP > > > and TCP headers. pf doesn't know about those. It always tries to d= o > > > updates on checksums assuming there's a full checksum. (Which is t= he > > > case, pf always calculates a full checksum in pf_check_out()) > > > > > > I believe the fix for this issue is to have pf be aware of the > > > pseudo-header checksums. The type of checksum can be determined > from the > > > CSUM_TCP and CSUM_TCP_IPV6 flags. Whenever pf changes a packet > header it > > > will have to check for those flags to figure out if the checksum > should > > > be updated or not. > > > > > > > > I would be inclined to say that the real solution is not check packets > > generated from the host, > This also applies to forwarded packets. Think of things like NAT or port > rewriting. We have to change IP addresses for nat, and thus also update > checksums. Normally you would expect those with correct checksum AFAIK since the input path validates those as well! > > > otherwise you will have to go into complicated checksum handling which > > might not be worth it. > We should end up with slightly more complicated checksum code (because > sometimes we'll update and sometimes we won't), but I expect it to > actually be faster. Right now we always calculate a full checksum on > outbound packets. In the new case we'll either not touch the checksum, > or only calculate updates (depending on scenario). > > > I know Open has done some work on checksum handling altogether which > might > > be a good reason > > to go and look there first. > > > Right, I'll check. > > After looking at Open code, their solution is to not check anymore checksums on pf(4) code but only mark them to be recalculated if pf(NAT/RDR/BINAT) touches the packet and let the OS mechanics deal with them. Certainly they went to length on doing this allover but it seems a better choice, also same effort as making the checksum handling right, rather than duplicating whole portions of code which are difficult to get right and keep in sync. How do you say that we try to do the same in FreeBSD and seems a better path in maintainability rather than a complex code that can get out-of-date soon. Relying on the protocol routines deal with checksum handling, since they have already the code, is the better path and with so many offloading features on the NICs nowadays it does not make much sense to have it on pf(4), taking also in consideration the input path verifies them and output path handles them properly even during forwarding and even during host packet transmission. This also has the benefit to increase the performance! > > > - PR 172648 > > > This bug started out with an issue with TCP reassembly, but I've n= ot > > > been able to reproduce that. There is a problem with rdr targets f= or > > > ipv6 though. > > > ... > > I will try to look back at this but as a general rule look first at Ope= n > if > > they have already fixed this. > > IPv4 code has some security belts on such packets in pf(4) code to avoi= d > > doing the wrong thing. > > > I think their checks in ip6_output() are less strict than ours. > Regards, > Kristof > --=20 Ermal From owner-freebsd-pf@freebsd.org Wed Aug 26 14:10:54 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4C1539C2BEE for ; Wed, 26 Aug 2015 14:10:54 +0000 (UTC) (envelope-from Andrej.Kolontai@verwaltung.uni-muenchen.de) Received: from mailto1.verwaltung.uni-muenchen.de (mailto1.verwaltung.uni-muenchen.de [141.84.149.5]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "IronPort Appliance Demo Certificate", Issuer "IronPort Appliance Demo Certificate" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A56F369B for ; Wed, 26 Aug 2015 14:10:53 +0000 (UTC) (envelope-from Andrej.Kolontai@verwaltung.uni-muenchen.de) X-IronPort-AV: E=McAfee;i="5700,7163,7904"; a="8077054" X-IronPort-AV: E=Sophos;i="5.17,416,1437429600"; d="scan'208";a="8077054" Received: from cashts2.zuv.uni-muenchen.de ([10.153.81.104]) by smtpout1.verwaltung.uni-muenchen.de with ESMTP/TLS/AES256-SHA; 26 Aug 2015 16:09:41 +0200 Received: from MXS2.zuv.uni-muenchen.de ([fe80::e8db:cdb2:9a:a69f]) by CASHTS2.zuv.uni-muenchen.de ([::1]) with mapi id 14.03.0248.002; Wed, 26 Aug 2015 16:09:41 +0200 From: Kolontai Andrej To: "'freebsd-pf@freebsd.org'" Subject: RE: Machine freezes when loading pf ruleset Thread-Topic: Machine freezes when loading pf ruleset Thread-Index: AdDfSL0WCcADlcE1Qv6+xfswQgISowAjb+AAAAltW9A= Date: Wed, 26 Aug 2015 14:09:40 +0000 Message-ID: <894145A3DDBDEF4880E00D334DCD87263EC814A8@MXS2.zuv.uni-muenchen.de> References: In-Reply-To: Accept-Language: de-DE, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.23.107.156] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 14:10:54 -0000 PjEuNWsgcnVsZXMgc2VlbXMgbGlrZSBhIGxvdCBmb3IgUEYgdG8gaGFuZGxlLg0KPg0KPklzIHRo YXQgMS41ayBydWxlcyB5b3UndmUgd3JpdHRlbiBpbiB0aGUgY29uZiwgb3IgMS41ayBydWxlcyBm cm9tIGBwZmN0bCAtc3IgfCB3YyAtbCcgPw0KDQpZZXMsIHRoYXQncyB3aGF0IGlzIGluIHRoZSBj b25mIGZpbGVzLiBUaGUgbGF0dGVyIGNvbW1hbmQgZ2l2ZXMgYXJvdW5kIDM0MDAuLi4NCg0KPkkg d291bGQgc3VnZ2VzdCB5b3UgZmluZCBhIHdheSB0byBkcmFzdGljYWxseSBsb3dlciB0aGF0Lg0K DQpHaXZlbiB0aGUgbnVtYmVyIG9mIG5ldHdvcmtzLCBkZXZpY2VzIGFuZCBhcHBsaWNhdGlvbnMg dGhhdCBzZWVtcyB0byBiZSBkaWZmaWN1bHQuIEFsc28sIHdlIGhhdmUgc29tZSBwb2xpY2llcyBv biBvdXIgZmlyZXdhbGwgd2hpY2ggY29udHJpYnV0ZSB0byB0aGUgbGFyZ2UgbnVtYmVyIG9mIHJ1 bGVzOg0KKiBzdHJpY3QgbWluaW11bSBwcmluY2lwbGUuIE9ubHkgdHJhZmZpYyB0aGF0IGlzIGV4 cGxpY2l0bHkgbmVlZGVkIHBhc3NlcyB0aGUgZncuIFRoYXQgZ2l2ZXMgYSBsb3Qgb2YgY29tYmlu YXRpb25zLg0KKiBtb3N0IHJ1bGVzIGFyZSBlaXRoZXIgInBhc3MgaW4iIG9yICJwYXNzIG91dCIg LiBTbyBldmVyeSBjb25uZWN0aW9uIG5lZWRzIHRvIGhhdmUgYSBwYXNzIGluIGFuZCBhIHBhc3Mg b3V0IHJ1bGUuIFRoaXMgZG9lcyBub3QgbWVhbiB0aGF0IHdlIGhhdmUgZXZlcnkgcnVsZSB0d2lj ZS4gSXQncyBqdXN0IHRoYXQgaW5ib3VuZCBhbmQgb3V0Ym91bmQgcG9saWNpZXMgYXJlIGRpZmZl cmVudCB0aGluZ3MgYW5kIHdlIHRyeSBub3QgdG8gbWl4IHRoZW0gKGFjY2lkZW50bHkpLiANCiog dXNpbmcgInF1aWNrIiBpcyBhY3R1YWxseSBub3QgYWxsb3dlZCBleGNlcHQgaW4gdmVyeSBzcGVj aWFsIGNhc2VzIGFzIGl0IHRlbmRzIHRvIGNhdXNlIHVuZXhwZWN0ZWQgc2lkZSBlZmZlY3RzLiAN Cg0KV2hhdCB3ZSBkbyBpcywgb2YgY291cnNlLCB1c2luZyB0YWJsZXMgYW5kIGFuY2hvcnMgdG8g bWFrZSB0aGUgZXZhbHVhdGlvbiBvZiB0aGUgcnVsZSBzZXQgYXMgZWZmaWNpZW50IGFzIHBvc3Np YmxlLiANCg0KSSBuZWVkIHRvIHNheSB0aGF0IGluIG5vcm1hbCBvcGVyYXRpb24sIHRoZSBtYWNo aW5lcyBwZXJmb3JtIHJlYWxseSB3ZWxsIHdpdGggYWJzb2x1dGVseSBubyBzaWduIG9mIHNob3J0 YWdlIGluIGFueSByZXNvdXJjZS4gDQoNCkEgdHlwaWNhbCB0b3Agc2NyZWVuIGxvb2tzIGxpa2Ug dGhpczoNCg0KbGFzdCBwaWQ6IDU5ODc1OyAgbG9hZCBhdmVyYWdlczogIDAuMjksICAwLjMzLCAg MC4zNiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIHVwIDArMDI6NTU6NTEgIDE1OjAyOjIyDQozOCBwcm9j ZXNzZXM6ICAxIHJ1bm5pbmcsIDM3IHNsZWVwaW5nDQpDUFU6ICAwLjAlIHVzZXIsICAwLjAlIG5p Y2UsICAwLjAlIHN5c3RlbSwgIDAuNyUgaW50ZXJydXB0LCA5OS4yJSBpZGxlDQpNZW06IDM1ODhL IEFjdGl2ZSwgMTg4M00gSW5hY3QsIDIzMzNNIFdpcmVkLCAyMDM3TSBCdWYsIDg5RyBGcmVlDQpT d2FwOiAxMTdHIFRvdGFsLCAxMTdHIEZyZWUNCi4uLg0KDQpUaGVyZSdzIHBsZW50eSBvZiBtZW1v cnkgYW5kIGNwdSBwb3dlci4gTmV0d29ya2luZyBpcyAocGh5c2ljYWxseSkgY2FwYWJsZSBvZiA0 MCBnYml0L3MgICg0IGFnZ3JlZ2F0ZWQgMTAgZ2JpdCBsaW5rcykuIEkgaGF2ZSBuZXZlciBzZWVu IGEgY3B1IGxvYWQgaGlnaGVyIHRoYW4gMS5zb21ldGhpbmcuIA0KDQpUaGUgcHJvYmxlbSBvY2N1 cnMgb25seSBpbiB0aGUgbW9tZW50IEkgbG9hZCB0aGUgcnVsZXNldCBhbmQgb25seSBpZiB0aGUg bWFjaGluZSBpcyBpbiBtYXN0ZXIgc3RhdGUuIElmIEkgb25seSBrbmV3IHdoYXQgaXMgaGFwcGVu aW5nIGF0IHRoYXQgcG9pbnQuLi4gSSBjYW4ndCBtYWtlIGFueSBzZW5zZSBvZiB0aGUgbG9nIG91 dHB1dHMuIA0KDQo+WW91IG1heSBhbHNvIHdpc2ggdG8gZW5zdXJlIDoNCj4tIHlvdSdyZSB1c2lu ZywgaWYgYXQgYWxsIHBvc3NpYmxlLCBhICpkZWRpY2F0ZWQqIHBmc3luYyBsaW5rIChsaWtlIGEg Y2FibGUgb24gYSBkZWRpY2F0ZWQgaW50ZXJmYWNlIGJldHdlZW4gdGhlIGJveGVzKQ0KDQpZZXMs IHRvIHNvbWUgZGVncmVlIHRoYXQgaXMgdHJ1ZS4gUGZzeW5jIHJ1bnMgb3ZlciBzZXBhcmF0ZSBw aHlzaWNhbCBpbnRlcmZhY2VzLiBCdXQgdGhlIG1hY2hpbmVzIGFyZSBhdCBkaWZmZXJlbnQgcGxh Y2VzLiBUaGF0IG1lYW5zIHRoYXQgdGhlIHBmc3luYyBuZXQgaXMgaW4gZmFjdCBqdXN0IGFub3Ro ZXIgdmxhbiB3aGljaCBydW5zIG92ZXIgdGhlIHNhbWUgZmliZXIgbGluayBhcyBldmVyeXRoaW5n IGVsc2UuIFllcywgd2UgdXNlZCB0byB1c2UgSVBzZWMgdG8gc2VjdXJlIGl0LiBSaWdodCBub3cs IElwc2VjIGlzIHR1cm5lZCBvZmYgZm9yIHBmc3luYyB0byBleGNsdWRlIHRoZSBwb3NzaWJpbGl0 eSB0aGF0IHRoaXMgY2F1c2VzIHRoZSBwcm9ibGVtLiBCdXQgdGhhdCdzIGFwcGFyZW50bHkgbm90 IHRoZSBjYXNlLiBJdCBzdGlsbCBoYXBwZW5zLiANCg0KwqANCg== From owner-freebsd-pf@freebsd.org Wed Aug 26 14:44:52 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A9D19C39BB for ; Wed, 26 Aug 2015 14:44:52 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-lb0-f176.google.com (mail-lb0-f176.google.com [209.85.217.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D922978 for ; Wed, 26 Aug 2015 14:44:50 +0000 (UTC) (envelope-from ml@my.gd) Received: by lbbpu9 with SMTP id pu9so121412648lbb.3 for ; Wed, 26 Aug 2015 07:44:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=qrRMHpXQ4YvMXKJuhpckkRIHllXSSbUCih9tFoFhTlg=; b=R52YGAxSyvi/G/Zb66f7yrT+6f9/m+jBFi4Cz93PRrtevKuoK5PTxvWGFML3wCBO/k sOPoQnM0wyrlIDilh/dE5XnNHNUDP6tXBdF+5RaL/VCTwpdH3me17nfNQHhFTbzm9RpN 0Ao4W7VyMwAYbqoKV6v0QrKPcFjp0NgmIXXZ9NFgApy/HtAki91ZbpQF2ynsU5SFrPff 6JFl42tKBfy+WgxvG5bxBEy1jkHtNzCe0XYSzMqzDPQ/HBXu03uwL9yZDxYQJ4M8M9jG ofKn/t/7tJ6t/b1EtghSFUo7Rm8Cu93cOQONtv8FccSCQySdPgDAGdHaKBpbFn7gXtVe CtAQ== X-Gm-Message-State: ALoCoQntQkoV6m01xXFQJkAXY2Ec/4s40VGzAJ9QJHM0BRJJ9St2XRhFlGZ3+wVFDw7FIyHQ60YF MIME-Version: 1.0 X-Received: by 10.152.28.105 with SMTP id a9mr31796541lah.9.1440599808943; Wed, 26 Aug 2015 07:36:48 -0700 (PDT) Received: by 10.112.60.34 with HTTP; Wed, 26 Aug 2015 07:36:48 -0700 (PDT) In-Reply-To: <894145A3DDBDEF4880E00D334DCD87263EC814A8@MXS2.zuv.uni-muenchen.de> References: <894145A3DDBDEF4880E00D334DCD87263EC814A8@MXS2.zuv.uni-muenchen.de> Date: Wed, 26 Aug 2015 16:36:48 +0200 Message-ID: Subject: Re: Machine freezes when loading pf ruleset From: Damien Fleuriot To: Kolontai Andrej Cc: "freebsd-pf@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 14:44:52 -0000 On 26 August 2015 at 16:09, Kolontai Andrej < Andrej.Kolontai@verwaltung.uni-muenchen.de> wrote: > >1.5k rules seems like a lot for PF to handle. > > > >Is that 1.5k rules you've written in the conf, or 1.5k rules from `pfctl > -sr | wc -l' ? > > Yes, that's what is in the conf files. The latter command gives around > 3400... > > >I would suggest you find a way to drastically lower that. > > Given the number of networks, devices and applications that seems to be > difficult. Also, we have some policies on our firewall which contribute to > the large number of rules: > * strict minimum principle. Only traffic that is explicitly needed passes > the fw. That gives a lot of combinations. > * most rules are either "pass in" or "pass out" . So every connection > needs to have a pass in and a pass out rule. This does not mean that we > have every rule twice. It's just that inbound and outbound policies are > different things and we try not to mix them (accidently). > * using "quick" is actually not allowed except in very special cases as it > tends to cause unexpected side effects. > > What we do is, of course, using tables and anchors to make the evaluation > of the rule set as efficient as possible. > > I need to say that in normal operation, the machines perform really well > with absolutely no sign of shortage in any resource. > > A typical top screen looks like this: > > last pid: 59875; load averages: 0.29, 0.33, 0.36 > > up 0+02:55:51 15:02:22 > 38 processes: 1 running, 37 sleeping > CPU: 0.0% user, 0.0% nice, 0.0% system, 0.7% interrupt, 99.2% idle > Mem: 3588K Active, 1883M Inact, 2333M Wired, 2037M Buf, 89G Free > Swap: 117G Total, 117G Free > ... > > There's plenty of memory and cpu power. Networking is (physically) capable > of 40 gbit/s (4 aggregated 10 gbit links). I have never seen a cpu load > higher than 1.something. > > The problem occurs only in the moment I load the ruleset and only if the > machine is in master state. If I only knew what is happening at that > point... I can't make any sense of the log outputs. > > >You may also wish to ensure : > >- you're using, if at all possible, a *dedicated* pfsync link (like a > cable on a dedicated interface between the boxes) > > Yes, to some degree that is true. Pfsync runs over separate physical > interfaces. But the machines are at different places. That means that the > pfsync net is in fact just another vlan which runs over the same fiber link > as everything else. Yes, we used to use IPsec to secure it. Right now, > Ipsec is turned off for pfsync to exclude the possibility that this causes > the problem. But that's apparently not the case. It still happens. > > > Well the machine seems to be very apt for the job indeed with plenty of spare resources. While I believe your filtering on egress traffic to be somewhat redundant since you already filter inbound connections, it is not my place to comment on that, not to mention that would certainly entail huge changes on your side. I do believe however that when you reload the ruleset PF sets some kind of a lock. I'm afraid I'm not familiar enough with the internals of PF to dig deeper into this, but that may be worth a look into. From owner-freebsd-pf@freebsd.org Thu Aug 27 12:42:17 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 164829C3B7B for ; Thu, 27 Aug 2015 12:42:17 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: from mail-yk0-x234.google.com (mail-yk0-x234.google.com [IPv6:2607:f8b0:4002:c07::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CFEEC1926 for ; Thu, 27 Aug 2015 12:42:16 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: by ykll84 with SMTP id l84so17947892ykl.0 for ; Thu, 27 Aug 2015 05:42:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=19bhhWCpo7R+J0KUZOL4ppp8FIbLhHW7SuvqM+o2M8A=; b=quJ0jRzIOPUrcVP3pVP7byxRwvs0/oBLvMZJoyKxAJ9oINHdi5FXHV/syzfOrL/X7/ wxAZ0gdk3doyXo8i9pgKDDpwWfMV88c456zQftLoUAntQMRQuynNJx7v4Wkh0aRuI5tQ vRBb9y1G2vYqbkrxaxvv18Ebek1Jh9nxhhokq/QElaE1bdajdm+zJgufFeMRa8ioiB3U UP1hMr48inpLoy2oP58hgtEW24+qIxY9IX2/bvrf4J/Qkmsd8dyeBBYTBpTFQusqnI6X +cP4Wl1/vQqaMWFRpYxeWEMcsXLwYyCTs4a+T12CCoyJE2+DjWZAtX93nHJ78RjRME49 utCQ== MIME-Version: 1.0 X-Received: by 10.129.117.4 with SMTP id q4mr3078297ywc.15.1440679335847; Thu, 27 Aug 2015 05:42:15 -0700 (PDT) Received: by 10.129.73.205 with HTTP; Thu, 27 Aug 2015 05:42:15 -0700 (PDT) In-Reply-To: <894145A3DDBDEF4880E00D334DCD87263EC814A8@MXS2.zuv.uni-muenchen.de> References: <894145A3DDBDEF4880E00D334DCD87263EC814A8@MXS2.zuv.uni-muenchen.de> Date: Thu, 27 Aug 2015 14:42:15 +0200 Message-ID: Subject: Re: Machine freezes when loading pf ruleset From: =?UTF-8?Q?Ermal_Lu=C3=A7i?= To: Kolontai Andrej Cc: "freebsd-pf@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2015 12:42:17 -0000 On Wed, Aug 26, 2015 at 4:09 PM, Kolontai Andrej < Andrej.Kolontai@verwaltung.uni-muenchen.de> wrote: > >1.5k rules seems like a lot for PF to handle. > > > >Is that 1.5k rules you've written in the conf, or 1.5k rules from `pfctl > -sr | wc -l' ? > > Yes, that's what is in the conf files. The latter command gives around > 3400... > > >I would suggest you find a way to drastically lower that. > > Given the number of networks, devices and applications that seems to be > difficult. Also, we have some policies on our firewall which contribute to > the large number of rules: > * strict minimum principle. Only traffic that is explicitly needed passes > the fw. That gives a lot of combinations. > * most rules are either "pass in" or "pass out" . So every connection > needs to have a pass in and a pass out rule. This does not mean that we > have every rule twice. It's just that inbound and outbound policies are > different things and we try not to mix them (accidently). > * using "quick" is actually not allowed except in very special cases as it > tends to cause unexpected side effects. > > What we do is, of course, using tables and anchors to make the evaluation > of the rule set as efficient as possible. > > I need to say that in normal operation, the machines perform really well > with absolutely no sign of shortage in any resource. > > A typical top screen looks like this: > > last pid: 59875; load averages: 0.29, 0.33, 0.36 > > up 0+02:55:51 15:02:22 > 38 processes: 1 running, 37 sleeping > CPU: 0.0% user, 0.0% nice, 0.0% system, 0.7% interrupt, 99.2% idle > Mem: 3588K Active, 1883M Inact, 2333M Wired, 2037M Buf, 89G Free > Swap: 117G Total, 117G Free > ... > > There's plenty of memory and cpu power. Networking is (physically) capable > of 40 gbit/s (4 aggregated 10 gbit links). I have never seen a cpu load > higher than 1.something. > > The problem occurs only in the moment I load the ruleset and only if the > machine is in master state. If I only knew what is happening at that > point... I can't make any sense of the log outputs. > > >You may also wish to ensure : > >- you're using, if at all possible, a *dedicated* pfsync link (like a > cable on a dedicated interface between the boxes) > > Yes, to some degree that is true. Pfsync runs over separate physical > interfaces. But the machines are at different places. That means that the > pfsync net is in fact just another vlan which runs over the same fiber link > as everything else. Yes, we used to use IPsec to secure it. Right now, > Ipsec is turned off for pfsync to exclude the possibility that this causes > the problem. But that's apparently not the case. It still happens. > > > The patch provided at https://reviews.freebsd.org/D3503 should help your case. During a full ruleset reload, taking into account so many rules, you will impact normal packet processing. Hence you have the feeling of the box being frozen or not forwarding traffic. That patch reduces the overhead of reloading a ruleset. Though even more lock breakdown is necessary on pf(4) but that is another topic. > _______________________________________________ > freebsd-pf@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-pf > To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" > -- Ermal From owner-freebsd-pf@freebsd.org Thu Aug 27 13:32:28 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D3F8F9C220E for ; Thu, 27 Aug 2015 13:32:28 +0000 (UTC) (envelope-from Andrej.Kolontai@verwaltung.uni-muenchen.de) Received: from mailto1.verwaltung.uni-muenchen.de (mailto1.verwaltung.uni-muenchen.de [141.84.149.5]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "IronPort Appliance Demo Certificate", Issuer "IronPort Appliance Demo Certificate" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3CD779E1 for ; Thu, 27 Aug 2015 13:32:27 +0000 (UTC) (envelope-from Andrej.Kolontai@verwaltung.uni-muenchen.de) X-IronPort-AV: E=McAfee;i="5700,7163,7905"; a="8086288" X-IronPort-AV: E=Sophos;i="5.17,422,1437429600"; d="scan'208";a="8086288" Received: from cashts2.zuv.uni-muenchen.de ([10.153.81.104]) by smtpout1.verwaltung.uni-muenchen.de with ESMTP/TLS/AES256-SHA; 27 Aug 2015 15:32:24 +0200 Received: from MXS2.zuv.uni-muenchen.de ([fe80::e8db:cdb2:9a:a69f]) by CASHTS2.zuv.uni-muenchen.de ([::1]) with mapi id 14.03.0248.002; Thu, 27 Aug 2015 15:32:24 +0200 From: Kolontai Andrej To: =?utf-8?B?J0VybWFsIEx1w6dpJw==?= CC: "'freebsd-pf@freebsd.org'" Subject: RE: Machine freezes when loading pf ruleset Thread-Topic: Machine freezes when loading pf ruleset Thread-Index: AdDfSL0WCcADlcE1Qv6+xfswQgISowAjb+AAAAltW9AALjXrgAAF5hpw Date: Thu, 27 Aug 2015 13:32:23 +0000 Message-ID: <894145A3DDBDEF4880E00D334DCD87263EC83B6C@MXS2.zuv.uni-muenchen.de> References: <894145A3DDBDEF4880E00D334DCD87263EC814A8@MXS2.zuv.uni-muenchen.de> In-Reply-To: Accept-Language: de-DE, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.23.107.156] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2015 13:32:28 -0000 PlRoZSBwYXRjaCBwcm92aWRlZCBhdMKgaHR0cHM6Ly9yZXZpZXdzLmZyZWVic2Qub3JnL0QzNTAz IHNob3VsZCBoZWxwIHlvdXIgY2FzZS4NCj5EdXJpbmcgYSBmdWxsIHJ1bGVzZXQgcmVsb2FkLCB0 YWtpbmcgaW50byBhY2NvdW50IHNvIG1hbnkgcnVsZXMsIHlvdSB3aWxsIGltcGFjdCBub3JtYWwg cGFja2V0IHByb2Nlc3NpbmcuDQo+SGVuY2UgeW91IGhhdmUgdGhlIGZlZWxpbmcgb2YgdGhlIGJv eCBiZWluZyBmcm96ZW4gb3Igbm90IGZvcndhcmRpbmcgdHJhZmZpYy4NCg0KPlRoYXQgcGF0Y2gg cmVkdWNlcyB0aGUgb3ZlcmhlYWQgb2YgcmVsb2FkaW5nIGEgcnVsZXNldC4NCj5UaG91Z2ggZXZl biBtb3JlIGxvY2sgYnJlYWtkb3duIGlzIG5lY2Vzc2FyeSBvbiBwZig0KSBidXQgdGhhdCBpcyBh bm90aGVyIHRvcGljLg0KDQpTb3VuZHMgZ3JlYXQuIEknbGwgdHJ5IHRoYXQuDQoNCkFuZHJlag0K wqANCg==