From owner-freebsd-pf@freebsd.org Mon Sep 11 04:41:59 2017 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DEF5FE1A02C for ; Mon, 11 Sep 2017 04:41:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CDAA46FB02 for ; Mon, 11 Sep 2017 04:41:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8B4fw6D044402 for ; Mon, 11 Sep 2017 04:41:59 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-pf@FreeBSD.org Subject: [Bug 222126] pf is not clearing expired states Date: Mon, 11 Sep 2017 04:41:59 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: maximos@als.nnov.ru X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-pf@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2017 04:42:00 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222126 Max changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |maximos@als.nnov.ru --- Comment #11 from Max --- (In reply to noah.bergbauer from comment #10) Are there any limits in your ruleset? And what does the "pfctl -vsi" show? Sounds familiar... just like bug #217997. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-pf@freebsd.org Mon Sep 11 10:42:39 2017 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2557AE038CC for ; Mon, 11 Sep 2017 10:42:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 140F07FDBC for ; Mon, 11 Sep 2017 10:42:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8BAgche069442 for ; Mon, 11 Sep 2017 10:42:38 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-pf@FreeBSD.org Subject: [Bug 222126] pf is not clearing expired states Date: Mon, 11 Sep 2017 10:42:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: noah.bergbauer@tum.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-pf@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2017 10:42:39 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222126 --- Comment #12 from noah.bergbauer@tum.de --- set limit { states 100000, src-nodes 10000 } One of my first attempts to fix this was increasing both limits 10x - didn't help though. # pfctl -vsi No ALTQ support in kernel ALTQ related functions disabled Status: Enabled for 1 days 14:44:53 Debug: Urgent Hostid: 0x4b1e78c2 Checksum: 0x67f2a9cbd7b0d65ce52864ecfc156ebb State Table Total Rate current entries 3839=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20 searches 360179452 2582.1/s inserts 594949 4.3/s removals 591110 4.2/s Source Tracking Table current entries 0=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20 searches 0 0.0/s inserts 0 0.0/s removals 0 0.0/s Counters match 689782 4.9/s bad-offset 0 0.0/s fragment 16 0.0/s short 0 0.0/s normalize 0 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 0 0.0/s proto-cksum 450 0.0/s state-mismatch 942 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s map-failed 0 0.0/s Limit Counters max states per rule 0 0.0/s max-src-states 0 0.0/s max-src-nodes 0 0.0/s max-src-conn 0 0.0/s max-src-conn-rate 0 0.0/s overload table insertion 0 0.0/s overload flush states 0 0.0/s --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-pf@freebsd.org Mon Sep 11 11:38:30 2017 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1F45E06354 for ; Mon, 11 Sep 2017 11:38:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D066F819ED for ; Mon, 11 Sep 2017 11:38:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8BBcUV8042596 for ; Mon, 11 Sep 2017 11:38:30 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-pf@FreeBSD.org Subject: [Bug 222126] pf is not clearing expired states Date: Mon, 11 Sep 2017 11:38:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: maximos@als.nnov.ru X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-pf@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2017 11:38:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222126 --- Comment #13 from Max --- (In reply to noah.bergbauer from comment #12) > Status: Enabled for 1 days 14:44:53 Have you had any issues during this period? And do you know which rule produces expired states? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-pf@freebsd.org Mon Sep 11 11:42:51 2017 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68575E06831 for ; Mon, 11 Sep 2017 11:42:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 572FA81E97 for ; Mon, 11 Sep 2017 11:42:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8BBgplj056963 for ; Mon, 11 Sep 2017 11:42:51 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-pf@FreeBSD.org Subject: [Bug 222126] pf is not clearing expired states Date: Mon, 11 Sep 2017 11:42:51 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: noah.bergbauer@tum.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-pf@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2017 11:42:51 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222126 --- Comment #14 from noah.bergbauer@tum.de --- (In reply to Max from comment #13) Maybe, maybe not. The point of my workaround is to get a mostly functioning machine. However, the reboot right before this period was necessary because= for some reason not even that workaround works consistently. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-pf@freebsd.org Mon Sep 11 11:53:45 2017 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B1F9E07073 for ; Mon, 11 Sep 2017 11:53:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 79EA082420 for ; Mon, 11 Sep 2017 11:53:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8BBrjST079693 for ; Mon, 11 Sep 2017 11:53:45 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-pf@FreeBSD.org Subject: [Bug 222126] pf is not clearing expired states Date: Mon, 11 Sep 2017 11:53:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kp@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-pf@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2017 11:53:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222126 --- Comment #15 from Kristof Provost --- (In reply to noah.bergbauer from comment #14) Given the nature of your workaround and what we've seen from Dtrace I don't think that #217997 is the problem. I'm also pretty sure that that fix is included in 11.1. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-pf@freebsd.org Mon Sep 11 12:06:00 2017 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9483E080CC for ; Mon, 11 Sep 2017 12:06:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A870982CAF for ; Mon, 11 Sep 2017 12:06:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8BC60xo034211 for ; Mon, 11 Sep 2017 12:06:00 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-pf@FreeBSD.org Subject: [Bug 222126] pf is not clearing expired states Date: Mon, 11 Sep 2017 12:06:00 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: maximos@als.nnov.ru X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-pf@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2017 12:06:00 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222126 --- Comment #16 from Max --- (In reply to noah.bergbauer from comment #14) I'll try to reproduce the problem. But I need some starting point. Rules, d= ead connections state entries... Anything? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-pf@freebsd.org Mon Sep 11 12:10:27 2017 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 80217E083DE for ; Mon, 11 Sep 2017 12:10:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6EF9F82F3A for ; Mon, 11 Sep 2017 12:10:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v8BCAQw5010157 for ; Mon, 11 Sep 2017 12:10:27 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-pf@FreeBSD.org Subject: [Bug 222126] pf is not clearing expired states Date: Mon, 11 Sep 2017 12:10:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: maximos@als.nnov.ru X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-pf@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2017 12:10:27 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222126 --- Comment #17 from Max --- (In reply to Kristof Provost from comment #15) You are right. It is not the problem. But it looks quite similar. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-pf@freebsd.org Thu Sep 14 14:21:10 2017 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F1B4E1B25A; Thu, 14 Sep 2017 14:21:10 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 355C775B25; Thu, 14 Sep 2017 14:21:09 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 6001320D6E; Thu, 14 Sep 2017 10:21:08 -0400 (EDT) Received: from web6 ([10.202.2.216]) by compute7.internal (MEProxy); Thu, 14 Sep 2017 10:21:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skunkwerks.at; h=content-transfer-encoding:content-type:date:from:message-id :mime-version:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s= mesmtp; bh=/xdndg8jFeCEH1B2zVXhFA5yogLPDx4bX31QXNl+u1U=; b=WXjbz m0H6dH0EE9M3/eY7tIyLBS4gyG978i9LhhXMQJCRnt+B1l9+ypCnN/By8vmj10FJ uNkrQ8bphED3jwz8VLPnR2dEgvO6UQfeC6LiXS4AfnJcbstkbnxTeETIGtGOZEUX AdMfR55f0OYDyCf3yX9j4Ww7SALlkwVXTMETzg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=/xdndg8jFeCEH1B2zVXhFA5yogLPD x4bX31QXNl+u1U=; b=dNs6qoNs+kB82nrmTXGFdOFPSyede34d9ev8p8wgKM4Jf A6pCx9zW7XZTScrW5PlKXYc7tBCtZkyvTg6RmUFbDK+7KYWRfjkl5DpA3UgUCYcP 5QbnXfsoHbrQxSAdfdv9d7bZjVFMbIbmOGKu+l6dH+7H/Qg0cRGtNVwuJ7Em9KDf koIEBVo/4ZW7x5Ir6R5VOiiK7JG47Dxf07xMnwHC61Fd836GL7dOwWKtfrLXIYD1 3F4Y/fGC3dTCLuhdCSLKokwsEbEFKGxSKrrZkDudisEwtFCX6MPg0ic4GrGuvrQw 8vsH0gmVeWUXmzIEE5nHyxrnARJ85F4/7HCT+GC3Q== X-ME-Sender: Received: by mailuser.nyi.internal (Postfix, from userid 99) id 2AA1A48004; Thu, 14 Sep 2017 10:21:08 -0400 (EDT) Message-Id: <1505398868.955393.1106053824.42CA3E40@webmail.messagingengine.com> From: Dave Cottlehuber To: freebsd-pf@freebsd.org, freebsd-net@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="utf-8" X-Mailer: MessagingEngine.com Webmail Interface - ajax-64b08692 Subject: NATted outbound traffic sometimes uses backup CARP IP on LACP/LAGG interface Date: Thu, 14 Sep 2017 16:21:08 +0200 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Sep 2017 14:21:10 -0000 Hi, Outgoing traffic (from a jail) via PF NAT over a LAGG/LACP sometimes has the *backup* CARP IP address assigned to it. Obivously as this IP is only active on the "other" server, the return TCP connection traffic never actually gets back to our CARP master, and the other server sees spurious TCP connections. This is very reproducible and appears to be deterministic, like a round robin IP allocation. In practice, inside a jail, `curl $URL` will fail repeatedly. Hopefully this is some misconfiguration on my part - what am I doing wrong? BTW I wrote this up a while back on the forums where the config files are easier to read: https://forums.freebsd.org/threads/61552 ############################### # /etc/rc.conf network ifconfig_igb0="up" ifconfig_igb1="up" cloned_interfaces="${cloned_interfaces} lagg0" defaultrouter="1.2.3.81" ipv6_defaultrouter="1:2:3:4::1" ifconfig_lagg0="inet 1.2.3.83/28 laggproto lacp laggport igb0 laggport igb1" ifconfig_lagg0_ipv6="inet6 1:2:3:4::83/64" # carp on kld_list="${kld_list} carp" ifconfig_lagg0_aliases="\ inet vhid 1 advskew 100 pass pw1 1.2.3.84/32 \ inet6 vhid 2 advskew 100 pass pw2 1:2:3:4::84/64 \ inet vhid 3 advskew 0 pass pw3 1.2.3.85/32 \ inet6 vhid 4 advskew 0 pass pw4 1:2:3:4::85/64 \ " # jail networks use their own separate cloned if cloned_interfaces="${cloned_interfaces} lo1" ifconfig_lo1_aliases="inet 10.241.0.0-15/16" ############################### # /etc/pf.conf # macros protocols = "{ tcp, udp, icmp }" # interfaces extl_if="lagg0" intl_if="lo0" jail_if="lo1" # networks intl_net = $intl_if:network jail_net = $jail_if:network internet = $extl_if:network # limits set limit { states 200000, frags 80000, src-nodes 80000 } set timeout { adaptive.start 180000, adaptive.end 200000 } # clean packets are happy packets scrub in all # jails are allowed outbound connections but not inbound nat on $extl_if proto $protocols from $jail_net to any -> ($extl_if) # o ye of little faith pass in all pass out all ############################### ######## running configs ###### pfctl indeed shows its a round-robin ############################### # pfctl -vnf /etc/pf.conf protocols = "{ tcp, udp, icmp }" extl_if = "lagg0" intl_if = "lo0" jail_if = "lo1" intl_net = "lo0:network" jail_net = "lo1:network" internet = "lagg0:network" set limit states 200000 set limit frags 80000 set limit src-nodes 80000 set timeout adaptive.start 180000 set timeout adaptive.end 200000 scrub in all fragment reassemble nat on lagg0 inet proto tcp from 10.241.0.0/16 to any -> (lagg0) round-robin nat on lagg0 inet proto tcp from 10.241.0.1 to any -> (lagg0) round-robin ... repeated for each IP ############################### # ifconfig lagg0: flags=8943 metric 0 mtu 1500 options=6403bb ether 78:45:c4:fa:d2:99 inet 1.2.3.82 netmask 0xfffffff0 broadcast 1.2.3.95 * inet 1.2.3.84 netmask 0xffffffff broadcast 1.2.3.84 vhid 1 * inet 1.2.3.85 netmask 0xffffffff broadcast 1.2.3.85 vhid 3 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! inet6 fe80::7a45:c4ff:fefa:d299%lagg0 prefixlen 64 scopeid 0x4 inet6 1:2:3:4::82 prefixlen 64 * inet6 1:2:3:4::84 prefixlen 64 vhid 2 * inet6 1:2:3:4::85 prefixlen 64 vhid 4 nd6 options=21 media: Ethernet autoselect status: active * carp: MASTER vhid 1 advbase 1 advskew 0 * carp: BACKUP vhid 3 advbase 1 advskew 100 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! * carp: MASTER vhid 2 advbase 1 advskew 0 * carp: BACKUP vhid 4 advbase 1 advskew 100 groups: lagg laggproto lacp lagghash l2,l3,l4 * laggport: igb0 flags=1c * laggport: igb1 flags=1c # I removed the lines appended with !!!!!!!!!!!.. so that the system actually works atm From owner-freebsd-pf@freebsd.org Thu Sep 14 21:32:24 2017 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BD84EE0D71F; Thu, 14 Sep 2017 21:32:24 +0000 (UTC) (envelope-from srs0=woc2=ap=sigsegv.be=kristof@codepro.be) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 86C006AA01; Thu, 14 Sep 2017 21:32:24 +0000 (UTC) (envelope-from srs0=woc2=ap=sigsegv.be=kristof@codepro.be) Received: from [192.168.228.1] (ptr-8ripyyf09rrerk0utqq.18120a2.ip6.access.telenet.be [IPv6:2a02:1811:2419:4e02:2812:4e88:3eaf:f212]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id E71D146074; Thu, 14 Sep 2017 23:32:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sigsegv.be; s=mail; t=1505424741; bh=a/9B4TIF1FknxE80ydLk9S19/O1NRT2F9pBzOBTjfzQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=2rqxWSegRGhej1M6wSurDXY/xehdQ8UEf6RgKrdxPMj+2W+tJRWgxdGqytNdHoxM9 7A9ANPjQBWZDAnaMMiGdCmNmFz+z4/+FicAu7SUHBfgZurOdbSLgB/tg/wOI3TvpkS Vs4xlU2jsbTI+WZMaWPB7aycM+d6c2sNkiir9wZQ= From: "Kristof Provost" To: "Dave Cottlehuber" Cc: freebsd-pf@freebsd.org, freebsd-net@freebsd.org Subject: Re: NATted outbound traffic sometimes uses backup CARP IP on LACP/LAGG interface Date: Thu, 14 Sep 2017 23:32:21 +0200 Message-ID: <4250D983-3CE2-4156-85FB-E580F6C592D2@sigsegv.be> In-Reply-To: <1505398868.955393.1106053824.42CA3E40@webmail.messagingengine.com> References: <1505398868.955393.1106053824.42CA3E40@webmail.messagingengine.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Mailer: MailMate (2.0BETAr6090) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Sep 2017 21:32:24 -0000 On 14 Sep 2017, at 16:21, Dave Cottlehuber wrote: > Outgoing traffic (from a jail) via PF NAT over a LAGG/LACP sometimes > has > the *backup* CARP IP address assigned to it. > > > ############################### > ######## running configs ###### > > pfctl indeed shows its a round-robin > > ############################### > # pfctl -vnf /etc/pf.conf > protocols = "{ tcp, udp, icmp }" > extl_if = "lagg0" > intl_if = "lo0" > jail_if = "lo1" > intl_net = "lo0:network" > jail_net = "lo1:network" > internet = "lagg0:network" > set limit states 200000 > set limit frags 80000 > set limit src-nodes 80000 > set timeout adaptive.start 180000 > set timeout adaptive.end 200000 > scrub in all fragment reassemble > nat on lagg0 inet proto tcp from 10.241.0.0/16 to any -> (lagg0) > round-robin > nat on lagg0 inet proto tcp from 10.241.0.1 to any -> (lagg0) > round-robin > I think this is your problem. You’re telling pf to nat to the IP address of lagg0, but lagg0 has multiple addresses assigned. ‘(lagg0:0)’ should work, or just use the IP address. Regards, Kristof From owner-freebsd-pf@freebsd.org Fri Sep 15 09:33:59 2017 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60944E09C9B; Fri, 15 Sep 2017 09:33:59 +0000 (UTC) (envelope-from srs0=pom8=aq=sigsegv.be=kristof@codepro.be) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 10B6083F38; Fri, 15 Sep 2017 09:33:58 +0000 (UTC) (envelope-from srs0=pom8=aq=sigsegv.be=kristof@codepro.be) Received: from [169.254.49.62] (vega.codepro.be [IPv6:2a01:4f8:162:1127::3]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id CCFA6466A3; Fri, 15 Sep 2017 11:33:55 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sigsegv.be; s=mail; t=1505468035; bh=Rb0+eTInI/bRd+H2L5jt+T1ZQs9Su7AGyjDx3QBGhp4=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=e+xkLc47HjV3MKwfQhoA4I0Rf7eAyFZVV/tg/4iKQQ67ZKrvKwRqHs1ErwEgAFqaS fJWu5AqW4jYmHerGCNuw36X2w9uzYDVnW6k0bVv6MfN/rAxmSVCTKVIhB7WpTgZy6d Nm+xQaxpquUXXRO/ES9zgBdr06CKhkdSbC+UsCk8= From: "Kristof Provost" To: "Dave Cottlehuber" Cc: freebsd-net@freebsd.org, freebsd-pf@freebsd.org Subject: Re: NATted outbound traffic sometimes uses backup CARP IP on LACP/LAGG interface Date: Fri, 15 Sep 2017 11:33:58 +0200 Message-ID: <91414E93-FAFF-4EE6-A91D-F10FDA959783@sigsegv.be> In-Reply-To: <1505467875.1199722.1107024744.004A81D1@webmail.messagingengine.com> References: <1505398868.955393.1106053824.42CA3E40@webmail.messagingengine.com> <4250D983-3CE2-4156-85FB-E580F6C592D2@sigsegv.be> <1505467875.1199722.1107024744.004A81D1@webmail.messagingengine.com> MIME-Version: 1.0 X-Mailer: MailMate (2.0BETAr6090) Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 09:33:59 -0000 On 15 Sep 2017, at 11:31, Dave Cottlehuber wrote: > Can you explain what $if:0 resolves to, for example how does it relate > to to the primary ipv4/6 addresses bound to that interface? > > I couldn't find a reference in the usual ifconfig manpages about this > (ifname:#) format, the BNF grammar for pf.conf doesn't cover it > either, > and `pfctl -vnf ...` simply shows (lagg0:0). > It tells pf to not use any of the alias addresses. It’s explained in the pf.conf man page: Interface names and interface group names can have modifiers appended: :network Translates to the network(s) attached to the interface. :broadcast Translates to the interface's broadcast address(es). :peer Translates to the point-to-point interface's peer address(es). :0 Do not include interface aliases. Host names may also have the :0 option appended to restrict the name resolution to the first of each v4 and v6 address found. Regards, Kristof From owner-freebsd-pf@freebsd.org Fri Sep 15 09:38:08 2017 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5E04E0A1B2; Fri, 15 Sep 2017 09:38:08 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A88F91E1; Fri, 15 Sep 2017 09:38:08 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id E5BDF20BF0; Fri, 15 Sep 2017 05:31:15 -0400 (EDT) Received: from web6 ([10.202.2.216]) by compute7.internal (MEProxy); Fri, 15 Sep 2017 05:31:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skunkwerks.at; h=cc:content-transfer-encoding:content-type:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-sender:x-me-sender:x-sasl-enc; s=mesmtp; bh=yiU5RfgLSLPW8M tdRPyV4gh48fM2XzLKF7KjGExWbiQ=; b=YyRh0X0XcYsO4npP7dzyjIt4L4kdg/ 9XSRAaB3k9qTcK8hVkSZ6qUPWj1+zRS90kVunbRLUWblLjCEYJau8hahfQ1c5D+L 4Xv89GnAjOI6It1HRt8sbTNA02MZRPNhZ/7PtJ6c3qPF4aP6L9/8TDKVOuS7HYN7 QLw63VtySHeFc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=yiU5Rf gLSLPW8MtdRPyV4gh48fM2XzLKF7KjGExWbiQ=; b=fx1PqEyzlUeWDXvQ5NsooL spS3uydy7WAo6KnDmXZxnSiiGQdqzdGGj2UT4MjGd+DqtrRiT8S3EOZadP0tUBvx fkrKEbWwN855dTXdAhqJuBXiiTsgRNHgaxwoGQqB8IUPmFexV4OsyqxlIEUfcG9i GFcaBi6gYPg7Yf23Xiv10PA/1eFILVZjKAtFRY8gsynwtS5W8/qRrJzOMRHsMMdn DZIXTek3z4vp09AL+e6tIzlw8DzrjFnErqGOW5cixNYrpZkm2VuX4UTKJtmnpHuu aU/msFyHX8zT12RKSxzvGsWFzIRUPbA2dkzLtJhONXOvlf75st9h478cWxE5nTow == X-ME-Sender: Received: by mailuser.nyi.internal (Postfix, from userid 99) id BC03A48004; Fri, 15 Sep 2017 05:31:15 -0400 (EDT) Message-Id: <1505467875.1199722.1107024744.004A81D1@webmail.messagingengine.com> From: Dave Cottlehuber To: Kristof Provost Cc: freebsd-net@freebsd.org, freebsd-pf@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" X-Mailer: MessagingEngine.com Webmail Interface - ajax-64b08692 Subject: Re: NATted outbound traffic sometimes uses backup CARP IP on LACP/LAGG interface In-Reply-To: <4250D983-3CE2-4156-85FB-E580F6C592D2@sigsegv.be> References: <1505398868.955393.1106053824.42CA3E40@webmail.messagingengine.com> <4250D983-3CE2-4156-85FB-E580F6C592D2@sigsegv.be> Date: Fri, 15 Sep 2017 11:31:15 +0200 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Sep 2017 09:38:08 -0000 On Thu, 14 Sep 2017, at 23:32, Kristof Provost wrote: > On 14 Sep 2017, at 16:21, Dave Cottlehuber wrote: > > Outgoing traffic (from a jail) via PF NAT over a LAGG/LACP sometimes=20 > > has the *backup* CARP IP address assigned to it. > > > I think this is your problem. You=E2=80=99re telling pf to nat to the IP= =20 > address of lagg0, but lagg0 has multiple addresses assigned. >=20 > =E2=80=98(lagg0:0)=E2=80=99 should work, or just use the IP address. Thanks Kristof! ($if:0) works perfectly, but I'll need to reboot to test this with the original carp setup though. Can you explain what $if:0 resolves to, for example how does it relate to to the primary ipv4/6 addresses bound to that interface? I couldn't find a reference in the usual ifconfig manpages about this (ifname:#) format, the BNF grammar for pf.conf doesn't cover it either, and `pfctl -vnf ...` simply shows (lagg0:0). A+ Dave