From owner-freebsd-ipfw@freebsd.org Sun Aug 27 01:38:22 2017 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 01A44DE0CBB for ; Sun, 27 Aug 2017 01:38:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E38AF8434F for ; Sun, 27 Aug 2017 01:38:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v7R1cLfJ074935 for ; Sun, 27 Aug 2017 01:38:21 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-ipfw@FreeBSD.org Subject: [Bug 103328] [ipfw] [request] sugestions about ipfw table Date: Sun, 27 Aug 2017 01:38:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jpaetzel@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Overcome By Events X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-ipfw@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Aug 2017 01:38:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D103328 Josh Paetzel changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Progress |Closed Resolution|--- |Overcome By Events CC| |jpaetzel@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-ipfw@freebsd.org Sun Aug 27 01:41:42 2017 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A5C3BDE10B0 for ; Sun, 27 Aug 2017 01:41:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 93E09846C5 for ; Sun, 27 Aug 2017 01:41:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v7R1fguh083126 for ; Sun, 27 Aug 2017 01:41:42 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-ipfw@FreeBSD.org Subject: [Bug 82724] [ipfw] [patch] [request] Add setnexthop and defaultroute features to ipfw2 Date: Sun, 27 Aug 2017 01:41:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 5.4-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jpaetzel@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Overcome By Events X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-ipfw@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Aug 2017 01:41:42 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D82724 Josh Paetzel changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jpaetzel@FreeBSD.org Resolution|--- |Overcome By Events Status|In Progress |Closed --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-ipfw@freebsd.org Sun Aug 27 01:36:28 2017 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 86B18DE0BAC for ; Sun, 27 Aug 2017 01:36:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 748A58427C for ; Sun, 27 Aug 2017 01:36:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v7R1aScn072354 for ; Sun, 27 Aug 2017 01:36:28 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-ipfw@FreeBSD.org Subject: [Bug 46159] [ipfw] [patch] [request] ipfw dynamic rules lifetime feature Date: Sun, 27 Aug 2017 01:36:28 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 4.7-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jpaetzel@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Overcome By Events X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-ipfw@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution cc bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Aug 2017 01:36:28 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D46159 Josh Paetzel changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |Overcome By Events CC| |jpaetzel@FreeBSD.org Status|In Progress |Closed --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-ipfw@freebsd.org Tue Aug 29 09:33:31 2017 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E828DD4791 for ; Tue, 29 Aug 2017 09:33:31 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from hapkido.dreamhost.com (hapkido.dreamhost.com [66.33.216.122]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 65FD67E907 for ; Tue, 29 Aug 2017 09:33:30 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from homiemail-a47.g.dreamhost.com (sub5.mail.dreamhost.com [208.113.200.129]) by hapkido.dreamhost.com (Postfix) with ESMTP id 677CA9176D for ; Tue, 29 Aug 2017 02:33:24 -0700 (PDT) Received: from homiemail-a47.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTP id E316A4411 for ; Tue, 29 Aug 2017 02:33:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=menhennitt.com.au; h=to :from:subject:message-id:date:mime-version:content-type :content-transfer-encoding; s=menhennitt.com.au; bh=Zk2pU18uL3Nt Nyr9LuTOORTFxPw=; b=hlXJnXdyFifRxzClNzNPrBUs+dgQDHnO+OukVaAf0yXE xlcVEZEV7cp25YQKm1xxL5BK/QITl8FEeZDBzVdrPMv1YxCNRqlwqGPU0WnwBPbm laHa5e9lLrnhdC6St13jRNvNduNBiIgabWNBL5IRqDlUtMLn+lvr6vYRDmxexhk= Received: from [203.2.73.68] (c122-107-208-156.mckinn3.vic.optusnet.com.au [122.107.208.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: graham@menhennitt.com.au) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTPSA id 7D4D9440F for ; Tue, 29 Aug 2017 02:33:17 -0700 (PDT) To: freebsd-ipfw@freebsd.org From: Graham Menhennitt Subject: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable Message-ID: Date: Tue, 29 Aug 2017 19:33:15 +1000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Aug 2017 09:33:31 -0000 I have two machines of similar CPU power that I use as routers. One is running 11-Stable as of a week ago and the other is 10-Stable from around the same time. They both run roughly the same IPFW rules (the syntax has changed slightly to run on the newer version). I've been using the 10-Stable box for a number of years without problems. However, the performance on the 11-Stable box is much worse. For file transfers I get about 1/10th the speed. Incoming TLS connections often fail to establish. Looking (from outside the box) at the interface in Wireshark shows lots of packets being retransmitted. This appears to be due to the NAT rule. If I remove that, the performance jumps up to be approximately the same as the 10-Stable box. The rules are pretty simple: nat 1 config if igb1 deny_in same_ports redirect_port udp XXX.XXX.XXX.XXX:YYYY YYYY nat 1 ip4 from any to any via igb1 I can provide the full set of rules if needed, but I think only those two lines are relevant. Does anybody please have any ideas on this, please? Thanks for any help, Graham From owner-freebsd-ipfw@freebsd.org Thu Aug 31 10:06:05 2017 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DEFECE1AE74 for ; Thu, 31 Aug 2017 10:06:05 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from forward1j.cmail.yandex.net (forward1j.cmail.yandex.net [IPv6:2a02:6b8:0:1630::14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 775B571951 for ; Thu, 31 Aug 2017 10:06:05 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from smtp2j.mail.yandex.net (smtp2j.mail.yandex.net [95.108.130.60]) by forward1j.cmail.yandex.net (Yandex) with ESMTP id 9C24320ED8; Thu, 31 Aug 2017 13:05:52 +0300 (MSK) Received: from smtp2j.mail.yandex.net (localhost.localdomain [127.0.0.1]) by smtp2j.mail.yandex.net (Yandex) with ESMTP id C263B3EC0C9F; Thu, 31 Aug 2017 13:05:51 +0300 (MSK) Received: by smtp2j.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id vUMJ7kVQRZ-5oHCWO4f; Thu, 31 Aug 2017 13:05:50 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1504173950; bh=nkjVo4XR4BsgQxoR7dHWq+9oaT3DmydZN0vJlBW5wYE=; h=Subject:To:References:From:Message-ID:Date:In-Reply-To; b=oNXXG8qJ0zfo60o3i7nl68GFhkeYx9ODPE79jHRDrnqQlmqoGmXGw+ZJ9Az6xupPX JltecJlcSWzi/jop0sTH5l/0IhgUx8wJto2L9AHVCAwyLa+gafjU4OHX94eYJBRon3 EEYygKMilL6y3b5bgchWupZCufQIgvYkW0fay89I= Authentication-Results: smtp2j.mail.yandex.net; dkim=pass header.i=@yandex.ru X-Yandex-Suid-Status: 1 0,1 0 Subject: Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable To: Graham Menhennitt , freebsd-ipfw@freebsd.org References: From: "Andrey V. Elsukov" Openpgp: id=E6591E1B41DA1516F0C9BC0001C5EA0410C8A17A Message-ID: Date: Thu, 31 Aug 2017 13:01:43 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="r4FUPnAEhXLAlnLv6E7XOGH8xCDOrltab" X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Aug 2017 10:06:06 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --r4FUPnAEhXLAlnLv6E7XOGH8xCDOrltab Content-Type: multipart/mixed; boundary="hbIqiIrExfMgihIbAfvQxsujbn9oxOKM7"; protected-headers="v1" From: "Andrey V. Elsukov" To: Graham Menhennitt , freebsd-ipfw@freebsd.org Message-ID: Subject: Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable References: In-Reply-To: --hbIqiIrExfMgihIbAfvQxsujbn9oxOKM7 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 29.08.2017 12:33, Graham Menhennitt wrote: > However, the performance on the 11-Stable box is much worse. For file > transfers I get about 1/10th the speed. Incoming TLS connections often > fail to establish. Looking (from outside the box) at the interface in > Wireshark shows lots of packets being retransmitted. >=20 > This appears to be due to the NAT rule. If I remove that, the > performance jumps up to be approximately the same as the 10-Stable box.= > The rules are pretty simple: > nat 1 config if igb1 deny_in same_ports redirect_port udp > XXX.XXX.XXX.XXX:YYYY YYYY > nat 1 ip4 from any to any via igb1 >=20 > I can provide the full set of rules if needed, but I think only those > two lines are relevant. >=20 > Does anybody please have any ideas on this, please? Can you show the output of `ifconfig igb1 | grep flags` on stable/10 and stable/11? --=20 WBR, Andrey V. Elsukov --hbIqiIrExfMgihIbAfvQxsujbn9oxOKM7-- --r4FUPnAEhXLAlnLv6E7XOGH8xCDOrltab Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEzBAEBCAAdFiEE5lkeG0HaFRbwybwAAcXqBBDIoXoFAlmn3ocACgkQAcXqBBDI oXolXQgAqDqR0ZYF98z8E8mFCKxnBWyn7YzSbDkuWuy/2Fk+7F+tlACQaWnuZcoY s/K3pbAWVochDXjobDPN1XQT/kdGtaVV8xv3xVHhKafj6UIyTueX6Ez2FEnrFi7U Ukdi3RwhkM07Nsb9+hSSNljmPkdhpioCWKWqCRSE5DEXm+k4sO69cayxI9YKHord M5g0n1/kh2RS06xyXYI0sXylMXPxqSDZZAP+elkL3gO6uUMCkzZzDcqYWASCC9ur GY+NXxqbqyXY1aJAPP6rLcQUNh4YRoTQ5z4POOD9QjzhmnxPV5CZPdhlba98UXYV 4y45Ti0fvO+qeGqqjEpWJiVfIwZddw== =xh9L -----END PGP SIGNATURE----- --r4FUPnAEhXLAlnLv6E7XOGH8xCDOrltab-- From owner-freebsd-ipfw@freebsd.org Thu Aug 31 10:08:05 2017 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 98E4BE1AF10 for ; Thu, 31 Aug 2017 10:08:05 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from forward3p.cmail.yandex.net (forward3p.cmail.yandex.net [IPv6:2a02:6b8:0:1465::13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 462AE719C5 for ; Thu, 31 Aug 2017 10:08:05 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from smtp4j.mail.yandex.net (smtp4j.mail.yandex.net [5.45.198.129]) by forward3p.cmail.yandex.net (Yandex) with ESMTP id 53FF420EB6; Thu, 31 Aug 2017 13:08:02 +0300 (MSK) Received: from smtp4j.mail.yandex.net (localhost.localdomain [127.0.0.1]) by smtp4j.mail.yandex.net (Yandex) with ESMTP id 6D69D3241090; Thu, 31 Aug 2017 13:07:59 +0300 (MSK) Received: by smtp4j.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id 8bkYgzHB3z-7x0miq81; Thu, 31 Aug 2017 13:07:59 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1504174079; bh=UbBL8PzOuV9qF+GoHBoEyDRtEo95ttnVAUzFYnWblsY=; h=Subject:From:To:References:Message-ID:Date:In-Reply-To; b=dZK7X5s+d5YAD9KX6Qq2oG8SyO9p1q3xF79N3TXWnNyiAiKwPSs5vy/GhmJu3oLzo UT8J36LvGpLvL+tILGNmWIpBnt9jSrKSwNVFeb2RZbfuRvTZYIeFDkWEuU2HCuNvpd pFoxpPM5ZmGRrIfhpdFgdjqZYS/cuLyl4G6F1cxs= Authentication-Results: smtp4j.mail.yandex.net; dkim=pass header.i=@yandex.ru X-Yandex-Suid-Status: 1 0,1 0 Subject: Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable From: "Andrey V. Elsukov" To: Graham Menhennitt , freebsd-ipfw@freebsd.org References: Openpgp: id=E6591E1B41DA1516F0C9BC0001C5EA0410C8A17A Message-ID: <87b38492-da4b-316f-37c2-e1043c2adee4@yandex.ru> Date: Thu, 31 Aug 2017 13:03:55 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="8xwk9Km87ekWne8wQ2pwdclGKX3MmPLTu" X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Aug 2017 10:08:05 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --8xwk9Km87ekWne8wQ2pwdclGKX3MmPLTu Content-Type: multipart/mixed; boundary="C2sTV7hA25VFvpeEA0qEQxtT7b6jrdVoS"; protected-headers="v1" From: "Andrey V. Elsukov" To: Graham Menhennitt , freebsd-ipfw@freebsd.org Message-ID: <87b38492-da4b-316f-37c2-e1043c2adee4@yandex.ru> Subject: Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable References: In-Reply-To: --C2sTV7hA25VFvpeEA0qEQxtT7b6jrdVoS Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 31.08.2017 13:01, Andrey V. Elsukov wrote: >> Does anybody please have any ideas on this, please? >=20 > Can you show the output of `ifconfig igb1 | grep flags` on stable/10 an= d > stable/11? Sorry, I wanted to write `ifconfig igb1 | grep options`. --=20 WBR, Andrey V. Elsukov --C2sTV7hA25VFvpeEA0qEQxtT7b6jrdVoS-- --8xwk9Km87ekWne8wQ2pwdclGKX3MmPLTu Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEzBAEBCAAdFiEE5lkeG0HaFRbwybwAAcXqBBDIoXoFAlmn3wsACgkQAcXqBBDI oXpq6QgAnvKv2RXwwUZIYnOGTu/ih3TllEGK+wbZOgbm8TFZHz4nujNGFlfnsiEC 3RZl6pP20I0lOgM8Llt7YviRP/d5bA30hhnVhWwLpPfNb0VuwFv0RDy0VTsmahDc j9WXtUexPTBxoHs3RpAmJs+pgz0iBgbvXKRjy70eNgQJI6DECKtAKcBwhoA7+/Qd ktcqV46MDMqGzD+21we7Ayc/oECwea0we+Oqg8fKefiXp8dZfxXuKY2pIFKmtTPt GAtgsmFEIgX2Km4ynlWLVxlsv5HmM+8If/lySur4NUL3k/drYeDx5uF7DTz38XVi YrFUQ8RrwkwjVxCAvBTTHHfU6HT/7w== =fmOU -----END PGP SIGNATURE----- --8xwk9Km87ekWne8wQ2pwdclGKX3MmPLTu-- From owner-freebsd-ipfw@freebsd.org Thu Aug 31 12:10:39 2017 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B1C1E1D649 for ; Thu, 31 Aug 2017 12:10:39 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from homiemail-a47.g.dreamhost.com (sub5.mail.dreamhost.com [208.113.200.129]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 71C3D759F9 for ; Thu, 31 Aug 2017 12:10:38 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from homiemail-a47.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTP id 49AF41B02 for ; Thu, 31 Aug 2017 05:10:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=menhennitt.com.au; h= subject:to:references:from:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s= menhennitt.com.au; bh=BbE7UjGXN1h0ZaMz1MbKn5tLvTI=; b=k+ICl5cZmH ZV1jGvqmXA20KSBxSqP4Z3tTYxQl4SLdjVc4lxrNmSqkbL7zmmeHHFKzla81BaCP fDR7EEGkqhu8ypj7vhroSXoe8+3XG2g5eml+x7ZxT4YqpMSlw0nqdImLv/NudL6l L/I+EBnIlZkLxvFylD/Y025OnXu9if2Rc= Received: from [203.2.73.68] (c122-107-208-156.mckinn3.vic.optusnet.com.au [122.107.208.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: graham@menhennitt.com.au) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTPSA id D179E1F39 for ; Thu, 31 Aug 2017 05:10:31 -0700 (PDT) Subject: Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable To: freebsd-ipfw@freebsd.org References: <87b38492-da4b-316f-37c2-e1043c2adee4@yandex.ru> From: Graham Menhennitt Message-ID: <580bc972-7800-96ff-c190-0be176c22d77@menhennitt.com.au> Date: Thu, 31 Aug 2017 22:10:30 +1000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <87b38492-da4b-316f-37c2-e1043c2adee4@yandex.ru> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Aug 2017 12:10:39 -0000 On 31/08/2017 20:03, Andrey V. Elsukov wrote: > On 31.08.2017 13:01, Andrey V. Elsukov wrote: >>> Does anybody please have any ideas on this, please? >> Can you show the output of `ifconfig igb1 | grep flags` on stable/10 >> and stable/11? > Sorry, I wanted to write `ifconfig igb1 | grep options`. Thanks for replying Andrey. On 10-Stable, the interface is re1. The output of 'ifconfig re1 | grep options' is: options=8209b nd6 options=29 On 11-Stable (the one with the problems), it's igb1 and the output of 'ifconfig igb1 | grep options' is: options=6403bb nd6 options=29 Thanks again for your help, Graham From owner-freebsd-ipfw@freebsd.org Thu Aug 31 12:32:14 2017 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BD95DE1DE50 for ; Thu, 31 Aug 2017 12:32:14 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from forward3p.cmail.yandex.net (forward3p.cmail.yandex.net [IPv6:2a02:6b8:0:1465::13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4842B76473 for ; Thu, 31 Aug 2017 12:32:14 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from smtp1p.mail.yandex.net (smtp1p.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b6:6]) by forward3p.cmail.yandex.net (Yandex) with ESMTP id DE70520C88; Thu, 31 Aug 2017 15:32:01 +0300 (MSK) Received: from smtp1p.mail.yandex.net (localhost.localdomain [127.0.0.1]) by smtp1p.mail.yandex.net (Yandex) with ESMTP id 8D96317807BC; Thu, 31 Aug 2017 15:32:00 +0300 (MSK) Received: by smtp1p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id 7rszsGU6kX-VptuiUg6; Thu, 31 Aug 2017 15:31:51 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1504182711; bh=hZSjDf8/YoY3WREGZlaikTzneb9D4euZlaCaeMo8u2o=; h=Subject:To:References:From:Message-ID:Date:In-Reply-To; b=lG5zYd7lbZXIB3iQjR5LZqCEbvN0a2J6/vS41NlT3o38RJ30b0qaUPlrn/bVtdUDq xYzp42hD2FpYkTOfM+1tBuFX7jpshuXYGfe0AT/3ZEgGdnQCFp9eHDcaGdS0OsN/2Z BhNMCt3rxyF8TsYRL+K5gFeLFseivjL3hvdCLg0s= Authentication-Results: smtp1p.mail.yandex.net; dkim=pass header.i=@yandex.ru X-Yandex-Suid-Status: 1 0,1 0 Subject: Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable To: Graham Menhennitt , freebsd-ipfw@freebsd.org References: <87b38492-da4b-316f-37c2-e1043c2adee4@yandex.ru> <580bc972-7800-96ff-c190-0be176c22d77@menhennitt.com.au> From: "Andrey V. Elsukov" Openpgp: id=E6591E1B41DA1516F0C9BC0001C5EA0410C8A17A Message-ID: <40f3bcab-5e0d-0905-ec95-8b4eec8cef89@yandex.ru> Date: Thu, 31 Aug 2017 15:27:47 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <580bc972-7800-96ff-c190-0be176c22d77@menhennitt.com.au> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="howNg4N7XQJ8CTlvjmL9dsNrE1m4DBIb6" X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Aug 2017 12:32:14 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --howNg4N7XQJ8CTlvjmL9dsNrE1m4DBIb6 Content-Type: multipart/mixed; boundary="reIJMrIWrelVH7UhR52rIDiLbSjEpaMG7"; protected-headers="v1" From: "Andrey V. Elsukov" To: Graham Menhennitt , freebsd-ipfw@freebsd.org Message-ID: <40f3bcab-5e0d-0905-ec95-8b4eec8cef89@yandex.ru> Subject: Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable References: <87b38492-da4b-316f-37c2-e1043c2adee4@yandex.ru> <580bc972-7800-96ff-c190-0be176c22d77@menhennitt.com.au> In-Reply-To: <580bc972-7800-96ff-c190-0be176c22d77@menhennitt.com.au> --reIJMrIWrelVH7UhR52rIDiLbSjEpaMG7 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 31.08.2017 15:10, Graham Menhennitt wrote: > On 10-Stable, the interface is re1. The output of 'ifconfig re1 | grep > options' is: > options=3D8209b >=20 > nd6 options=3D29 >=20 > On 11-Stable (the one with the problems), it's igb1 and the output of > 'ifconfig igb1 | grep options' is: > options=3D6403bb >=20 > nd6 options=3D29 >=20 You need to disable TSO on your interface, ipfw nat is not compatible with TCP segmentation offloading (this is noted in ipfw(8) BUGS section).= Try to use: ifconfig igb1 -vlanhwtso -tso4 You can add these option to "ifconfig_igb1" variable in rc.conf. --=20 WBR, Andrey V. Elsukov --reIJMrIWrelVH7UhR52rIDiLbSjEpaMG7-- --howNg4N7XQJ8CTlvjmL9dsNrE1m4DBIb6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEzBAEBCAAdFiEE5lkeG0HaFRbwybwAAcXqBBDIoXoFAlmoAMMACgkQAcXqBBDI oXraEggArt0uH7JKEB38A+3gRnSNLXDHK4riihLpkqEWHLxiTVviITm5gNWtxDCA 9hioArLNVY3NArOzyn4H1WZ9IaecGOhjqPgcqdqZDFidwD9X7uA6oawTNg41o9Bp hkwb3flRiTARzJwQO/zhkUSutmHA181t5Q4RgWu/sszP4mqT/A7Utyw6E/2Lzb6o WQTlMzX5a10u0yHwHXU7u0LBuloIBeJsZsF/TK1dEQ/fWMQ6X/cvrbRRebP5wnDi k9Ma9yk7hGt33CVD5k5t1Md1xvAK0zkRl4xrRQCKdCIm8REzxGERDojgmwTtKtO7 B972nhYw/DpM8e1buY5jMyqg5Qh+lw== =O4lq -----END PGP SIGNATURE----- --howNg4N7XQJ8CTlvjmL9dsNrE1m4DBIb6-- From owner-freebsd-ipfw@freebsd.org Fri Sep 1 07:08:21 2017 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 30574E0E59C for ; Fri, 1 Sep 2017 07:08:21 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D02F16463F for ; Fri, 1 Sep 2017 07:08:19 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id v81788h6062685; Fri, 1 Sep 2017 17:08:08 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Fri, 1 Sep 2017 17:08:08 +1000 (EST) From: Ian Smith To: "Andrey V. Elsukov" cc: Graham Menhennitt , freebsd-ipfw@freebsd.org Subject: Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable In-Reply-To: <40f3bcab-5e0d-0905-ec95-8b4eec8cef89@yandex.ru> Message-ID: <20170901162808.C23641@sola.nimnet.asn.au> References: <87b38492-da4b-316f-37c2-e1043c2adee4@yandex.ru> <580bc972-7800-96ff-c190-0be176c22d77@menhennitt.com.au> <40f3bcab-5e0d-0905-ec95-8b4eec8cef89@yandex.ru> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII Content-ID: <20170901162808.V23641@sola.nimnet.asn.au> X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Sep 2017 07:08:21 -0000 On Thu, 31 Aug 2017 15:27:47 +0300, Andrey V. Elsukov wrote: > On 31.08.2017 15:10, Graham Menhennitt wrote: > > On 10-Stable, the interface is re1. The output of 'ifconfig re1 | grep > > options' is: > > options=8209b > > > > nd6 options=29 > > > > On 11-Stable (the one with the problems), it's igb1 and the output of > > 'ifconfig igb1 | grep options' is: > > options=6403bb > > > > nd6 options=29 > > > > You need to disable TSO on your interface, ipfw nat is not compatible > with TCP segmentation offloading (this is noted in ipfw(8) BUGS section). > > Try to use: > ifconfig igb1 -vlanhwtso -tso4 > > You can add these option to "ifconfig_igb1" variable in rc.conf. Specifically: Due to the architecture of libalias(3), ipfw nat is not compatible with the TCP segmentation offloading (TSO). Thus, to reliably nat your net- work traffic, please disable TSO on your NICs using ifconfig(8). Since natd also uses libalias, does not that also apply when using natd? I forget, and neither libalias(3) nor natd(8) mentions 'tso|TSO'. Since this comes up so often, including on questions@, I'm wondering if an extra test in /etc/rc.d/ipfw at ipfw_prestart() for enablement of either $natd_enable (if applicable) or $firewall_nat_enable could then and there check ifconfig $natd_interface and/or $firewall_nat_interface for the presence of TSO4 and/or VLAN_HWTSO options, and so could warn the user - or just run "ifconfig $iface -vlanhwtso -tso4" directly? While some interfaces such as ngX or pppX need not be up or even exist when starting ipfw, such interfaces should never use TSO anyway? But I'm probably missing something obvious .. cheers, Ian From owner-freebsd-ipfw@freebsd.org Sat Sep 2 00:09:29 2017 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5042BE10517 for ; Sat, 2 Sep 2017 00:09:29 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from homiemail-a47.g.dreamhost.com (sub5.mail.dreamhost.com [208.113.200.129]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 35C0F6A0FB for ; Sat, 2 Sep 2017 00:09:28 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from homiemail-a47.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTP id B25C47322 for ; Fri, 1 Sep 2017 17:09:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=menhennitt.com.au; h= subject:to:references:from:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s= menhennitt.com.au; bh=A26mnkEO0ezTReHLGosp3mzqZlM=; b=jcoYG7L49q 1zaW04L9GPyEQy5L6+DhGY01mAkGMjlG5bVggLPp6P4Ux8QdxQ5zC8pw0uxdSS60 S2fGMqzmqzQVm4E1izSOhfxSPf27+9gIvZNF3a4PFxAvELzEYJjl65DBzy6jGPH+ 6AUaeZx8xJUozrBS/uqCY+P4CK8To8U4w= Received: from [203.2.73.68] (c122-107-208-156.mckinn3.vic.optusnet.com.au [122.107.208.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: graham@menhennitt.com.au) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTPSA id 4A5F17321 for ; Fri, 1 Sep 2017 17:09:22 -0700 (PDT) Subject: Re: ipfw kernel NAT performance much worse in 11-Stable than 10-Stable [SOLVED] To: freebsd-ipfw@freebsd.org References: <87b38492-da4b-316f-37c2-e1043c2adee4@yandex.ru> <580bc972-7800-96ff-c190-0be176c22d77@menhennitt.com.au> <40f3bcab-5e0d-0905-ec95-8b4eec8cef89@yandex.ru> From: Graham Menhennitt Message-ID: <0293e18b-d716-b259-6355-fcb59dfa11ab@menhennitt.com.au> Date: Sat, 2 Sep 2017 10:09:20 +1000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <40f3bcab-5e0d-0905-ec95-8b4eec8cef89@yandex.ru> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Sep 2017 00:09:29 -0000 On 31/08/2017 22:27, Andrey V. Elsukov wrote: > On 31.08.2017 15:10, Graham Menhennitt wrote: >> On 10-Stable, the interface is re1. The output of 'ifconfig re1 | grep >> options' is: >> options=8209b >> >> nd6 options=29 >> >> On 11-Stable (the one with the problems), it's igb1 and the output of >> 'ifconfig igb1 | grep options' is: >> options=6403bb >> >> nd6 options=29 >> > You need to disable TSO on your interface, ipfw nat is not compatible > with TCP segmentation offloading (this is noted in ipfw(8) BUGS section). > > Try to use: > ifconfig igb1 -vlanhwtso -tso4 > > You can add these option to "ifconfig_igb1" variable in rc.conf. > Thanks very much for that Andrey (and Ian). It fixes the performance problem. I did look an the man page for both igb and ipfw but must have missed this. I agree, Ian, it would be good if there was some kind of warning at runtime. So, that fixes the performance problems. I have another problem that I'll send a separate email about. Thanks again, Graham From owner-freebsd-ipfw@freebsd.org Sat Sep 2 01:45:00 2017 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 97D94E15A60 for ; Sat, 2 Sep 2017 01:45:00 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from homiemail-a47.g.dreamhost.com (sub5.mail.dreamhost.com [208.113.200.129]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 801626D350 for ; Sat, 2 Sep 2017 01:45:00 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from homiemail-a47.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTP id 173923B37 for ; Fri, 1 Sep 2017 18:44:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=menhennitt.com.au; h=to :from:subject:message-id:date:mime-version:content-type :content-transfer-encoding; s=menhennitt.com.au; bh=VffgUOs8eF06 N26TYdyPypF4Ybs=; b=QuTpNb2QZ6XshbeoWBBCPmfv/aVCpWFwseW9JV1IoeXf 6Fi/KbYyzGZJYxkUQIrG/pK0blJI3JfFBx2HLJQ86vF5QsCo0002Z5dU7QOUZUgX sDdjBHo9L5vG51+9CZA2kLaOw/ftUKGUCLjY0+3k9UYOZdtTus3umE9udGY/qQw= Received: from [203.2.73.68] (c122-107-208-156.mckinn3.vic.optusnet.com.au [122.107.208.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: graham@menhennitt.com.au) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTPSA id 9948C3B36 for ; Fri, 1 Sep 2017 18:44:53 -0700 (PDT) To: freebsd-ipfw@freebsd.org From: Graham Menhennitt Subject: IPFW NAT behaviour different on 10-Stable versus 11-Stable Message-ID: Date: Sat, 2 Sep 2017 11:44:51 +1000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Sep 2017 01:45:00 -0000 I have a problem that seems to be a difference between ipfw/NAT=20 behaviour in 10-Stable versus 11-Stable. I have two servers: one running=20 10-Stable and one running 11-Stable. I'm using the same rule set on both=20 (see below). It works correctly on 10-Stable but not on 11. The problem is seen on two places: an outgoing SMTP connection on port=20 465, and an incoming to an IMAP server on port 993. In both cases, there=20 are lost packets and retransmissions. See below for a tshark capture of=20 one attempted SMTP session. Setting sysctl net.inet.ip.fw.one_pass to one or zero makes no=20 difference. Deleting the sshguard rule (table 22) makes no difference.=20 Deleting the nat rule makes everything work for this SMTP session (but=20 breaks the other machines on my network obviously). I have no doubt that I have misconfigured the firewall, but I don't see=20 what. And why is 11 different to 10? Any help would be much appreciated. Thanks in advance, Graham Tshark: (XXX is the SMTP server, YYY is my public IP address) root# tshark -Y tcp.port=3D=3D465 -i igb1 Capturing on 'igb1' 4 0.722919 YYY =E2=86=92 XXX TLSv1.2 180 Client Key Exchange, Chan= ge=20 Cipher Spec, Encrypted Handshake Message 527 17.822843 YYY =E2=86=92 XXX TCP 180 [TCP Retransmission] 63024 =E2= =86=92 465=20 [PSH, ACK] Seq=3D1 Ack=3D1 Win=3D65535 Len=3D126 1335 51.814540 YYY =E2=86=92 XXX TCP 180 [TCP Retransmission] 63024 =E2= =86=92 465=20 [PSH, ACK] Seq=3D1 Ack=3D1 Win=3D65535 Len=3D126 1393 85.806537 YYY =E2=86=92 XXX TCP 180 [TCP Retransmission] 63024 =E2= =86=92 465=20 [PSH, ACK] Seq=3D1 Ack=3D1 Win=3D65535 Len=3D126 2142 107.799346 XXX =E2=86=92 YYY TCP 60 465 =E2=86=92 63024 [FIN, ACK]= Seq=3D1 Ack=3D1=20 Win=3D15544 Len=3D0 2143 107.799393 YYY =E2=86=92 XXX TCP 54 63024 =E2=86=92 465 [ACK] Seq=3D= 127 Ack=3D2=20 Win=3D65535 Len=3D0 2144 107.800135 YYY =E2=86=92 XXX TCP 54 63024 =E2=86=92 465 [FIN, ACK]= Seq=3D127 Ack=3D2=20 Win=3D65535 Len=3D0 2145 107.822047 YYY =E2=86=92 XXX TCP 74 53762 =E2=86=92 465 [SYN] Seq=3D= 0 Win=3D65535=20 Len=3D0 MSS=3D1460 WS=3D64 SACK_PERM=3D1 TSval=3D2591962 TSecr=3D0 2146 107.977234 XXX =E2=86=92 YYY TCP 60 465 =E2=86=92 63024 [RST] Seq=3D= 2 Win=3D0 Len=3D0 2149 108.001214 XXX =E2=86=92 YYY TCP 62 465 =E2=86=92 53762 [SYN, ACK]= Seq=3D0 Ack=3D1=20 Win=3D14600 Len=3D0 MSS=3D1460 SACK_PERM=3D1 2150 108.001270 YYY =E2=86=92 XXX TCP 54 53762 =E2=86=92 465 [ACK] Seq=3D= 1 Ack=3D1=20 Win=3D65535 Len=3D0 2151 108.009014 YYY =E2=86=92 XXX TLSv1 323 Client Hello 2160 108.187708 XXX =E2=86=92 YYY TCP 60 465 =E2=86=92 53762 [ACK] Seq=3D= 1 Ack=3D270=20 Win=3D15544 Len=3D0 2176 108.687644 XXX =E2=86=92 YYY TLSv1.2 1514 Server Hello 2177 108.687884 XXX =E2=86=92 YYY TCP 1514 465 =E2=86=92 53762 [PSH, AC= K] Seq=3D1461=20 Ack=3D270 Win=3D15544 Len=3D1460 [TCP segment of a reassembled PDU] 2178 108.687949 YYY =E2=86=92 XXX TCP 54 53762 =E2=86=92 465 [ACK] Seq=3D= 270 Ack=3D2921=20 Win=3D62874 Len=3D0 2179 108.688175 XXX =E2=86=92 YYY TCP 1230 465 =E2=86=92 53762 [PSH, AC= K] Seq=3D2921=20 Ack=3D270 Win=3D15544 Len=3D1176 [TCP segment of a reassembled PDU] 2180 108.704012 XXX =E2=86=92 YYY TCP 1514 465 =E2=86=92 53762 [ACK] Se= q=3D4097 Ack=3D270=20 Win=3D15544 Len=3D1460 [TCP segment of a reassembled PDU] 2181 108.704052 YYY =E2=86=92 XXX TCP 54 53762 =E2=86=92 465 [ACK] Seq=3D= 270 Ack=3D5557=20 Win=3D64240 Len=3D0 2182 108.704625 XXX =E2=86=92 YYY TLSv1.2 969 Certificate, Server Key=20 Exchange, Server Hello Done 2183 108.715222 YYY =E2=86=92 XXX TLSv1.2 180 Client Key Exchange, Chan= ge=20 Cipher Spec, Encrypted Handshake Message 2211 109.133829 YYY =E2=86=92 XXX TCP 180 [TCP Retransmission] 53762 =E2= =86=92 465=20 [PSH, ACK] Seq=3D270 Ack=3D6472 Win=3D65535 Len=3D126 2238 109.443030 XXX =E2=86=92 YYY TCP 969 [TCP Spurious Retransmission]= 465 =E2=86=92=20 53762 [PSH, ACK] Seq=3D5557 Ack=3D270 Win=3D15544 Len=3D915[Reassembly er= ror,=20 protocol TCP: New fragment overlaps old data (retransmission?)] 2239 109.443099 YYY =E2=86=92 XXX TCP 54 [TCP Dup ACK 2183#1] 53762 =E2= =86=92 465=20 [ACK] Seq=3D396 Ack=3D6472 Win=3D65535 Len=3D0 2244 109.772021 YYY =E2=86=92 XXX TCP 180 [TCP Retransmission] 53762 =E2= =86=92 465=20 [PSH, ACK] Seq=3D270 Ack=3D6472 Win=3D65535 Len=3D126 2301 110.827331 YYY =E2=86=92 XXX TCP 180 [TCP Retransmission] 53762 =E2= =86=92 465=20 [PSH, ACK] Seq=3D270 Ack=3D6472 Win=3D65535 Len=3D126 2402 112.770796 YYY =E2=86=92 XXX TCP 180 [TCP Retransmission] 53762 =E2= =86=92 465=20 [PSH, ACK] Seq=3D270 Ack=3D6472 Win=3D65535 Len=3D126 2612 116.391551 YYY =E2=86=92 XXX TCP 180 [TCP Retransmission] 53762 =E2= =86=92 465=20 [PSH, ACK] Seq=3D270 Ack=3D6472 Win=3D65535 Len=3D126 2711 119.018591 YYY =E2=86=92 XXX TCP 180 [TCP Retransmission] 53762 =E2= =86=92 465=20 [PSH, ACK] Seq=3D270 Ack=3D6472 Win=3D65535 Len=3D126 2737 123.957850 YYY =E2=86=92 XXX TCP 180 [TCP Retransmission] 53762 =E2= =86=92 465=20 [PSH, ACK] Seq=3D270 Ack=3D6472 Win=3D65535 Len=3D126 2789 133.632511 YYY =E2=86=92 XXX TCP 180 [TCP Retransmission] 53762 =E2= =86=92 465=20 [PSH, ACK] Seq=3D270 Ack=3D6472 Win=3D65535 Len=3D126 2859 152.776509 YYY =E2=86=92 XXX TCP 180 [TCP Retransmission] 53762 =E2= =86=92 465=20 [PSH, ACK] Seq=3D270 Ack=3D6472 Win=3D65535 Len=3D126 ^C32 packets captured root# Rules: # stop spoofing add deny all from LAN_NET to any in via OUTSIDE_IF add deny all from WIFI_NET to any in via OUTSIDE_IF # allow anything on the LAN add allow all from any to any via LAN_IF # and from the VPN add allow all from any to any via VPN_IF # allow anything from the wireless network to the outside world (but not=20 to the LAN) add allow ip from any to not LAN_NET via WIFI_IF # create a table of addresses to block #table 1 destroy #table 1 create type addr table 1 flush # add RFC1918 nets table 1 add 10.0.0.0/8 table 1 add 172.16.0.0/12 table 1 add 192.168.0.0/16 # and draft-manning-dsua-03.txt nets table 1 add 0.0.0.0/8 table 1 add 169.254.0.0/16 table 1 add 192.0.2.0/24 table 1 add 224.0.0.0/4 table 1 add 240.0.0.0/4 # stop entries in the table coming in on the outside interface add deny all from table(1) to any in recv OUTSIDE_IF # similarly for IPv6 #table 2 destroy #table 2 create type addr table 2 flush # Stop unique local unicast address on the outside interface table 2 add fc00::/7 # Stop site-local on the outside interface table 2 add fec0::/10 # Disallow "internal" addresses to appear on the wire. table 2 add ::ffff:0.0.0.0/96 # Disallow packets to malicious IPv4 compatible prefix. #table 2 add ::224.0.0.0/100 gives error "Use IPv4 instead of v4-compatib= le" #table 2 add ::127.0.0.0/104 ditto table 2 add ::0.0.0.0/104 #table 2 add ::255.0.0.0/104 ditto # table 2 add ::0.0.0.0/96 # Disallow packets to malicious 6to4 prefix. table 2 add 2002:e000::/20 table 2 add 2002:7f00::/24 table 2 add 2002:0000::/24 table 2 add 2002:ff00::/24 # table 2 add 2002:0a00::/24 table 2 add 2002:ac10::/28 table 2 add 2002:c0a8::/32 # table 2 add ff05::/16 # block these addresses both incoming and outgoing add deny all from table(2) to any via IPV6_IF add deny all from any to table(2) via IPV6_IF # block sshguard entries add reset ip from table(22) to me # allow setup of incoming SSH, IMAPS, and OpenVPN add allow tcp from any to me ssh setup add allow tcp from any to me6 ssh setup add allow tcp from any to me imaps setup add allow tcp from any to me6 imaps setup add allow tcp from any to me openvpn setup add allow tcp from any to me6 openvpn setup add allow udp from any to me openvpn # allow IPP, IMAPS, and SMTP from wireless add allow ip from any to LAN_NET dst-port printer setup via WIFI_IF add allow ip from any to me dst-port ipp setup via WIFI_IF add allow ip from any to me dst-port smtp setup via WIFI_IF add allow ip from any to me dst-port imaps setup via WIFI_IF # allow some ICMP types but nothing else add allow icmp from any to any icmptypes 0,3,8,11 add deny icmp from any to any #add allow ipv6 from any to any # NAT # redirect ports to PS4 nat 1 config if OUTSIDE_IF same_ports deny_in redirect_port tcp=20 PS4_ADDR:1935 1935 redirect_port tcp PS4_ADDR:3478 3478 redirect_port=20 tcp PS4_ADDR:3479 3479 redirect_port tcp PS4_ADDR:3480 3480=20 redirect_port udp PS4_ADDR:3478 3478 redirect_port udp PS4_ADDR:3479 3479 add nat 1 ip4 from any to any via OUTSIDE_IF # and block the above table again outbound add deny all from table(1) to any out xmit OUTSIDE_IF # allow TCP through if setup succeeded add pass tcp from any to any established # allow IP fragments to pass through add pass all from any to any frag # allow TCP ports needed for PS4 add allow tcp from any to PS4_ADDR 1935 in via OUTSIDE_IF setup add allow tcp from any to PS4_ADDR 3478 in via OUTSIDE_IF setup add allow tcp from any to PS4_ADDR 3479 in via OUTSIDE_IF setup add allow tcp from any to PS4_ADDR 3480 in via OUTSIDE_IF setup add allow udp from any to PS4_ADDR 3478 in via OUTSIDE_IF add allow udp from any to PS4_ADDR 3479 in via OUTSIDE_IF # allow DNS & NTP queries out to the world (and their replies back in) add allow udp from me to any 53 keep-state add allow udp from me to any 123 keep-state # but no other UDP in from outside add deny udp from any to any in via OUTSIDE_IF # and allow any other UDP add allow udp from any to any # reject all setup of incoming connections from the outside add deny tcp from any to any in via OUTSIDE_IF setup # reject all setup of incoming connections from the IPV6 tunnel add deny tcp from any to any in via gif0 setup # reject all setup of incoming connections from the wireless add deny tcp from any to any in via WIFI_IF setup # allow setup of any other TCP connection add pass tcp from any to any setup # Everything else is denied by default, unless the=20 IPFIREWALL_DEFAULT_TO_ACCEPT option is set in your kernel config file.=20 But we add this rule anyway to allow logging. add deny all from any to any From owner-freebsd-ipfw@freebsd.org Sat Sep 2 10:46:32 2017 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C6EEAE0C759 for ; Sat, 2 Sep 2017 10:46:32 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3D63081705 for ; Sat, 2 Sep 2017 10:46:31 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id v82AkKgx019250; Sat, 2 Sep 2017 20:46:20 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Sat, 2 Sep 2017 20:46:19 +1000 (EST) From: Ian Smith To: Graham Menhennitt cc: freebsd-ipfw@freebsd.org Subject: Re: IPFW NAT behaviour different on 10-Stable versus 11-Stable In-Reply-To: Message-ID: <20170902202655.T23641@sola.nimnet.asn.au> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Sep 2017 10:46:32 -0000 On Sat, 2 Sep 2017 11:44:51 +1000, Graham Menhennitt wrote: > I have a problem that seems to be a difference between ipfw/NAT > behaviour in 10-Stable versus 11-Stable. I have two servers: one running > 10-Stable and one running 11-Stable. I'm using the same rule set on both > (see below). It works correctly on 10-Stable but not on 11. > > The problem is seen on two places: an outgoing SMTP connection on port > 465, and an incoming to an IMAP server on port 993. In both cases, there > are lost packets and retransmissions. See below for a tshark capture of > one attempted SMTP session. > > Setting sysctl net.inet.ip.fw.one_pass to one or zero makes no > difference. Deleting the sshguard rule (table 22) makes no difference. > Deleting the nat rule makes everything work for this SMTP session (but > breaks the other machines on my network obviously). > > I have no doubt that I have misconfigured the firewall, but I don't see > what. And why is 11 different to 10? Any help would be much appreciated. > > Thanks in advance, > > Graham Mysterious. Unless this is some other networking issue, three thoughts: 1) given that YYY is your public IP address, are the problematic SMTP sessions actually going through NAT at all, or are they initiated from YYY directly? If the latter, it's hard to see why removing the NAT rule should affect these session at all? 2) does it make any difference if you split the NAT rules into separate rules, as per the ipfw(8) 'NAT, REDIRECT AND LSNAT' section in EXAMPLES? 3) given the tokens used in your ruleset, it appears that you are using a preproceesor to substitute values rather than shell variables? If so (or even if not) can you confirm that the resulting in-place rulesets shown by 'ipfw list' are absolutely identical on both machines? Just some long shots .. cheers, Ian From owner-freebsd-ipfw@freebsd.org Sat Sep 2 22:54:58 2017 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 50F07E08C77 for ; Sat, 2 Sep 2017 22:54:58 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from homiemail-a47.g.dreamhost.com (sub5.mail.dreamhost.com [208.113.200.129]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 380967376E for ; Sat, 2 Sep 2017 22:54:57 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from homiemail-a47.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTP id 029181F3C; Sat, 2 Sep 2017 15:54:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=menhennitt.com.au; h= subject:to:cc:references:from:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s= menhennitt.com.au; bh=4kdgJEBgmok3RwLHHzPFewlLmbw=; b=iLDWqbOlSa WvejuKEyvZjq/0qWF0J+YxhUDq/L8Abw121OICOTKIWbVnefSOqVH3ePmw/GvhGr ATK58Y67bYLpnJfuzT+ICvA8eFtw3uqzAMHhW7OCnGEq/vQoT49jPI/yGLqyYijl u/mnqyy3UjHYA05L/+ZlmEPL8inG2arns= Received: from [203.2.73.68] (c122-107-208-156.mckinn3.vic.optusnet.com.au [122.107.208.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: graham@menhennitt.com.au) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTPSA id 57BD81F39; Sat, 2 Sep 2017 15:54:55 -0700 (PDT) Subject: Re: IPFW NAT behaviour different on 10-Stable versus 11-Stable [SOLVED] To: Ian Smith Cc: freebsd-ipfw@freebsd.org References: <20170902202655.T23641@sola.nimnet.asn.au> From: Graham Menhennitt Message-ID: <026e695f-4fb7-7c86-fddb-e49ccdcbdcda@menhennitt.com.au> Date: Sun, 3 Sep 2017 08:54:53 +1000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <20170902202655.T23641@sola.nimnet.asn.au> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Sep 2017 22:54:58 -0000 On 02/09/2017 20:46, Ian Smith wrote: > On Sat, 2 Sep 2017 11:44:51 +1000, Graham Menhennitt wrote: > > > I have a problem that seems to be a difference between ipfw/NAT > > behaviour in 10-Stable versus 11-Stable. I have two servers: one running > > 10-Stable and one running 11-Stable. I'm using the same rule set on both > > (see below). It works correctly on 10-Stable but not on 11. > > > > The problem is seen on two places: an outgoing SMTP connection on port > > 465, and an incoming to an IMAP server on port 993. In both cases, there > > are lost packets and retransmissions. See below for a tshark capture of > > one attempted SMTP session. > > > > Setting sysctl net.inet.ip.fw.one_pass to one or zero makes no > > difference. Deleting the sshguard rule (table 22) makes no difference. > > Deleting the nat rule makes everything work for this SMTP session (but > > breaks the other machines on my network obviously). > > > > I have no doubt that I have misconfigured the firewall, but I don't see > > what. And why is 11 different to 10? Any help would be much appreciated. > > > > Thanks in advance, > > > > Graham > > Mysterious. Unless this is some other networking issue, three thoughts: > > 1) given that YYY is your public IP address, are the problematic SMTP > sessions actually going through NAT at all, or are they initiated from > YYY directly? If the latter, it's hard to see why removing the NAT rule > should affect these session at all? > > 2) does it make any difference if you split the NAT rules into separate > rules, as per the ipfw(8) 'NAT, REDIRECT AND LSNAT' section in EXAMPLES? > > 3) given the tokens used in your ruleset, it appears that you are using > a preproceesor to substitute values rather than shell variables? If so > (or even if not) can you confirm that the resulting in-place rulesets > shown by 'ipfw list' are absolutely identical on both machines? > > Just some long shots .. > > cheers, Ian Thanks for replying, Ian. Well I solved it. Similarly to my previous problem, the solution was to disable the TXCSUM option on the interface. So, now the entry in /etc/rc.conf says: ifconfig_igb1="DHCP -vlanhwtso -tso4 -txcsum" And it all works. Thanks again, Graham