From owner-freebsd-pf@FreeBSD.ORG Sun Jun 21 11:19:58 2015 Return-Path: Delivered-To: freebsd-pf@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0C9CC39A for ; Sun, 21 Jun 2015 11:19:58 +0000 (UTC) (envelope-from ian.freislich@capeaugusta.com) Received: from mail-wi0-f182.google.com (mail-wi0-f182.google.com [209.85.212.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9C531B2D for ; Sun, 21 Jun 2015 11:19:57 +0000 (UTC) (envelope-from ian.freislich@capeaugusta.com) Received: by wiwl6 with SMTP id l6so13705131wiw.0 for ; Sun, 21 Jun 2015 04:19:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references :mime-version:content-type:content-id:date:message-id; bh=cUzKO83F9TXzMBrMQQJl2m+oQ45Z+Ocg/lub/bTzYI0=; b=S+mYJ/icy116xVp9Lyl8YlGTZ0fTrnOmmerOyz+MtRTV400omS9W5kwFAWKmh+e3WP YcFIO06H+JjxIqo1PLyGtIl0g4+MGBUSmtg1bxfqwY+TiCqiaNbuoPlWcPcE1gxeeVNr 4LOSRoye+jg+c239vUCbhAo8hlLR/jhfaJ2LnX70h+/9ihLMP3IbqXpsgoyT3eB84sWr gtkQ0CHOoEH/efYXRI+jKWNwdwK+ni712XffkgsstnGvhmDpQ9tiJA1vLtMzcJXfutPD mLglT/Ivg9u5xqSSnf+VQRkC20UVGslSyQc1nWtyJMAMnjit5wQ+9w8e2J3okwioHODt BBQA== X-Gm-Message-State: ALoCoQlQ1AQTFyDxBRGn5ETtdhRgscDcX2z5jjtrBaP9g/86e4FLxT4K0sJ/lDsGLPb2nHq4/goI X-Received: by 10.180.215.101 with SMTP id oh5mr23160698wic.6.1434885595424; Sun, 21 Jun 2015 04:19:55 -0700 (PDT) Received: from clue.co.za ([197.89.156.54]) by mx.google.com with ESMTPSA id gw7sm12009261wib.15.2015.06.21.04.19.54 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 21 Jun 2015 04:19:54 -0700 (PDT) From: Ian FREISLICH X-Google-Original-From: Ian FREISLICH Received: from localhost ([127.0.0.1] helo=zen) by clue.co.za with esmtp (Exim 4.85 (FreeBSD)) (envelope-from ) id 1Z6dHz-0000uu-D8; Sun, 21 Jun 2015 07:19:51 -0400 To: Milan Obuch cc: freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem In-Reply-To: <20150620182432.62797ec5@zeta.dino.sk> References: <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> X-Attribution: BOFH MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <3526.1434885591.1@zen> Date: Sun, 21 Jun 2015 07:19:51 -0400 Message-Id: X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jun 2015 11:19:58 -0000 Milan Obuch wrote: > Ian FREISLICH wrote: > > > How many NAT states in your table? > > How can I find out? Is there another statistics collected I can gert > out of pfctl? pfctl -s nat -v Ian -- Ian Freislich From owner-freebsd-pf@FreeBSD.ORG Sun Jun 21 11:32:41 2015 Return-Path: Delivered-To: freebsd-pf@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 941F97BF for ; Sun, 21 Jun 2015 11:32:41 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 11E19F34 for ; Sun, 21 Jun 2015 11:32:40 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Sun, 21 Jun 2015 13:32:36 +0200 id 00EB0AC8.5586A0D4.00016167 Date: Sun, 21 Jun 2015 13:32:36 +0200 From: Milan Obuch To: Ian FREISLICH Cc: freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150621133236.75a4d86d@zeta.dino.sk> In-Reply-To: References: <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jun 2015 11:32:41 -0000 On Sun, 21 Jun 2015 07:19:51 -0400 Ian FREISLICH wrote: > Milan Obuch wrote: > > Ian FREISLICH wrote: > > > > > How many NAT states in your table? > > > > How can I find out? Is there another statistics collected I can gert > > out of pfctl? > > pfctl -s nat -v > > Ian > My nat rule evaluates into 12 nat 'paragraphs' in this listing, totalling around 19500 states, plus 4 small nat's with one state, plus 50 binat's with total 1000 states approx. One observation, on pfctl -vs info output - when src-limit counters rises to 30 or so, I am getting first messages someone has problem. Is it only coincidence or is there really some relation to my problem? Also, could there be some known bug in pf code, which could explain the behaviour I see? Just for completeness, my system is actually i386 9.3-STABLE #0 r276659: Sun Jan 4 16:36:17, I have 2 GB RAM in my system. Regards, Milan From owner-freebsd-pf@FreeBSD.ORG Sun Jun 21 12:38:10 2015 Return-Path: Delivered-To: freebsd-pf@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 98F5A930 for ; Sun, 21 Jun 2015 12:38:10 +0000 (UTC) (envelope-from ian.freislich@capeaugusta.com) Received: from mail-wi0-f179.google.com (mail-wi0-f179.google.com [209.85.212.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 344B014E for ; Sun, 21 Jun 2015 12:38:10 +0000 (UTC) (envelope-from ian.freislich@capeaugusta.com) Received: by wicgi11 with SMTP id gi11so52848673wic.0 for ; Sun, 21 Jun 2015 05:38:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references :mime-version:content-type:content-id:date:message-id; bh=3a7RJzWkCzmVvRIoHAQrT73CKgH3wIDLdeZBp+s/mUs=; b=YMFdJ07Y+rT8AhFsQW97dHIJOlNFt3QdRUtsxkaWwe8fH8YE7BT08qipst2TTWm1vy 7GK0YT2MH+JI+9nBGtrWdK6o/QLNI6HoFluJgUi6AP/tLJlfl5pCOxu1qPEP3yZgjpwl On64vFs399KFunf+kjs8QtIh5XxjtztEJL6x1pTZG10eLAHBnlfQx/a9uTK0P5PUtYYh WSYloJfcWszibEoFKMQnHhe0ALF20FNrD1S8lwEFaD78/chJvjFfTaAJ2NGkP0l11Pab TTt2YMfkKu/hWKcugsRfWa77CVbBvBQm4PoxHMHLpS44Kuec+ivnI+Yo5fotdPkuDZIn 2oJw== X-Gm-Message-State: ALoCoQmBxukLpaCnYla4tUHdpUlv9pBsy25YLyftzEQ46/jqfHy+d/Tb/aHtKKVzj3AMgWtgtN45 X-Received: by 10.194.176.68 with SMTP id cg4mr42681105wjc.106.1434890288440; Sun, 21 Jun 2015 05:38:08 -0700 (PDT) Received: from clue.co.za ([197.89.156.54]) by mx.google.com with ESMTPSA id 12sm25561541wjw.17.2015.06.21.05.38.07 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 21 Jun 2015 05:38:07 -0700 (PDT) From: Ian FREISLICH X-Google-Original-From: Ian FREISLICH Received: from localhost ([127.0.0.1] helo=zen) by clue.co.za with esmtp (Exim 4.85 (FreeBSD)) (envelope-from ) id 1Z6eVg-0000yz-Ar; Sun, 21 Jun 2015 08:38:04 -0400 To: Milan Obuch cc: freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem In-Reply-To: <20150621133236.75a4d86d@zeta.dino.sk> References: <20150621133236.75a4d86d@zeta.dino.sk> <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> X-Attribution: BOFH MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <3779.1434890284.1@zen> Date: Sun, 21 Jun 2015 08:38:04 -0400 Message-Id: X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jun 2015 12:38:10 -0000 Milan Obuch wrote: > On Sun, 21 Jun 2015 07:19:51 -0400 > Ian FREISLICH wrote: > > > Milan Obuch wrote: > > > Ian FREISLICH wrote: > > > > > > > How many NAT states in your table? > > > > > > How can I find out? Is there another statistics collected I can gert > > > out of pfctl? > > > > pfctl -s nat -v > > > > Ian > > > > My nat rule evaluates into 12 nat 'paragraphs' in this listing, > totalling around 19500 states, plus 4 small nat's with one state, plus > 50 binat's with total 1000 states approx. That's not many states. > One observation, on pfctl -vs info output - when src-limit counters > rises to 30 or so, I am getting first messages someone has problem. Is > it only coincidence or is there really some relation to my problem? Perhaps. These are the options I had set. You probably don't want the if-bound one. # Options # ~~~~~~~ set timeout { \ adaptive.start 900000, \ adaptive.end 1800000 \ } set block-policy return set state-policy if-bound set optimization normal set ruleset-optimization basic set limit states 1500000 set limit frags 40000 set limit src-nodes 150000 --- /etc/sysctl.conf --- net.inet.ip.fastforwarding=1 --- I also had some other settings regarding interrupt moderation on the NIC, netisr threads, queue depth and dispatch. I disabled entropy harvesting on interrupts, and the network path. Some of these settings are loader.conf settings, some are runtime sysctls. I still think that if it's possible, you should give 10-STABLE a try. Ian -- Ian Freislich From owner-freebsd-pf@FreeBSD.ORG Sun Jun 21 14:43:44 2015 Return-Path: Delivered-To: freebsd-pf@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A28E721B for ; Sun, 21 Jun 2015 14:43:44 +0000 (UTC) (envelope-from vegeta@tuxpowered.net) Received: from mail-wi0-f171.google.com (mail-wi0-f171.google.com [209.85.212.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3E79B326 for ; Sun, 21 Jun 2015 14:43:43 +0000 (UTC) (envelope-from vegeta@tuxpowered.net) Received: by wicnd19 with SMTP id nd19so55634124wic.1 for ; Sun, 21 Jun 2015 07:43:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:organization :user-agent:in-reply-to:references:mime-version:content-type; bh=asAtg+fkiqj0HluSa5+ipAamF7R5JFIWQY4aRiXOYS4=; b=WWHPYOOVx5SMHJP8H5Uc2zkzVp/BESZBV9ss9aKh71p9+Hl6OPb5vDqU+494igAFbb rxq1MF5udXdKJP54STJPj0PhYt9Vk8cm1RCHjRv6IpD95URo3EfHFk8pdu8rFIMW/F9Q 3jG1Of6QnP4aKuVKQ+9AMe06kvo9Kjbcx14uy/wsMn9rvLkWEZlJnDqI+ILjf4lX5ANX h1pnlYQzkfSzdAVJ8BBt8mi5TihMPI7MM8HyscJqOFJjvaE1AfvhUmX8IqXJbUi8PFjS SdeI7KlRuwsTC1QjkTMh7322Ff2y1et0fu+1xaaMMrtoDZCDWUkCkjsaxreXOtHGeDuZ mTrA== X-Gm-Message-State: ALoCoQkucI6fvipfcjIUHCgFDuFl+0H+C3QdTS8L34/wD1jVR/qnlE2HB6ceMRY0AahBN3YNKpb7 X-Received: by 10.194.109.36 with SMTP id hp4mr42304194wjb.4.1434897822120; Sun, 21 Jun 2015 07:43:42 -0700 (PDT) Received: from energia.localnet ([37.80.202.85]) by mx.google.com with ESMTPSA id gz3sm17713374wib.0.2015.06.21.07.43.40 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 21 Jun 2015 07:43:41 -0700 (PDT) From: Kajetan Staszkiewicz To: "Chuck @ Mantis" Cc: freebsd-pf@freebsd.org Subject: Re: adding an additional block & gateway Date: Sun, 21 Jun 2015 16:43:37 +0200 Message-ID: <2646372.HZAPXgGrRl@energia> Organization: tuxpowered.net User-Agent: KMail/4.14.1 (Linux/3.19.0-trunk-amd64; KDE/4.14.2; x86_64; ; ) In-Reply-To: <55843762.3040106@mantis.biz> References: <55839619.8000603@mantis.biz> <1704069.kZvlBVo68Y@energia> <55843762.3040106@mantis.biz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart12590657.O1HbZtgWJA"; micalg="pgp-sha1"; protocol="application/pgp-signature" X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jun 2015 14:43:44 -0000 --nextPart12590657.O1HbZtgWJA Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Dnia pi=C4=85tek, 19 czerwca 2015 11:38:10 Chuck @ Mantis pisze: > Our data center responded to your question, here is the text: >=20 > We can confirm that the new netblock is routed direct via your vlan a= s with > your original netblock >=20 > VLAN: vlan655-cbcbmedi-809, Created at: Mon Oct 20 13:42:05 2014 > 802.1Q Tag: 655, Internal index: 205, Admin State: Enabled, Origin: S= tatic > Layer 3 interface: vlan.655 (UP) > IPV4 addresses: > 60.34.75.209/28 > 79.112.227.33/27 > Protocol: Port Mode, Mac aging time: 300 seconds > Number of interfaces: Tagged 0 (Active =3D 0), Untagged 1 (Active =3D= 1) > ge-5/0/20.0*, untagged, access >=20 Just add another subnet and hosts in it (carp?) to the usual interface = and use=20 the existing gateway, it should work just fine. Both gateways are the s= ame=20 device which will happily accept traffic from any source. Or start using real routing. =2D-=20 | pozdrawiam / greetings | powered by Debian, FreeBSD and CentOS | | Kajetan Staszkiewicz | jabber,email: vegeta()tuxpowered net | | Vegeta | www: http://vegeta.tuxpowered.net | `------------------------^---------------------------------------' --nextPart12590657.O1HbZtgWJA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part. Content-Transfer-Encoding: 7Bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEUEABECAAYFAlWGzZsACgkQ47RQr217OhQm3gCYoduyH1QvaddZ/XeIsL0/LaUa DwCeO7lpKV9MjAls9dfqGytQawxdgTw= =7KWS -----END PGP SIGNATURE----- --nextPart12590657.O1HbZtgWJA-- From owner-freebsd-pf@FreeBSD.ORG Sun Jun 21 17:58:05 2015 Return-Path: Delivered-To: freebsd-pf@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BF81BE25 for ; Sun, 21 Jun 2015 17:58:05 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 55B8C3F6 for ; Sun, 21 Jun 2015 17:58:04 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Sun, 21 Jun 2015 19:57:54 +0200 id 00EB08DC.5586FB22.00017FD7 Date: Sun, 21 Jun 2015 19:57:53 +0200 From: Milan Obuch To: Ian FREISLICH Cc: freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150621195753.7b162633@zeta.dino.sk> In-Reply-To: References: <20150621133236.75a4d86d@zeta.dino.sk> <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jun 2015 17:58:05 -0000 On Sun, 21 Jun 2015 08:38:04 -0400 Ian FREISLICH wrote: [ snip ] > > One observation, on pfctl -vs info output - when src-limit counters > > rises to 30 or so, I am getting first messages someone has problem. > > Is it only coincidence or is there really some relation to my > > problem? > > Perhaps. These are the options I had set. You probably don't want > the if-bound one. > Well, it hit me again, and I am not sure there is any relation. Anyway, I tried pfctl -F with various classes, even all, but nothing helped. Only after /etc/rc.d/pf restart affected clients begun to work again. > # Options > # ~~~~~~~ > set timeout { \ > adaptive.start 900000, \ > adaptive.end 1800000 \ > } > set block-policy return > set state-policy if-bound > set optimization normal > set ruleset-optimization basic > set limit states 1500000 > set limit frags 40000 > set limit src-nodes 150000 > > --- /etc/sysctl.conf --- > net.inet.ip.fastforwarding=1 > --- > I think I have set it up similarly. I think there could be some bug hitting me, but no idea how to check and what. > I also had some other settings regarding interrupt moderation on > the NIC, netisr threads, queue depth and dispatch. I disabled > entropy harvesting on interrupts, and the network path. Some of > these settings are loader.conf settings, some are runtime sysctls. > > I still think that if it's possible, you should give 10-STABLE a > try. > This will take some time to do. Unfortunatelly, I did not think about possibilities to test various version when the system was installed. My bad. Now it is not easy, but I am trying to find usable way to do it. Regards, Milan From owner-freebsd-pf@FreeBSD.ORG Tue Jun 23 05:39:14 2015 Return-Path: Delivered-To: freebsd-pf@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C69097E7 for ; Tue, 23 Jun 2015 05:39:14 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5CD9D340 for ; Tue, 23 Jun 2015 05:39:13 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Tue, 23 Jun 2015 07:39:03 +0200 id 00EB081E.5588F0F7.0000EACB Date: Tue, 23 Jun 2015 07:38:56 +0200 From: Milan Obuch To: Ian FREISLICH Cc: freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150623073856.334ebd61@zeta.dino.sk> In-Reply-To: <20150621195753.7b162633@zeta.dino.sk> References: <20150621133236.75a4d86d@zeta.dino.sk> <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 05:39:14 -0000 On Sun, 21 Jun 2015 19:57:53 +0200 Milan Obuch wrote: > On Sun, 21 Jun 2015 08:38:04 -0400 > Ian FREISLICH wrote: > [ snip ] > > I also had some other settings regarding interrupt moderation on > > the NIC, netisr threads, queue depth and dispatch. I disabled > > entropy harvesting on interrupts, and the network path. Some of > > these settings are loader.conf settings, some are runtime sysctls. > > > > I still think that if it's possible, you should give 10-STABLE a > > try. > > > > This will take some time to do. Unfortunatelly, I did not think about > possibilities to test various version when the system was installed. > My bad. Now it is not easy, but I am trying to find usable way to do > it. > > Regards, > Milan > As a first step, I did small upgrade, so now I run FreeBSD 9.3-STABLE #0 r284695: Mon Jun 22 08:55:29 CEST 2015. I still see the issue, but I found simpler workaround when bad state ocurs - using pfctl -k pfctl -K in this order seems to remedy the issue for this one affected client without affecting other clients. This still does not solve the problem, just eases the reaction. Also, not sure yet, but it seems when it occurs, if more clients are natted using the same public IP, all are affected the same way. Using mentioned workaround for all of them makes them all work again. Regards, Milan From owner-freebsd-pf@FreeBSD.ORG Tue Jun 23 07:50:09 2015 Return-Path: Delivered-To: freebsd-pf@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6ED74FE7 for ; Tue, 23 Jun 2015 07:50:09 +0000 (UTC) (envelope-from ian.freislich@capeaugusta.com) Received: from mail-wg0-f46.google.com (mail-wg0-f46.google.com [74.125.82.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 01860BF9 for ; Tue, 23 Jun 2015 07:50:08 +0000 (UTC) (envelope-from ian.freislich@capeaugusta.com) Received: by wgck11 with SMTP id k11so1965115wgc.0 for ; Tue, 23 Jun 2015 00:50:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references :mime-version:content-type:content-id:content-transfer-encoding:date :message-id; bh=LuF0ROSMtuLLzHJpd7+H6Bw7XSlvnFGS8Axnmh0LjqU=; b=lTp0zsH/VWyuhWb51aZzP7WcWOgeFND8KsM0yIcJGRmOpp65LER89Y080bBixRE9JK 5h1cnNU3GuvcwuxK9mc3sUR9c3qbngqCdhRJtdzpAlH87N2nbuTDeVQJ/tSPsh/q0BsJ a2JeWieFH9zUZxnEd5cDZyyzfd3p+YhdUxs1iW/hmNPTshASvILBpYlVOmW/aNb7mA7K xcbmNReIfGLaiJOu5YOQSBk0lzk800Yq6nfFBUY7BPGup8qP51fYGNCPHqvfGc6vj2tv T4JwfTrG/iuEQH/7TNkzDC9FhfOoKmhvFIMrffROmxgxmmHPle1WfqtLMDKBJ+duIZV2 LX5A== X-Gm-Message-State: ALoCoQkpajmNNS+u9xS8aLsp0D3TfrPGJ2ZhF0GB5e0d14eYqGZFI/P2EE4eYNrgPlmqJ+V/AYZS X-Received: by 10.180.91.76 with SMTP id cc12mr890995wib.67.1435045801485; Tue, 23 Jun 2015 00:50:01 -0700 (PDT) Received: from clue.co.za ([197.89.156.54]) by mx.google.com with ESMTPSA id ul1sm34292806wjc.30.2015.06.23.00.49.59 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Jun 2015 00:50:00 -0700 (PDT) From: Ian FREISLICH X-Google-Original-From: Ian FREISLICH Received: from localhost ([127.0.0.1] helo=zen) by clue.co.za with esmtp (Exim 4.85 (FreeBSD)) (envelope-from ) id 1Z7Ixx-0006K1-5p; Tue, 23 Jun 2015 03:49:57 -0400 To: Milan Obuch cc: freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem In-Reply-To: <20150623073856.334ebd61@zeta.dino.sk> References: <20150623073856.334ebd61@zeta.dino.sk> <20150621133236.75a4d86d@zeta.dino.sk> <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> X-Attribution: BOFH MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <24303.1435045797.1@zen> Content-Transfer-Encoding: quoted-printable Date: Tue, 23 Jun 2015 09:49:57 +0200 Message-Id: X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 07:50:09 -0000 Milan Obuch wrote: > As a first step, I did small upgrade, so now I run FreeBSD 9.3-STABLE > #0 r284695: Mon Jun 22 08:55:29 CEST 2015. > = > I still see the issue, but I found simpler workaround when bad state > ocurs - using > = > pfctl -k > pfctl -K > = > in this order seems to remedy the issue for this one affected client > without affecting other clients. This still does not solve the problem, > just eases the reaction. How is your NAT rule defined? I had a closer look at the way I did it: nat on vlan46 from 10.8.0.0/15 to ! -> xx.xx.xx.xx/24 round-ro= bin sticky-address I think you may be missing the "round-robin" that spreads the mapping over your pool. The manual says that when more than 1 address is specified, round-robin is the only pool type allowed, it does not say that when more than 1 address is specified this is the default pool option. You can check your state table to see if it is indeed round-robin. #pfctl -s sta |grep " (" ... all tcp a.b.c.d:53802 (10.0.0.220:42808) -> 41.246.55.66:24 ESTABLIS= HED:ESTABLISHED all tcp a.b.c.e:60794 (10.0.0.38:47825) -> 216.58.223.10:443 ESTABLI= SHED:FIN_WAIT_2 If all your addresses "a.b.c.X" are the same, it's not round-robin and that's your problem. Ian -- = Ian Freislich From owner-freebsd-pf@FreeBSD.ORG Tue Jun 23 08:12:29 2015 Return-Path: Delivered-To: freebsd-pf@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 688052A2 for ; Tue, 23 Jun 2015 08:12:29 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F25C99DA for ; Tue, 23 Jun 2015 08:12:28 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Tue, 23 Jun 2015 10:12:26 +0200 id 00EB08DE.558914EA.0001135D Date: Tue, 23 Jun 2015 10:12:25 +0200 From: Milan Obuch To: Ian FREISLICH Cc: freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150623101225.4bc7f2d0@zeta.dino.sk> In-Reply-To: References: <20150623073856.334ebd61@zeta.dino.sk> <20150621133236.75a4d86d@zeta.dino.sk> <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 08:12:29 -0000 On Tue, 23 Jun 2015 09:49:57 +0200 Ian FREISLICH wrote: > Milan Obuch wrote: > > As a first step, I did small upgrade, so now I run FreeBSD > > 9.3-STABLE #0 r284695: Mon Jun 22 08:55:29 CEST 2015. > > > > I still see the issue, but I found simpler workaround when bad state > > ocurs - using > > > > pfctl -k > > pfctl -K > > > > in this order seems to remedy the issue for this one affected client > > without affecting other clients. This still does not solve the > > problem, just eases the reaction. > > How is your NAT rule defined? I had a closer look at the way I did > it: > > nat on vlan46 from 10.8.0.0/15 to ! -> xx.xx.xx.xx/24 > round-robin sticky-address > > I think you may be missing the "round-robin" that spreads the mapping > over your pool. The manual says that when more than 1 address is > specified, round-robin is the only pool type allowed, it does not > say that when more than 1 address is specified this is the default > pool option. > Thanks for hint, however, this is not the case I think. My definition is nat on $if_ext from to any -> $pool_ext round-robin sticky-address where contains contains some /24 segments from 10.0.0.0/8 range and one /24 and one /15 segment from 172.16.0.0/12 range, $pool_ext is one /23 public segment. > You can check your state table to see if it is indeed round-robin. > > #pfctl -s sta |grep " (" > ... > all tcp a.b.c.d:53802 (10.0.0.220:42808) -> 41.246.55.66:24 > ESTABLISHED:ESTABLISHED all tcp a.b.c.e:60794 (10.0.0.38:47825) -> > 216.58.223.10:443 ESTABLISHED:FIN_WAIT_2 > > If all your addresses "a.b.c.X" are the same, it's not round-robin > and that's your problem. > Well, this is something I do not fully understand. If my pool were a.b.c.0/24, then what you wrote could not be any other way - I think this is not what you meant. Or did you think there will be only one IP used? That's definitelly not the case, I see many IPs from my /23 segment here. One strange thing occured, however - it looks like if one IP from this /23 range gets used, trouble occurs. I do pfctl -k and pfctl -K for this address and all is well again. As long as this one IP is not used, everything works. When it gets used again, voila, trouble again. As this does not occur that fast, I need to check every now and then, and I am checking the other way too, but it is really annoying if it hits any customer. Regards, Milan From owner-freebsd-pf@FreeBSD.ORG Tue Jun 23 08:57:17 2015 Return-Path: Delivered-To: freebsd-pf@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7BEA5576 for ; Tue, 23 Jun 2015 08:57:17 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: from mail-yk0-x22b.google.com (mail-yk0-x22b.google.com [IPv6:2607:f8b0:4002:c07::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 34533E82 for ; Tue, 23 Jun 2015 08:57:17 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: by ykdt186 with SMTP id t186so1737256ykd.0 for ; Tue, 23 Jun 2015 01:57:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=9v7/iFOuwiTGgAFOhcC/a9TyC6f2nKCTxapbQB7vdZM=; b=LY6Cjd9F91bQWa9uhDNStVOnNW1EUzsb3i88Ui+aZae4Y7Xvr8xviKdoRvHYIK6vGI jIodLAMcs6/SvucafupNb5eYTSwFltUPyS7saPtM9H8BGoSotzYhOKIBGqXN2hvvUG9r bLt82llaDwHIpWfx8RfQtyEUxMVuD0fYnO9fFpSxm/0T8fBlYKFP3yKp6GzM3usrhNLj AFTQ7AV5co0mPu9BWX2CCN5jU0EQ0p4w6Rd1LRNRmPAXGocfPW9J4bqwJU+lZso+AzYa Gx0ciPpsb+4ir9KWuPX+dYg4jVPdXysZnZCnf/lplcvGQEeIZATk8qzabi8KmwyhPlsr 3BTw== MIME-Version: 1.0 X-Received: by 10.170.198.142 with SMTP id p136mr29347516yke.70.1435049836193; Tue, 23 Jun 2015 01:57:16 -0700 (PDT) Sender: ermal.luci@gmail.com Received: by 10.129.123.137 with HTTP; Tue, 23 Jun 2015 01:57:16 -0700 (PDT) In-Reply-To: <20150623101225.4bc7f2d0@zeta.dino.sk> References: <20150623073856.334ebd61@zeta.dino.sk> <20150621133236.75a4d86d@zeta.dino.sk> <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623101225.4bc7f2d0@zeta.dino.sk> Date: Tue, 23 Jun 2015 10:57:16 +0200 X-Google-Sender-Auth: 5U445p71fE-A4IdGdcaYxLjfMyg Message-ID: Subject: Re: Large scale NAT with PF - some weird problem From: =?UTF-8?Q?Ermal_Lu=C3=A7i?= To: Milan Obuch Cc: Ian FREISLICH , "freebsd-pf@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 08:57:17 -0000 On Tue, Jun 23, 2015 at 10:12 AM, Milan Obuch wrote: > On Tue, 23 Jun 2015 09:49:57 +0200 > Ian FREISLICH wrote: > > > Milan Obuch wrote: > > > As a first step, I did small upgrade, so now I run FreeBSD > > > 9.3-STABLE #0 r284695: Mon Jun 22 08:55:29 CEST 2015. > > > > > > I still see the issue, but I found simpler workaround when bad state > > > ocurs - using > > > > > > pfctl -k > > > pfctl -K > > > > > > in this order seems to remedy the issue for this one affected client > > > without affecting other clients. This still does not solve the > > > problem, just eases the reaction. > > > > How is your NAT rule defined? I had a closer look at the way I did > > it: > > > > nat on vlan46 from 10.8.0.0/15 to ! -> xx.xx.xx.xx/24 > > round-robin sticky-address > > > > I think you may be missing the "round-robin" that spreads the mapping > > over your pool. The manual says that when more than 1 address is > > specified, round-robin is the only pool type allowed, it does not > > say that when more than 1 address is specified this is the default > > pool option. > > > > Thanks for hint, however, this is not the case I think. My definition is > > nat on $if_ext from to any -> $pool_ext round-robin > sticky-address > > where contains contains some /24 segments from 10.0.0.0/8 > range and one /24 and one /15 segment from 172.16.0.0/12 range, > $pool_ext is one /23 public segment. > > > You can check your state table to see if it is indeed round-robin. > > > > #pfctl -s sta |grep " (" > > ... > > all tcp a.b.c.d:53802 (10.0.0.220:42808) -> 41.246.55.66:24 > > ESTABLISHED:ESTABLISHED all tcp a.b.c.e:60794 (10.0.0.38:47825) -> > > 216.58.223.10:443 ESTABLISHED:FIN_WAIT_2 > > > > If all your addresses "a.b.c.X" are the same, it's not round-robin > > and that's your problem. > > > > Well, this is something I do not fully understand. If my pool were > a.b.c.0/24, then what you wrote could not be any other way - I think > this is not what you meant. Or did you think there will be only one IP > used? That's definitelly not the case, I see many IPs from my /23 > segment here. > > One strange thing occured, however - it looks like if one IP from > this /23 range gets used, trouble occurs. I do pfctl -k and pfctl -K > for this address and all is well again. As long as this one IP is not > used, everything works. When it gets used again, voila, trouble again. > > Can you check if you are reaching the limits on source entries set limit src-nodes 2000 sets the maximum number of entries in the memory pool used for tracking source IP addresses (generated by the sticky-address and src.track options) to 2000. > As this does not occur that fast, I need to check every now and then, > and I am checking the other way too, but it is really annoying if it > hits any customer. > > Regards, > Milan > _______________________________________________ > freebsd-pf@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-pf > To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" > > -- > Ermal > From owner-freebsd-pf@FreeBSD.ORG Tue Jun 23 09:05:47 2015 Return-Path: Delivered-To: freebsd-pf@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9550B68F for ; Tue, 23 Jun 2015 09:05:47 +0000 (UTC) (envelope-from thomas@gibfest.dk) Received: from mail.tyknet.dk (mail.tyknet.dk [144.76.253.226]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 525DF2EE for ; Tue, 23 Jun 2015 09:05:46 +0000 (UTC) (envelope-from thomas@gibfest.dk) Received: from [10.10.13.2] (217.71.4.82.static.router4.bolignet.dk [217.71.4.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.tyknet.dk (Postfix) with ESMTPSA id 23D712BDF8 for ; Tue, 23 Jun 2015 09:05:38 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.9.2 mail.tyknet.dk 23D712BDF8 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=gibfest.dk; s=default; t=1435050338; bh=HDslMJAFsmTplWYG7OI5BDh71B/lIfY2dpxe5TGI6hU=; h=To:From:Subject:Date; b=gRuiylDnICUBR7Lzi3fGqFTkEFg588L7KcRepqIkiVgg0eZ0Mdz8ZH1487I7SlyJf NQVmnYytqnzNVDayWEMF0Fj8kJXuxvMpQZYHmwp/BHSkhFrTtGB9akbV9YvaG5bqi5 K8q1EDFoSvdPEnlE/Jyu/CkOsTYjhcoASDsCDa99gkLsqMOH29qWSwNW8xF0wa27jA SuSWDmoY+jAy9H6njqzKMJj/8FIqRHz840nWyBQ5frA59SoDBrP49Ur8Hn8kOmy1Jj NcsZ2khPSjDtAKUXIt9gV4zDXl9vPIYhZgyYUbYWl0Or/IF93gsRxvLiE3WRT/L58l gI7AMWRc/2ORA== To: freebsd-pf@freebsd.org From: Thomas Steen Rasmussen Subject: problem with pf ($interface) expansion on freebsd 10.1 with > 64 ip adresses on interface X-Enigmail-Draft-Status: N1010 Message-ID: <55892161.7000205@gibfest.dk> Date: Tue, 23 Jun 2015 11:05:37 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.0.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 09:05:47 -0000 Hello list, I have this rule in my pf.conf: pass in quick on $if proto tcp from { } to ($if) port 22 The rule permits SSH to all adresses on $if of course. The problem is that the enumeration of IPs on the interface that happens at boottime fails when the number of IP adresses exceed 64 IPs. If I reboot with 65 IPs on the interface the rule matches nothing and I get the following error in dmesg: pfi_table_update: cannot set 65 new addresses into table igb1: 22 This is on FreeBSD 10.1-STABLE FreeBSD 10.1-STABLE #0 r284163 If I add or remove an IP to the interface manually after the boot finishes the enumeration works fine, and all IPs on the interface are permitted SSH. The problem occurs only at boottime - when (I assume) pf tries to add all the IPs at once. I was going to open a PR for this but I wanted to hear if the list has any input first? Thanks! /Thomas Steen Rasmussen From owner-freebsd-pf@FreeBSD.ORG Tue Jun 23 09:15:25 2015 Return-Path: Delivered-To: freebsd-pf@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 82B9D74E for ; Tue, 23 Jun 2015 09:15:25 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F0ADEA2E; Tue, 23 Jun 2015 09:15:24 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Tue, 23 Jun 2015 11:15:21 +0200 id 00EB08E8.558923A9.00011946 Date: Tue, 23 Jun 2015 11:15:20 +0200 From: Milan Obuch To: Ermal =?ISO-8859-1?Q?Lu=E7i?= Cc: Ian FREISLICH , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150623111520.1679794b@zeta.dino.sk> In-Reply-To: References: <20150623073856.334ebd61@zeta.dino.sk> <20150621133236.75a4d86d@zeta.dino.sk> <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623101225.4bc7f2d0@zeta.dino.sk> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 09:15:25 -0000 On Tue, 23 Jun 2015 10:57:16 +0200 Ermal Lu=C3=A7i wrote: > On Tue, Jun 23, 2015 at 10:12 AM, Milan Obuch > wrote: >=20 > > On Tue, 23 Jun 2015 09:49:57 +0200 > > Ian FREISLICH wrote: [ snip ] > > > How is your NAT rule defined? I had a closer look at the way I > > > did it: > > > > > > nat on vlan46 from 10.8.0.0/15 to ! -> xx.xx.xx.xx/24 > > > round-robin sticky-address > > > > > > I think you may be missing the "round-robin" that spreads the > > > mapping over your pool. The manual says that when more than 1 > > > address is specified, round-robin is the only pool type allowed, > > > it does not say that when more than 1 address is specified this > > > is the default pool option. > > > > > > > Thanks for hint, however, this is not the case I think. My > > definition is > > > > nat on $if_ext from to any -> $pool_ext round-robin > > sticky-address > > > > where contains contains some /24 segments from 10.0.0.0/8 > > range and one /24 and one /15 segment from 172.16.0.0/12 range, > > $pool_ext is one /23 public segment. > > > > > You can check your state table to see if it is indeed round-robin. > > > > > > #pfctl -s sta |grep " (" > > > ... > > > all tcp a.b.c.d:53802 (10.0.0.220:42808) -> 41.246.55.66:24 > > > ESTABLISHED:ESTABLISHED all tcp a.b.c.e:60794 (10.0.0.38:47825) -> > > > 216.58.223.10:443 ESTABLISHED:FIN_WAIT_2 > > > > > > If all your addresses "a.b.c.X" are the same, it's not round-robin > > > and that's your problem. > > > > > > > Well, this is something I do not fully understand. If my pool were > > a.b.c.0/24, then what you wrote could not be any other way - I think > > this is not what you meant. Or did you think there will be only one > > IP used? That's definitelly not the case, I see many IPs from my /23 > > segment here. > > > > One strange thing occured, however - it looks like if one IP from > > this /23 range gets used, trouble occurs. I do pfctl -k and pfctl -K > > for this address and all is well again. As long as this one IP is > > not used, everything works. When it gets used again, voila, trouble > > again. > > > > > Can you check if you are reaching the limits on source entries > set limit src-nodes 2000 >=20 > sets the maximum number of entries in the memory pool used > for tracking source IP addresses (generated by the sticky-address and > src.track options) to 2000. > Well, I think it is big enough - pfctl -s memory: states hard limit 500000 src-nodes hard limit 100000 frags hard limit 50000 tables hard limit 5000 table-entries hard limit 500000 Excerpt from pfctl -vs info: Source Tracking Table current entries 418 =20 searches 1435901 36.2/s inserts 4577 0.1/s removals 4159 0.1/s My gut feeling is there is just much more space than necessary, but this should not hurt, I think. Thanks, Milan From owner-freebsd-pf@FreeBSD.ORG Tue Jun 23 09:23:35 2015 Return-Path: Delivered-To: freebsd-pf@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 668CE84A for ; Tue, 23 Jun 2015 09:23:35 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EEEB1E4E for ; Tue, 23 Jun 2015 09:23:33 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Tue, 23 Jun 2015 11:23:31 +0200 id 00EB08D9.55892593.00011A2A Date: Tue, 23 Jun 2015 11:23:31 +0200 From: Milan Obuch To: Ian FREISLICH Cc: freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150623112331.668395d1@zeta.dino.sk> In-Reply-To: References: <20150623101225.4bc7f2d0@zeta.dino.sk> <20150623073856.334ebd61@zeta.dino.sk> <20150621133236.75a4d86d@zeta.dino.sk> <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 09:23:35 -0000 On Tue, 23 Jun 2015 10:57:44 +0200 Ian FREISLICH wrote: > Milan Obuch wrote: > > > all tcp a.b.c.d:53802 (10.0.0.220:42808) -> 41.246.55.66:24 > > > ESTABLISHED:ESTABLISHED all tcp a.b.c.e:60794 (10.0.0.38:47825) -> > > > 216.58.223.10:443 ESTABLISHED:FIN_WAIT_2 > > > > > > If all your addresses "a.b.c.X" are the same, it's not round-robin > > > and that's your problem. > > > > > > > Well, this is something I do not fully understand. If my pool were > > a.b.c.0/24, then what you wrote could not be any other way - I think > > this is not what you meant. Or did you think there will be only one > > IP used? That's definitelly not the case, I see many IPs from my /23 > > segment here. > > I just wanted to check that more than 1 address was being used. > OK, it is. If only one IP were used for all traffic, I would run into issues much earlier. > So, I think that the problem is with 9-STABLE. I hate "upgrade to > solve your problems" answers because they may not. I do know that > 10 has seen a lot of work and none of that work will make it back > into 9 because of the PF rewrite. Maybe someone else in this group > will chime in. > That's OK. I am a bit conservative on upgrades here because with hundreds - thousands users you need a bit of stability too, but upgrade to 10-STABLE is currently being prepared. That being written, it will not occur today. > I ran 10-CURRENT in production for as long 10 was CURRENT and then > went to 10-STABLE precisely because I was having state issues > forwarding performance issues with 9. Gleb Smirnof did a significant > rewrite of PF to improve SMP performance. He had access to my > system for debugging on a large installation. > Well, we'll see. I'll let you know how it goes when upgrade will be done. > If you're not already doing so, I'd recomend running CARP + pfsync > so you can test updates while maintaining a known working backup. > If you're running pfsync, I recommend you run it on a different > interface to the one with your traffic and with a cross-over cable > between your machines. The pfsync packet rate caused a small amount > packet loss on other network traffic. > I did not experiment much with CARP and pfsync. I plan to use it, but that means more hardware... and I am not the one who pays for it. Anyway, I would try to redesign the whole thing so it will be easier to maintain and, if necessary, troubleshooting. Regards, Milan From owner-freebsd-pf@freebsd.org Thu Jun 25 18:30:16 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 970A698C731 for ; Thu, 25 Jun 2015 18:30:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 840E011AD for ; Thu, 25 Jun 2015 18:30:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t5PIUGH0049583 for ; Thu, 25 Jun 2015 18:30:16 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-pf@FreeBSD.org Subject: [Bug 193620] Problem with igb multiqueue together with pf Date: Thu, 25 Jun 2015 18:30:16 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-RELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: sbruno@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jun 2015 18:30:16 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193620 Sean Bruno changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sbruno@FreeBSD.org Keywords| |IntelNetworking -- You are receiving this mail because: You are on the CC list for the bug.