From owner-freebsd-pf@FreeBSD.ORG  Tue Jun 23 05:39:14 2015
Return-Path: <owner-freebsd-pf@FreeBSD.ORG>
Delivered-To: freebsd-pf@nevdull.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id C69097E7
 for <freebsd-pf@nevdull.freebsd.org>; Tue, 23 Jun 2015 05:39:14 +0000 (UTC)
 (envelope-from freebsd-pf@dino.sk)
Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72])
 (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 5CD9D340
 for <freebsd-pf@freebsd.org>; Tue, 23 Jun 2015 05:39:13 +0000 (UTC)
 (envelope-from freebsd-pf@dino.sk)
Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan)
 by mailhost.netlabit.sk with ESMTPA; Tue, 23 Jun 2015 07:39:03 +0200
 id 00EB081E.5588F0F7.0000EACB
Date: Tue, 23 Jun 2015 07:38:56 +0200
From: Milan Obuch <freebsd-pf@dino.sk>
To: Ian FREISLICH <ian.freislich@capeaugusta.com>
Cc: freebsd-pf@freebsd.org
Subject: Re: Large scale NAT with PF - some weird problem
Message-ID: <20150623073856.334ebd61@zeta.dino.sk>
In-Reply-To: <20150621195753.7b162633@zeta.dino.sk>
References: <20150621133236.75a4d86d@zeta.dino.sk>
 <20150620182432.62797ec5@zeta.dino.sk>
 <20150619091857.304b707b@zeta.dino.sk>
 <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com>
 <E1Z6dHz-0000uu-D8@clue.co.za> <E1Z6eVg-0000yz-Ar@clue.co.za>
 <20150621195753.7b162633@zeta.dino.sk>
X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-pf@freebsd.org
X-Mailman-Version: 2.1.20
Precedence: list
List-Id: "Technical discussion and general questions about packet filter
 \(pf\)" <freebsd-pf.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-pf>,
 <mailto:freebsd-pf-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-pf/>
List-Post: <mailto:freebsd-pf@freebsd.org>
List-Help: <mailto:freebsd-pf-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-pf>,
 <mailto:freebsd-pf-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 23 Jun 2015 05:39:14 -0000

On Sun, 21 Jun 2015 19:57:53 +0200
Milan Obuch <freebsd-pf@dino.sk> wrote:

> On Sun, 21 Jun 2015 08:38:04 -0400
> Ian FREISLICH <ian.freislich@capeaugusta.com> wrote:
> 

[ snip ]

> > I also had some other settings regarding interrupt moderation on
> > the NIC, netisr threads, queue depth and dispatch.  I disabled
> > entropy harvesting on interrupts, and the network path.  Some of
> > these settings are loader.conf settings, some are runtime sysctls.
> > 
> > I still think that if it's possible, you should give 10-STABLE a
> > try.
> > 
> 
> This will take some time to do. Unfortunatelly, I did not think about
> possibilities to test various version when the system was installed.
> My bad. Now it is not easy, but I am trying to find usable way to do
> it.
> 
> Regards,
> Milan
>

As a first step, I did small upgrade, so now I run FreeBSD 9.3-STABLE
#0 r284695: Mon Jun 22 08:55:29 CEST 2015.

I still see the issue, but I found simpler workaround when bad state
ocurs - using

pfctl -k <ip.of.affected.client>
pfctl -K <ip.of.affected.client>

in this order seems to remedy the issue for this one affected client
without affecting other clients. This still does not solve the problem,
just eases the reaction.

Also, not sure yet, but it seems when it occurs, if more clients are
natted using the same public IP, all are affected the same way. Using
mentioned workaround for all of them makes them all work again.

Regards,
Milan