From owner-freebsd-pf@FreeBSD.ORG Sat May 17 16:18:56 2008 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 91A241065675 for ; Sat, 17 May 2008 16:18:56 +0000 (UTC) (envelope-from johan@stromnet.se) Received: from core.stromnet.se (core.stromnet.se [83.218.84.131]) by mx1.freebsd.org (Postfix) with ESMTP id 43A718FC20 for ; Sat, 17 May 2008 16:18:56 +0000 (UTC) (envelope-from johan@stromnet.se) Received: from localhost (core.stromnet.se [83.218.84.131]) by core.stromnet.se (Postfix) with ESMTP id BBDCEF59403; Sat, 17 May 2008 18:01:16 +0200 (CEST) X-Virus-Scanned: amavisd-new at stromnet.se X-Spam-Flag: NO X-Spam-Score: 0.294 X-Spam-Level: X-Spam-Status: No, score=0.294 tagged_above=0 required=6.2 tests=[AWL=2.100, BAYES_00=-2.599, RDNS_DYNAMIC=0.1, SPF_FAIL=0.693] Received: from core.stromnet.se ([83.218.84.131]) by localhost (core.stromnet.se [83.218.84.131]) (amavisd-new, port 10024) with ESMTP id Y9G3KTO53EEo; Sat, 17 May 2008 18:01:14 +0200 (CEST) Received: from johan-mp.stromnet.se (90-224-172-102-no129.tbcn.telia.com [90.224.172.102]) by core.stromnet.se (Postfix) with ESMTP id BAEE3F58D78; Sat, 17 May 2008 18:01:12 +0200 (CEST) Message-Id: <679DB462-75D6-45CC-949C-1BE8E12C22CD@stromnet.se> From: =?ISO-8859-1?Q?Johan_Str=F6m?= To: Alex Trull In-Reply-To: <1211037564.6326.27.camel@porksoda> Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Apple Message framework v919.2) Date: Sat, 17 May 2008 18:01:10 +0200 References: <678A03F5-5E8A-4CF6-90DF-AA9A4F30FBE1@stromnet.se> <1211037564.6326.27.camel@porksoda> X-Mailer: Apple Mail (2.919.2) Cc: freebsd-net@freebsd.org, freebsd-stable , freebsd-pf@freebsd.org Subject: Re: connect(): Operation not permitted X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 16:18:56 -0000 First of all, for freebsd-pf subscribers, I posted my original problem =20= (in the bottom) to freebsd-net earlier, but replies seems to point to =20= PF so I'll CC there too.. On May 17, 2008, at 5:19 PM, Alex Trull wrote: > Hi Johan and List, > > In my case a few months ago it was pahu. Don't give that fine fellow =20= > an > account on your precious system ! > > > But seriously, I had a pf-firewalled jail being being used for DNS > testing, with large numbers of udp "connections" hanging around in pf > state. While the default udp timeout settings in PF are lower than =20 > those > of the tcp timeouts, it is was still too high for it to to remove the > states in time before hitting the default 10k state limit! > > If this is the case with you - run 'pfctl -s state | wc -l' - when =20 > there > is traffic load you may see that hitting 10k states if you've not =20 > tuned > that variable. > > What to do next - up the state limit or lower the state timeouts. I =20= > did > both, to be safe. > > in /etc/pf.conf these must be at the very top of the file: > > # options > # 10k is insanely low, lets raise it.. > set limit { frags 16384, states 32768 } > # timeouts - see 'pfctl -s timeouts' for options - you will want to > # change the tcp ones rather than the udp ones for your smtp setup. > # but these are mine, I set them for the dns traffic. > set timeout { udp.first 15, udp.single 5, udp.multiple 30 } > > > don't forget to: > > $ /etc/rc.d/pf check && /etc/rc.d/pf reload Ok, looked over the PF states now, but I'm not quite sure thats what =20 causing it. I have default limit on 10k states, normally I seem to =20 have around ~800 states, and when I start my test script that tries to =20= send as many mails as possible (using PHP's Pear::Mail, creating a =20 connection, sending, disconnecting, creating new connection.. and so =20 on), I can clearly see the PF state counter (pfctl -vsi) increase, but =20= the script aborts with Operation not permitted way before I hit 10k, =20 its rather around 3-4k.. If I then wait a few seconds and run the script again, I can see the =20 number of states increase even more, and if I do this enough times I =20 finally hit around 9700 states. But at this point (states exhausted), =20= I don't get Operation not permitted, instead it just seems that the =20 script blocks up a few seconds while states clear up, then continues =20 running until it gets a Operation not permitted. So, from the above results, I cant say that it looks like its the =20 states? Just tried to disable the altq rule now too, no changes (not that I =20 expected one, since its on bce0 not lo0). Another thing, which might be more approriate in freebsd-pf though.. =20 Why would it create states at all for this traffic, when my pf.conf =20 rule is "pass on lo0 inet from $jail to $jail" (i have a block drop in =20= rule to drop all traffic)? A check with pfctl -vsr reveals that the =20 actual rule inserted is "pass on lo0 inet from 123.123.123.123 to =20 123.123.123.123 flags S/SA keep state". Where did that "keep state" =20 come from? Thanks for ideas :) > > > HTH, > > Alex > > On Sat, 2008-05-17 at 16:33 +0200, Johan Str=F6m wrote: >> Hello >> >> I got a FreeBSD 7 machine running mail services (among other things). >> This machine recently replaced a FreeBSD 6.2 machine doing the same >> tasks. >> Now and then I need to send alot of mail to customers (mailing list), >> and one thing i've noticed now after the change is that when I use a >> lot of connections subsequently (high connection rate, even if they >> are very shortlived) inside a jail (dunno if that has anything to do >> with it though), I start to get Operation not permitted in return to >> connect(). >> I've seen this in the PHP app that sends mail, when it tried to >> connect to localhost, as well as from postfix when it have been =20 >> trying >> to connect to amavisd on localhost, but also from postfix when it has >> tried to connect to remote SMTP servers. >> >> I do have PF for filtering, but there are no max-src-conn-rate limits >> enabled for any rules that is used for this. However, from one of the >> jail I do have a hfsc queue limiting the outgoing mail traffic from >> one jailed IP. But I'm not sure that this would be the problem, since >> I've also seen the problem when doing localhost connects in the jail, >> and also in other jails on an entierly different IP that is not >> affected. >> >> Does anyone have any clues about what I can look at and tune to fix >> this? >> >> Thanks! >> >> -- >> Johan Str=F6m >> Stromnet >> johan@stromnet.se >> http://www.stromnet.se/ >> >> >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org=20 >> "