Date: Sat, 17 May 2008 18:01:10 +0200 From: =?ISO-8859-1?Q?Johan_Str=F6m?= <johan@stromnet.se> To: Alex Trull <alex@trull.org> Cc: freebsd-net@freebsd.org, freebsd-stable <freebsd-stable@freebsd.org>, freebsd-pf@freebsd.org Subject: Re: connect(): Operation not permitted Message-ID: <679DB462-75D6-45CC-949C-1BE8E12C22CD@stromnet.se> In-Reply-To: <1211037564.6326.27.camel@porksoda> References: <678A03F5-5E8A-4CF6-90DF-AA9A4F30FBE1@stromnet.se> <1211037564.6326.27.camel@porksoda>
next in thread | previous in thread | raw e-mail | index | archive | help
First of all, for freebsd-pf subscribers, I posted my original problem =20=
(in the bottom) to freebsd-net earlier, but replies seems to point to =20=
PF so I'll CC there too..
On May 17, 2008, at 5:19 PM, Alex Trull wrote:
> Hi Johan and List,
>
> In my case a few months ago it was pahu. Don't give that fine fellow =20=
> an
> account on your precious system !
>
>
> But seriously, I had a pf-firewalled jail being being used for DNS
> testing, with large numbers of udp "connections" hanging around in pf
> state. While the default udp timeout settings in PF are lower than =20
> those
> of the tcp timeouts, it is was still too high for it to to remove the
> states in time before hitting the default 10k state limit!
>
> If this is the case with you - run 'pfctl -s state | wc -l' - when =20
> there
> is traffic load you may see that hitting 10k states if you've not =20
> tuned
> that variable.
>
> What to do next - up the state limit or lower the state timeouts. I =20=
> did
> both, to be safe.
>
> in /etc/pf.conf these must be at the very top of the file:
>
> # options
> # 10k is insanely low, lets raise it..
> set limit { frags 16384, states 32768 }
> # timeouts - see 'pfctl -s timeouts' for options - you will want to
> # change the tcp ones rather than the udp ones for your smtp setup.
> # but these are mine, I set them for the dns traffic.
> set timeout { udp.first 15, udp.single 5, udp.multiple 30 }
>
>
> don't forget to:
>
> $ /etc/rc.d/pf check && /etc/rc.d/pf reload
Ok, looked over the PF states now, but I'm not quite sure thats what =20
causing it. I have default limit on 10k states, normally I seem to =20
have around ~800 states, and when I start my test script that tries to =20=
send as many mails as possible (using PHP's Pear::Mail, creating a =20
connection, sending, disconnecting, creating new connection.. and so =20
on), I can clearly see the PF state counter (pfctl -vsi) increase, but =20=
the script aborts with Operation not permitted way before I hit 10k, =20
its rather around 3-4k..
If I then wait a few seconds and run the script again, I can see the =20
number of states increase even more, and if I do this enough times I =20
finally hit around 9700 states. But at this point (states exhausted), =20=
I don't get Operation not permitted, instead it just seems that the =20
script blocks up a few seconds while states clear up, then continues =20
running until it gets a Operation not permitted.
So, from the above results, I cant say that it looks like its the =20
states?
Just tried to disable the altq rule now too, no changes (not that I =20
expected one, since its on bce0 not lo0).
Another thing, which might be more approriate in freebsd-pf though.. =20
Why would it create states at all for this traffic, when my pf.conf =20
rule is "pass on lo0 inet from $jail to $jail" (i have a block drop in =20=
rule to drop all traffic)? A check with pfctl -vsr reveals that the =20
actual rule inserted is "pass on lo0 inet from 123.123.123.123 to =20
123.123.123.123 flags S/SA keep state". Where did that "keep state" =20
come from?
Thanks for ideas :)
>
>
> HTH,
>
> Alex
>
> On Sat, 2008-05-17 at 16:33 +0200, Johan Str=F6m wrote:
>> Hello
>>
>> I got a FreeBSD 7 machine running mail services (among other things).
>> This machine recently replaced a FreeBSD 6.2 machine doing the same
>> tasks.
>> Now and then I need to send alot of mail to customers (mailing list),
>> and one thing i've noticed now after the change is that when I use a
>> lot of connections subsequently (high connection rate, even if they
>> are very shortlived) inside a jail (dunno if that has anything to do
>> with it though), I start to get Operation not permitted in return to
>> connect().
>> I've seen this in the PHP app that sends mail, when it tried to
>> connect to localhost, as well as from postfix when it have been =20
>> trying
>> to connect to amavisd on localhost, but also from postfix when it has
>> tried to connect to remote SMTP servers.
>>
>> I do have PF for filtering, but there are no max-src-conn-rate limits
>> enabled for any rules that is used for this. However, from one of the
>> jail I do have a hfsc queue limiting the outgoing mail traffic from
>> one jailed IP. But I'm not sure that this would be the problem, since
>> I've also seen the problem when doing localhost connects in the jail,
>> and also in other jails on an entierly different IP that is not
>> affected.
>>
>> Does anyone have any clues about what I can look at and tune to fix
>> this?
>>
>> Thanks!
>>
>> --
>> Johan Str=F6m
>> Stromnet
>> johan@stromnet.se
>> http://www.stromnet.se/
>>
>>
>> _______________________________________________
>> freebsd-stable@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to =
"freebsd-stable-unsubscribe@freebsd.org=20
>> "
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?679DB462-75D6-45CC-949C-1BE8E12C22CD>