Date: Fri, 4 Jul 2008 14:30:48 -0700 From: "Kian Mohageri" <kian.mohageri@gmail.com> To: "Jeremy Chadwick" <koitsu@freebsd.org> Cc: freebsd-stable <freebsd-stable@freebsd.org>, stef@memberwebs.com, freebsd-net@freebsd.org, freebsd-pf@freebsd.org, Alex Trull <alex@trull.org> Subject: Re: connect(): Operation not permitted Message-ID: <fee88ee40807041430r6640df26u45b546b106b5732c@mail.gmail.com> In-Reply-To: <20080704113213.GA13586@eos.sc1.parodius.com> References: <678A03F5-5E8A-4CF6-90DF-AA9A4F30FBE1@stromnet.se> <1211037564.6326.27.camel@porksoda> <679DB462-75D6-45CC-949C-1BE8E12C22CD@stromnet.se> <482FD877.6050707@infracaninophile.co.uk> <B44C565F-65A5-498A-9B79-3FFE15E33A7A@stromnet.se> <fee88ee40805181029x31755a75i3b61731864e995fd@mail.gmail.com> <20080703003955.859BCF180C0@mx.npubs.com> <fee88ee40807030855l6d7e6a78hf2138e4783e4beb4@mail.gmail.com> <20080704113213.GA13586@eos.sc1.parodius.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Jul 4, 2008 at 4:32 AM, Jeremy Chadwick <koitsu@freebsd.org> wrote: > On Thu, Jul 03, 2008 at 08:55:21AM -0700, Kian Mohageri wrote: >> On Wed, Jul 2, 2008 at 5:39 PM, Stef <stef-list@memberwebs.com> wrote: >> > Kian Mohageri wrote: >> >> On Sun, May 18, 2008 at 3:33 AM, Johan Ström <johan@stromnet.se> wrote: >> >>> On May 18, 2008, at 9:19 AM, Matthew Seaman wrote: >> >>> >> >>>> Johan Ström wrote: >> >>>> >> >>>>> drop all traffic)? A check with pfctl -vsr reveals that the actual rule >> >>>>> inserted is "pass on lo0 inet from 123.123.123.123 to 123.123.123.123 flags >> >>>>> S/SA keep state". Where did that "keep state" come from? >> >>>> 'flags S/SA keep state' is the default now for tcp filter rules -- that >> >>>> was new in 7.0 reflecting the upstream changes made between the 4.0 and >> >>>> 4.1 >> >>>> releases of OpenBSD. If you want a stateless rule, append 'no state'. >> >>>> >> >>>> http://www.openbsd.org/faq/pf/filter.html#state >> >>> Thanks! I was actually looking around in the pf.conf manpage but failed to >> >>> find it yesterday, but looking closer today I now saw it. >> >>> Applied the no state (and quick) to the rule, and now no state is created. >> >>> And the problem I had in the first place seems to have been resolved too >> >>> now, even though it didn't look like a state problem.. (started to deny new >> >>> connections much earlier than the states was full, altough maybee i wasnt >> >>> looking for updates fast enough or something). >> >>> >> >> >> >> I'd be willing to bet it's because you're reusing the source port on a >> >> new connection before the old state expires. >> >> >> >> You'll know if you check the state-mismatch counter. >> >> >> >> Anyway, glad you found a resolution. >> > >> > I've been experiencing this "Operation not permitted" too. I've been >> > trying to track down the problem for many months, but due to the >> > complexity of my firewalls (scores of jails each with scores of rules), >> > I wasn't brave enough to ask for help :) >> > >> > As a work around we started creating rules without state, whenever we >> > would run into the problem. >> > >> > Thanks for the pointer about state-mismatch. The state-mismatch counter >> > does is in fact high in my case (see below). How would I go about >> > getting the pf state timeout and the reuse of ports for outbound >> > connections to match? Or is this an intractable problem, that just needs >> > to be worked around? >> >> Make sure your state-mismatch counter is increasing at the same times >> you experience the problem (and isn't just high from some unrelated >> issue). >> >> A similar/related problem was addressed in OpenBSD 4.3 >> (http://www.openbsd.org/plus43.html). >> >> * In pf(4), allow state reuse if both sides are in FIN_WAIT_2 and a >> new SYN arrives. >> >> I'm not sure if it's been imported yet. If not, you could try tuning >> your timeout values (see pf.conf(5)). >> >> The specific issue I was experienced was solved by shortening >> tcp.closed, IIRC. It's been a while though. > > When administrators see state-mismatch increasing, they get concerned. > The common scapegoat is tcp.closed, which people don't even bother to > describe (pf has an internal value of 10 seconds applied to that value, > e.g. tcp.closed=5 means 15 seconds). > > You can set tcp.closed as low as you want, but chances are random > Internet users will have equipment with IP stacks that re-use outbound > sockets which haven't fully closed down within the aforementioned > interval. pf cannot fix this. > > For example, on our production/hosting systems, we see state-mismatch > increase fairly often. I just pfctl -F info'd our main webserver, and > within about 15 minutes, state-mismatch was up to 22. We use tcp.closed > of 5 (which means 15 seconds). > > Workarounds such as "no state" suffice, but if you use rdr rules, you > MUST track state, which means there's no way of winning in that case. > For sake of example, OpenBSD spamd requires the use of rdr rules. > > Administrators then ask 3 questions: > For the sake of a helpful archive... > 1) How do I determine whether or not state-mismatch increasing is a > sign of bad things, or due to peoples' broken IP stacks, You can't. Only way you know is probably when people complain, or you notice scripts/page loads failing. > 2) What happens to packets which cause state-mismatch to increment, > e.g. are they blocked, passed, or what? Dropped. In the case of a state-mismatch during TCP handshake, an RST is sent. That's why the failure happens immediately. > 3) Why isn't state-mismatch described in detail in the documentation? > Good question. I guess because it would be difficult to document all of the reasons a state wouldn't match. It would be easier to simply document what a state _is_, but that's already in the archives. -Kian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?fee88ee40807041430r6640df26u45b546b106b5732c>