From owner-freebsd-pf@FreeBSD.ORG Thu Jun 9 05:47:54 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5A775106566C for ; Thu, 9 Jun 2011 05:47:54 +0000 (UTC) (envelope-from mail@miketm.com) Received: from mail-yi0-f54.google.com (mail-yi0-f54.google.com [209.85.218.54]) by mx1.freebsd.org (Postfix) with ESMTP id 1266F8FC0A for ; Thu, 9 Jun 2011 05:47:53 +0000 (UTC) Received: by yie13 with SMTP id 13so886960yie.13 for ; Wed, 08 Jun 2011 22:47:53 -0700 (PDT) MIME-Version: 1.0 Received: by 10.101.131.29 with SMTP id i29mr255062ann.47.1307596717160; Wed, 08 Jun 2011 22:18:37 -0700 (PDT) Received: by 10.100.125.12 with HTTP; Wed, 8 Jun 2011 22:18:37 -0700 (PDT) X-Originating-IP: [123.243.191.201] In-Reply-To: <4DEFB0D1.3080607@my.gd> References: <4DEF7844.7070208@my.gd> <4DEFB0D1.3080607@my.gd> Date: Thu, 9 Jun 2011 15:18:37 +1000 Message-ID: From: Mike M To: Damien Fleuriot Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-pf@freebsd.org Subject: Re: rule not responding to incoming packets X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jun 2011 05:47:54 -0000 Damien, Further responses below. On Thu, Jun 9, 2011 at 3:26 AM, Damien Fleuriot wrote: > Hey up Mike, > > > Sorry about the delay, was busy at work ;) > > On 6/8/11 3:55 PM, Mike M wrote: > > Hi Damien, > > > > Thanks for helping out, I've provided responses beneath yours below. > > > > On Wed, Jun 8, 2011 at 11:25 PM, Damien Fleuriot > > wrote: > > > > Hey up Mike, > > > > > > My responses in between your own text. > > > > On 6/8/11 9:58 AM, Mike M wrote: > > > Hi, > > > > [snip] > > > > > Another thing I've noticed is that the src limits seem to have an > > > effect (state table is typically 4k-7k entries), as without this in > > > place, the state table fills rapidly, rendering the box near > unusable. > > > Using 'synproxy state' also seems to have a similar effect. I > have > > > never observed any IP addresses within the table > (via > > > 'pfctl -T show -t attacksource') > > > > > > > Reduce your tcp timeout values so states are removed faster. > I understand that lowering these values helps remove the states quicker (and thus helps prevent the table from filling up so easily), but I don't believe this is the actual problem here (and the state table will still fill regardless when the packet rate is high). What I'm observing, is packets not being returned and thus connections in the handshake not being fully established. However, it's not happening for all connecting clients... I can connect to the web server fine because I match the table rule, as can others in the rule. The problem is only prevalent for users matching the rule containing the overload. However, if I remove the 'overload' bits with the max src settings, the state table fills instantly and renders the box unusable. However, the table never contains any entries, so how does this make sense? > > > > > > I think I've already done this but if you have specific sets or values > > that you can suggest, I'll definitely take them on board. > > > > These are the values we use on a production, high traffic website (3k > hits/s) > > $ sudo pfctl -st > tcp.first 5s > tcp.opening 5s > tcp.established 30s > tcp.closing 5s > tcp.finwait 10s > tcp.closed 5s > tcp.tsdiff 10s > udp.first 10s > udp.single 10s > udp.multiple 30s > icmp.first 10s > icmp.error 5s > other.first 10s > other.single 10s > other.multiple 10s > frag 30s > interval 5s > adaptive.start 156000 states > adaptive.end 312000 states > src.track 0s > > > > > # -- adaptive timeouts > > > set timeout { tcp.first 20, adaptive.start 30000, adaptive.end > > 1800000 } > > > > If you're under DDoS, adjust your timers so that TCP syn packets time > > out much faster. > > > > You can also set it only for your port 80 rule. > > > > > > Any specific instruction for this? Open to suggestion! > > > > This is an example of a timeout set only for a very specific rule: > pass quick on $vpnif keep state ( tcp.established 36000 ) > > > > > # -- disallow basic spoof > > > antispoof quick for { lo } > > > > I do not see what you're hoping to achieve here. > > Also, you've set skip on lo, so you're adding rules that won't ever > be > > applied. > > > > > > This one was added on suggestion I had read in forums, if it is > > unnecessary or not useful, I'll happily remove it? > > > > Please do, this antispoof won't help you. > > It would if you applied it on your PHYSICAL interfaces, but on loopback > ? no sense at all. > Roger that, will do. > > > > > block quick from > > > > If these are spoofed IPs generated randomly, you'll saturate your > table > > and you'll make your firewall work a lot for not much... > > > > > > The idea behind this was from the 'overload' statement in my public > > 80/TCP rule below -- if I can identify those attacking sources, I should > > be able to drop those packets as quickly and efficiently as possible, > > no? What I've observed however, is that *zero* source IP's have been > > added to the table so I'm not sure it's doing its job. > > This is part of my reason to mail this list, I really don't know if this > > is a bug, or the behaviour I'm seeing is part of my configuration. If > > the latter, I'm getting undesired results so am interested in getting > > suggestions to mitigate this DDoS and remain serviceable to my public > > clients. > > > > Obviously they're not hitting the limit you set in your pass rule for > port 80 then. > But how is it, that when I remove this bit from the rule, the state table fills instantly? It seems to have some sort of effect, but not the one I'd expect. This, is confusing. > > > > > > > > # -- HTTP public > > > pass in proto tcp from any to $h_pub port 80 flags S/SA keep state > > > (max-src-conn 100, max-src-conn-rate 20/5, overload > > > flush global) > > > > flags S/SA is optional as it is the default. > > Why did you set $if_pub and $if_priv if you're not using them ? > > > > > > Yeah, understand the bit about the flags but this is handled when the > > conf is processed no? So I assume the inclusion adds no extra > > processing load. > > > > Indeed, makes your ruleset file easier to read to the human eye though. > > > > As far as the $if_priv var goes, yeah this is not utilised but again, > > wouldn't imagine the lack of use would create a noticeable load? It's > > merely there for future use. > > > > You may safely use them like so: > > pass in on $if_pub inet proto tcp from any to $if_pub port 80 keep state > (source track...) > > Notice that when you say "to $if_pub", PF automagically takes the IPs > assigned to your public interface. > > In case of multiple IPs, PF creates one rule per IP. > > > > > 1/ I would suggest enabling logging on your default drop rule and > run: > > tcpdump -nei pflog0 ip and port 80 > > > > > > Adding 'log' to my rules and monitoring pflog0 creates a substantial > > load on the box (and fills up the hard drive quickly in /var/log... so > > I'm not all that keen to add 'log' statements in my rules to do this. > > I've been looking at em0 with tcpdump to do my diagnostics. Any reason > > why this wouldn't suffice? > > > > Indeed, the idea is to not leave this running for very long... > > Find below the output produced by tcpdump'ing pflog0 with the -e flag > set (-e adds rule numbers): > > # tcpdump -nei pflog0 ip > tcpdump: WARNING: pflog0: no IPv4 address assigned > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size > 96 bytes > 19:21:41.070165 rule 47/0(match): pass in on bce0: 109.0.65.3.12628 > > 88.190.222.254.53: [|domain] > 19:21:42.819873 rule 24/0(match): block in on bce0: 88.190.13.231.5353 > > 224.0.0.251.5353: [|domain] > 19:21:43.821475 rule 24/0(match): block in on bce0: 88.190.13.231.5353 > > 224.0.0.251.5353: [|domain] > ^C > 3 packets captured > 9 packets received by filter > 0 packets dropped by kernel > > > Notice how the log tells me what rule the packet matched (and ultimately > passed or dropped). > I've actually already done this and I can see the rule in question, is the one being matched. But there is no other block rule or anything else that helps me understand why packets don't go back out, and thus ending up with a fully established connection. If it's the kernel causing it, why can sources in the and table establish connections without problem? Is what I'm seeing, some sort of bug? > > Later on, I can check my rules like so: pfctl -vvsr > > This outputs rules with their number, for example rule 47 is: > @47 pass in log on bce0 inet proto udp from any to 88.190.222.254 port = > domain keep state (if-bound) > [ Evaluations: 4130668 Packets: 107887 Bytes: 10484013 States: > 0 ] > [ Inserted: uid 0 pid 20941 ] > > > > > You'll see what rule matches when dropping your packets to port 80. > > > > Chances are it will be your default drop rule, if so, this means your > > port 80 rule is not allowed to create any more state entries. > > > > > > I really don't understand why the box isn't responding to packets that > > are matched by the public rule for 80/TCP when those matched by > > or tables are. This tells me that the DDoS load is > > not the cause of the problem, but rather the pf rule itself. Can anyone > > suggest why what I'm seeing, is actually happening? > > > > See above my response about "pfctl -vvsr", it will give you counters for > each of your rules. > > > I can't see any drop rules occurring when 'log' is enabled, it's just > > that the incoming packets don't have anything go back 'out'. Is this > > kernel related? > > > > Then that would mean your rule works and connections are sent to your > backend web server (same machine ? diff machine ?). > The webserver is the same machine. Routing back works, if I stick the source IP's that are failing with this rule, in the or tables, everything works fine. This leads me to believe it's not a kernel setting (which is not dependant on IP), but rather a problem with this particular rule. But how to fix it? > > You will want to ensure your backend web server can actually reach your > clients' public IPs. > > > > > 2/ Double check that your remote client test IP isn't in either > > or > > > > > > > > Again, there are *zero* entries in or -- what > > strikes me as weird, is that even though the overload entry exists in > > that rule, no IP's are actually inside (via 'pfctl -T > > show -t attacksource') -- but when I remove the overload statement and > > reload pf, the state table fills up rapidly and renders the box useless. > > This tells me that this overload condition is having an effect, but I > > don't understand why the table is empty. > > > > See above about "pfctl -vvsr" > Unfortunately this doesn't help me. I can see the inbound rule is matched, but there is no explanation for why the packets don't get sent back... the handshake never completes so the connection isn't established... herein lies my problem! > > > /still pulling hair out... please help! :> > > > > - Mike > Any ideas? :> Cheers, - Mike