Date: Wed, 08 Jun 2011 19:26:41 +0200 From: Damien Fleuriot <ml@my.gd> To: Mike M <mail@miketm.com> Cc: freebsd-pf@freebsd.org Subject: Re: rule not responding to incoming packets Message-ID: <4DEFB0D1.3080607@my.gd> In-Reply-To: <BANLkTikO5%2BNDhK5CRj3--5KCKdwBSnjosg@mail.gmail.com> References: <BANLkTi=ca09018UP0FJwou6dbXh4EmM=Eg@mail.gmail.com> <4DEF7844.7070208@my.gd> <BANLkTikO5%2BNDhK5CRj3--5KCKdwBSnjosg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hey up Mike,
Sorry about the delay, was busy at work ;)
On 6/8/11 3:55 PM, Mike M wrote:
> Hi Damien,
>
> Thanks for helping out, I've provided responses beneath yours below.
>
> On Wed, Jun 8, 2011 at 11:25 PM, Damien Fleuriot <ml@my.gd
> <mailto:ml@my.gd>> wrote:
>
> Hey up Mike,
>
>
> My responses in between your own text.
>
> On 6/8/11 9:58 AM, Mike M wrote:
> > Hi,
> >
[snip]
>
> > Another thing I've noticed is that the src limits seem to have an
> > effect (state table is typically 4k-7k entries), as without this in
> > place, the state table fills rapidly, rendering the box near unusable.
> > Using 'synproxy state' also seems to have a similar effect. I have
> > never observed any IP addresses within the <attacksource> table (via
> > 'pfctl -T show -t attacksource')
> >
>
> Reduce your tcp timeout values so states are removed faster.
>
>
> I think I've already done this but if you have specific sets or values
> that you can suggest, I'll definitely take them on board.
>
These are the values we use on a production, high traffic website (3k
hits/s)
$ sudo pfctl -st
tcp.first 5s
tcp.opening 5s
tcp.established 30s
tcp.closing 5s
tcp.finwait 10s
tcp.closed 5s
tcp.tsdiff 10s
udp.first 10s
udp.single 10s
udp.multiple 30s
icmp.first 10s
icmp.error 5s
other.first 10s
other.single 10s
other.multiple 10s
frag 30s
interval 5s
adaptive.start 156000 states
adaptive.end 312000 states
src.track 0s
> > # -- adaptive timeouts
> > set timeout { tcp.first 20, adaptive.start 30000, adaptive.end
> 1800000 }
>
> If you're under DDoS, adjust your timers so that TCP syn packets time
> out much faster.
>
> You can also set it only for your port 80 rule.
>
>
> Any specific instruction for this? Open to suggestion!
>
This is an example of a timeout set only for a very specific rule:
pass quick on $vpnif keep state ( tcp.established 36000 )
> > # -- disallow basic spoof
> > antispoof quick for { lo }
>
> I do not see what you're hoping to achieve here.
> Also, you've set skip on lo, so you're adding rules that won't ever be
> applied.
>
>
> This one was added on suggestion I had read in forums, if it is
> unnecessary or not useful, I'll happily remove it?
>
Please do, this antispoof won't help you.
It would if you applied it on your PHYSICAL interfaces, but on loopback
? no sense at all.
> > block quick from <attacksource>
>
> If these are spoofed IPs generated randomly, you'll saturate your table
> and you'll make your firewall work a lot for not much...
>
>
> The idea behind this was from the 'overload' statement in my public
> 80/TCP rule below -- if I can identify those attacking sources, I should
> be able to drop those packets as quickly and efficiently as possible,
> no? What I've observed however, is that *zero* source IP's have been
> added to the <attacksource> table so I'm not sure it's doing its job.
> This is part of my reason to mail this list, I really don't know if this
> is a bug, or the behaviour I'm seeing is part of my configuration. If
> the latter, I'm getting undesired results so am interested in getting
> suggestions to mitigate this DDoS and remain serviceable to my public
> clients.
>
Obviously they're not hitting the limit you set in your pass rule for
port 80 then.
> >
> > # -- HTTP public
> > pass in proto tcp from any to $h_pub port 80 flags S/SA keep state
> > (max-src-conn 100, max-src-conn-rate 20/5, overload <attacksource>
> > flush global)
>
> flags S/SA is optional as it is the default.
> Why did you set $if_pub and $if_priv if you're not using them ?
>
>
> Yeah, understand the bit about the flags but this is handled when the
> conf is processed no? So I assume the inclusion adds no extra
> processing load.
>
Indeed, makes your ruleset file easier to read to the human eye though.
> As far as the $if_priv var goes, yeah this is not utilised but again,
> wouldn't imagine the lack of use would create a noticeable load? It's
> merely there for future use.
>
You may safely use them like so:
pass in on $if_pub inet proto tcp from any to $if_pub port 80 keep state
(source track...)
Notice that when you say "to $if_pub", PF automagically takes the IPs
assigned to your public interface.
In case of multiple IPs, PF creates one rule per IP.
>
> 1/ I would suggest enabling logging on your default drop rule and run:
> tcpdump -nei pflog0 ip and port 80
>
>
> Adding 'log' to my rules and monitoring pflog0 creates a substantial
> load on the box (and fills up the hard drive quickly in /var/log... so
> I'm not all that keen to add 'log' statements in my rules to do this.
> I've been looking at em0 with tcpdump to do my diagnostics. Any reason
> why this wouldn't suffice?
>
Indeed, the idea is to not leave this running for very long...
Find below the output produced by tcpdump'ing pflog0 with the -e flag
set (-e adds rule numbers):
# tcpdump -nei pflog0 ip
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size
96 bytes
19:21:41.070165 rule 47/0(match): pass in on bce0: 109.0.65.3.12628 >
88.190.222.254.53: [|domain]
19:21:42.819873 rule 24/0(match): block in on bce0: 88.190.13.231.5353 >
224.0.0.251.5353: [|domain]
19:21:43.821475 rule 24/0(match): block in on bce0: 88.190.13.231.5353 >
224.0.0.251.5353: [|domain]
^C
3 packets captured
9 packets received by filter
0 packets dropped by kernel
Notice how the log tells me what rule the packet matched (and ultimately
passed or dropped).
Later on, I can check my rules like so: pfctl -vvsr
This outputs rules with their number, for example rule 47 is:
@47 pass in log on bce0 inet proto udp from any to 88.190.222.254 port =
domain keep state (if-bound)
[ Evaluations: 4130668 Packets: 107887 Bytes: 10484013 States:
0 ]
[ Inserted: uid 0 pid 20941 ]
>
> You'll see what rule matches when dropping your packets to port 80.
>
> Chances are it will be your default drop rule, if so, this means your
> port 80 rule is not allowed to create any more state entries.
>
>
> I really don't understand why the box isn't responding to packets that
> are matched by the public rule for 80/TCP when those matched by
> <whitelist> or <staff> tables are. This tells me that the DDoS load is
> not the cause of the problem, but rather the pf rule itself. Can anyone
> suggest why what I'm seeing, is actually happening?
>
See above my response about "pfctl -vvsr", it will give you counters for
each of your rules.
> I can't see any drop rules occurring when 'log' is enabled, it's just
> that the incoming packets don't have anything go back 'out'. Is this
> kernel related?
>
Then that would mean your rule works and connections are sent to your
backend web server (same machine ? diff machine ?).
You will want to ensure your backend web server can actually reach your
clients' public IPs.
>
> 2/ Double check that your remote client test IP isn't in either
> <blacklist> or <attacksource>
>
>
>
> Again, there are *zero* entries in <blacklist> or <attacksource> -- what
> strikes me as weird, is that even though the overload entry exists in
> that rule, no IP's are actually inside <attacksource> (via 'pfctl -T
> show -t attacksource') -- but when I remove the overload statement and
> reload pf, the state table fills up rapidly and renders the box useless.
> This tells me that this overload condition is having an effect, but I
> don't understand why the table is empty.
>
See above about "pfctl -vvsr"
> /still pulling hair out... please help! :>
>
> - Mike
>
>
>
> _______________________________________________
> freebsd-pf@freebsd.org <mailto:freebsd-pf@freebsd.org> mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-pf
> To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org
> <mailto:freebsd-pf-unsubscribe@freebsd.org>"
>
>
>
>
> --
> Cheers,
>
> - Mike M
> mail@miketm.com <mailto:mail@miketm.com>
> +61 409 721 722
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4DEFB0D1.3080607>