Date: Wed, 8 Jun 2011 23:55:53 +1000 From: Mike M <mail@miketm.com> To: freebsd-pf@freebsd.org, Damien Fleuriot <ml@my.gd> Subject: Re: rule not responding to incoming packets Message-ID: <BANLkTikO5%2BNDhK5CRj3--5KCKdwBSnjosg@mail.gmail.com> In-Reply-To: <4DEF7844.7070208@my.gd> References: <BANLkTi=ca09018UP0FJwou6dbXh4EmM=Eg@mail.gmail.com> <4DEF7844.7070208@my.gd>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Damien,
Thanks for helping out, I've provided responses beneath yours below.
On Wed, Jun 8, 2011 at 11:25 PM, Damien Fleuriot <ml@my.gd> wrote:
> Hey up Mike,
>
>
> My responses in between your own text.
>
> On 6/8/11 9:58 AM, Mike M wrote:
> > Hi,
> >
> > I have an issue with pf where incoming packets matching a particular
> > rule, are not being responded to, resulting in public users being
> > unable to access a web server. I'm receiving a SYN flood on 80/TCP
> > (currently ~50mbit @ 100kpps) so am trying to implement some
> > protection on the box. I don't believe the current DDoS is actually
> > causing this issue though, as packets matching the <whitelist> and
> > <staff> tables, can establish connections without a problem.
> >
> > Other packets matching this other rule however, seem to be unable to
> > establish a connection. I see the SYN packets in via tcpdump, but
> > they are not returned. Eventually, the connection closes.
> >
>
> If they're spoofed IPs, you're blacklisting randomly generated IPs.
>
> Call your upstream provider, THEY do have tools to fight DDoS, you don't.
>
> Yeah, I'm aware of that but unfortunately not all providers can or will
assist. My approach is to try and deal with what reaches my box as best I
can.
> > Another thing I've noticed is that the src limits seem to have an
> > effect (state table is typically 4k-7k entries), as without this in
> > place, the state table fills rapidly, rendering the box near unusable.
> > Using 'synproxy state' also seems to have a similar effect. I have
> > never observed any IP addresses within the <attacksource> table (via
> > 'pfctl -T show -t attacksource')
> >
>
> Reduce your tcp timeout values so states are removed faster.
>
>
I think I've already done this but if you have specific sets or values that
you can suggest, I'll definitely take them on board.
> > System is FreeBSD 8.2-RELEASE, 1GB RAM, Intel P4 3GHz (2 x CPU w/SMP)
> >
> > Have provided some sanitized information below -- any assistance would
> > be much appreciated.... I'm pulling my hair out. Any other DDoS
> > hardening advice based on below is also very welcome :>
> >
> > Please advise if more information is required.
> >
> > Cheers,
> >
> > - Mike
> >
> >
> >
> > [root@sb ~]# more /etc/pf.conf.conf
> > # --- firewall
> > #
> >
> >
> > # ---- interfaces
> > if_pub = "em0"
> > if_priv = "em1"
> >
> > # -- loopback
> > if_loop = "lo0"
> >
> >
> >
> > # ---- hosts
> >
> > # -- public interface
> > h_pub = "10.0.1.1"
> >
> > # -- external hosts
> > # TBA
> >
> >
> >
> > # ---- tables
> > table <attacksource> persist
> > table <blacklist> persist file "/etc/pf/blacklist.pf"
> > table <whitelist> persist file "/etc/pf/whitelist.pf"
> > table <staff> persist file "/etc/pf/staff.pf"
> >
> >
> >
> > # ---- set policies
> >
> > # -- rule optimization
> > set optimization aggressive
> > #set optimization normal
> >
> > # -- adaptive timeouts
> > set timeout { tcp.first 20, adaptive.start 30000, adaptive.end 1800000 }
>
> If you're under DDoS, adjust your timers so that TCP syn packets time
> out much faster.
>
> You can also set it only for your port 80 rule.
>
>
Any specific instruction for this? Open to suggestion!
This box publicly, is *only* a webserver.
>
> > # -- set max states
> > set limit states 1800000
> >
> > # -- statistics logging
> > set loginterface $if_pub
> >
> > # -- don't filter on interface lo
> > set skip on lo
> >
> > # -- normalization
> > scrub in
> >
> >
> >
> > # ---- filter rules
> >
> > # -- block to start
> > block in
> >
> > # -- disallow basic spoof
> > antispoof quick for { lo }
>
> I do not see what you're hoping to achieve here.
> Also, you've set skip on lo, so you're adding rules that won't ever be
> applied.
>
This one was added on suggestion I had read in forums, if it is unnecessary
or not useful, I'll happily remove it?
>
> >
> > # -- whitelist
> > pass quick from <whitelist>
>
> Ok.
>
> >
> > # -- blacklists
> > block quick from <blacklist>
>
> Ok.
>
> > block quick from <attacksource>
>
> If these are spoofed IPs generated randomly, you'll saturate your table
> and you'll make your firewall work a lot for not much...
>
>
The idea behind this was from the 'overload' statement in my public 80/TCP
rule below -- if I can identify those attacking sources, I should be able to
drop those packets as quickly and efficiently as possible, no? What I've
observed however, is that *zero* source IP's have been added to the
<attacksource> table so I'm not sure it's doing its job. This is part of my
reason to mail this list, I really don't know if this is a bug, or the
behaviour I'm seeing is part of my configuration. If the latter, I'm
getting undesired results so am interested in getting suggestions to
mitigate this DDoS and remain serviceable to my public clients.
> >
> > # -- block juno flood traffic
> > block in quick proto tcp from any port { 1024, 3072 } to any
>
> Some people are actually stupid enough to use static source ports when
> flooding ? Ok.
>
>
Yep, this happens, so this rule is fairly common to drop these packets
quickly.
> >
> > # -- block UDP floods
> > block in quick proto udp from any to $h_pub
>
> I do hope you don't host DNS on this box.
>
>
Indeed I don't. The only public traffic I need to receive, is HTTP.
> >
> > # -- HTTP public
> > pass in proto tcp from any to $h_pub port 80 flags S/SA keep state
> > (max-src-conn 100, max-src-conn-rate 20/5, overload <attacksource>
> > flush global)
>
> flags S/SA is optional as it is the default.
> Why did you set $if_pub and $if_priv if you're not using them ?
>
>
Yeah, understand the bit about the flags but this is handled when the conf
is processed no? So I assume the inclusion adds no extra processing load.
As far as the $if_priv var goes, yeah this is not utilised but again,
wouldn't imagine the lack of use would create a noticeable load? It's
merely there for future use.
> >
> > # -- HTTP staff
> > pass in proto tcp from <staff> to any port 80
> >
> > # ---- allow all outbound
> > pass out keep state
>
> If you already allow all out, you should adjust your rules above (for
> the whitelist for example) so they only match INbound packets imo.
>
>
Point taken, will adjust, thanks.
>
>
> 1/ I would suggest enabling logging on your default drop rule and run:
> tcpdump -nei pflog0 ip and port 80
>
Adding 'log' to my rules and monitoring pflog0 creates a substantial load on
the box (and fills up the hard drive quickly in /var/log... so I'm not all
that keen to add 'log' statements in my rules to do this. I've been looking
at em0 with tcpdump to do my diagnostics. Any reason why this wouldn't
suffice?
> You'll see what rule matches when dropping your packets to port 80.
>
> Chances are it will be your default drop rule, if so, this means your
> port 80 rule is not allowed to create any more state entries.
>
I really don't understand why the box isn't responding to packets that are
matched by the public rule for 80/TCP when those matched by <whitelist> or
<staff> tables are. This tells me that the DDoS load is not the cause of
the problem, but rather the pf rule itself. Can anyone suggest why what I'm
seeing, is actually happening?
I can't see any drop rules occurring when 'log' is enabled, it's just that
the incoming packets don't have anything go back 'out'. Is this kernel
related?
>
>
> 2/ Double check that your remote client test IP isn't in either
> <blacklist> or <attacksource>
>
Again, there are *zero* entries in <blacklist> or <attacksource> -- what
strikes me as weird, is that even though the overload entry exists in that
rule, no IP's are actually inside <attacksource> (via 'pfctl -T show -t
attacksource') -- but when I remove the overload statement and reload pf,
the state table fills up rapidly and renders the box useless. This tells me
that this overload condition is having an effect, but I don't understand why
the table is empty.
/still pulling hair out... please help! :>
- Mike
>
> _______________________________________________
> freebsd-pf@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-pf
> To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org"
>
--
Cheers,
- Mike M
mail@miketm.com
+61 409 721 722
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BANLkTikO5%2BNDhK5CRj3--5KCKdwBSnjosg>