Date: Fri, 19 Sep 2014 12:33:09 +0400 From: "Alexander V. Chernikov" <melifaro@FreeBSD.org> To: Freddie Cash <fjwcash@gmail.com>, "freebsd-ipfw@freebsd.org" <freebsd-ipfw@freebsd.org> Subject: Re: High intr CPU % and slow throughput Message-ID: <541BEA45.6060001@FreeBSD.org> In-Reply-To: <CAOjFWZ79RN_Pr%2Bk6w6E0fW0QzWiYrSuaHVJnN3ft5Bzg4sXK6A@mail.gmail.com> References: <CAOjFWZ7DjjTUmk%2Ba9VdLuetwuTrZdQ9OkrrS3FX3c%2BWs18E-pQ@mail.gmail.com> <CAOjFWZ79RN_Pr%2Bk6w6E0fW0QzWiYrSuaHVJnN3ft5Bzg4sXK6A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 18.09.2014 23:30, Freddie Cash wrote: > Aha! I believe I've found the cause of our current issue. > > In an effort to allow reloading of the firewall rules during the day > without disconnecting anyone (dropping TCP connections), I started playing > with rule sets. And everything appeared to be working wonderfully, in that > I could restart the rules multiple times without dropping any packets or > disconnecting anyone. > > But, CPU usage skyrocketed on large downloads and > > we were capped at a little less than 40 Mbps. :( > > It seems that if you do the following (at least twice, to make sure rules > are in both sets), your CPU will melt: > - clear set 1 > - disable set 1 > - load 4000 rules into set 1 > - enable set 1 > - swap sets 1 and 0 > - disable set 1 > > I thought that would leave only the rules in set 0 active, which would be > the equivalent of only having loaded rules into set 0. However, it seems > that ipfw still checks rules in disabled sets! Or does some kind of > processing with disabled sets. Yes. _All_ rules in all sets are referenced inside single array. ipfw does not process disabled rules itself, but 1) it has to load given rule to cpu cache 2) check if disabled set mask matches > > pmcstat was showing lots (200-2000) of unresolved samples and ipfw.ko > sitting at 80-90% in the list, even when CPU usage was around 30%. > > I did the above, but added "ipfw -f set 1 flush" as the last step, and > everything is back to normal. pmstat is now empty (0 unresolved). > > We can now push 75 Mbps through the firewall with CPU usage under 80%. > More importantly, though, other traffic is not impacted by large downloads > and speedtests and streaming video! And, CPU usage is sitting at under 10% > for "normal" traffic. > > Yes, I know 4000 rules is a lot (doing NAT for 66 systems and 2 local > subnets). Until now I was focusing on getting things working (migrating > from FreeBSD 7 using IPFW+natd with lots of private IP to private IP rules; > to FreeBSD 10 using IPFW + in-kernel NAT and proper double-NAT across > networks using public IPs only). Optimisation work is just now beginning. > :) >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?541BEA45.6060001>