Date: Thu, 18 Sep 2014 09:26:09 -0700 From: Freddie Cash <fjwcash@gmail.com> To: "freebsd-ipfw@freebsd.org" <freebsd-ipfw@freebsd.org> Subject: High intr CPU % and slow throughput Message-ID: <CAOjFWZ7DjjTUmk%2Ba9VdLuetwuTrZdQ9OkrrS3FX3c%2BWs18E-pQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
[Not sure if this is more appropriate for the -ipfw or -stable mailing
lists.]
64-bit FreeBSD 10.0-p7
Dual-core AMD Opteron 1218 CPU @ 2.6 GHz
=E2=80=8B2 GB of DDR2 RAM
Intel i350-T4 quad-port gigabit NIC using igb(4)
Each of the gigabit NIC ports are connected to gigabit links (we have a
gigabit fibre link to our ISP, which has dual 10 Gbps links to the public
Internet).
Using the following simple ruleset (there are more rules, but these are the
ones that match when we test transfers to/from the Internet):
ipfw nat 8668 config ip 142.24.
=E2=80=8Bx.y=E2=80=8B
same_ports
10 allow ip from any to any via lo0
12 allow carp from any to any
20 reject log logamount 10000 ip from 10.0.0.0/8 to any in recv igb0
22 reject log logamount 10000 ip from 127.0.0.0/8 to any in recv igb0
=E2=80=8B2=E2=80=8B
4 reject log logamount 10000 ip from 172.16.0.0/20 to any in recv igb0
26 reject log logamount 10000 ip from 192.168.0.0/16 to any in recv igb0
50 skipto 65000 ip from 192.168.0.0/24 to not 142.24.
=E2=80=8Bx.z
/25 in recv igb2
52 skipto 65000 ip from not 142.24.13.128/25 to 142.24.
=E2=80=8Bx.y
in recv igb0
65000 allow ip from 192.168.0.0/24 to any in recv igb2
65002 nat 8668 ip from 192.168.0.0/24 to any out xmit igb0
65004 allow ip from 142.24.
=E2=80=8Bx.y=E2=80=8B
to any out xmit igb0
65006 nat 8668 ip from any to 142.24.
=E2=80=8Bx.y=E2=80=8B
in recv igb0
65008 allow ip from any to 192.168.0.0/24 in recv igb0
65010 allow ip from any to 192.168.0.0/24 out xmit igb2
When we start a large download or file transfer from the Internet (a single
file from a single server), CPU usage for the [intr{irq256: igb0:que}]
kernel thread jumps to over 90% (one CPU core) and causes all traffic
through the firewall (even traffic that doesn't go through igb0) to grind
to a standstill. Some TCP connections through other interfaces are even
dropped.=E2=80=8B During this time, the other CPU core is under 50% usage.
IIUIC, the [intr{irq256: igb0:que}] isn't showing actual CPU usage for
processing hardware interrupts, but is showing the CPU usage used to
process the packets going through IPFW. Correct? "vmstat -i" shows only
10-15 interrupts per second for each of the igb interfaces.
The really depressing part is that throughput (as shown by "iftop -i igb0"
and snmp graphing) never goes above 40 Mbps. :(
What can I do to try and track down exactly why this is occurring?
Is there anything I can do to reduce or mitigate this CPU usage?
Or, is this simply a case of the CPU being too old?
/boot/loader.conf currently has the following (been playing with most of
these lately, without much change in CPU usage):
## Tune the igb(4) interfaces a little
hw.igb.enable_aim=3D"1"
hw.igb.enable_msix=3D"1"
hw.igb.header_split=3D"0"
hw.igb.max_interrupt_rate=3D"16000"
hw.igb.num_queues=3D"0"
hw.igb.rx_process_limit=3D"1000"
hw.igb.rxd=3D"4096"
hw.igb.txd=3D"4096"
## Configure kernel
kern.hz=3D"4000"
## Configure IPFW
net.inet.ip.fw.default_to_accept=3D"1"
net.inet.ip.fw.verbose=3D"1"
## Configure network threads
net.isr.bindthreads=3D"1"
net.isr.direct=3D"1"
net.isr.maxthreads=3D"2"
=E2=80=8B/etc/sysctl.conf has the following (haven't changed these in a lon=
g time):
=E2=80=8B# IPFW options
net.inet.ip.fw.autoinc_step=3D2
net.inet.ip.fw.enable=3D1
net.inet.ip.fw.one_pass=3D1
net.inet.ip.fw.verbose=3D1
net.inet.ip.fw.verbose_limit=3D10000
At lunch today, we'll be failing-over to the other firewall, which will be
running without any /boot/loader.conf or /etc/sysctl.conf entries to see if
my "optimisations" are actually "pessimisations".
--=20
Freddie Cash
fjwcash@gmail.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOjFWZ7DjjTUmk%2Ba9VdLuetwuTrZdQ9OkrrS3FX3c%2BWs18E-pQ>