Date: Thu, 18 Sep 2014 20:41:38 +0400 From: "Alexander V. Chernikov" <melifaro@FreeBSD.org> To: Freddie Cash <fjwcash@gmail.com>, "freebsd-ipfw@freebsd.org" <freebsd-ipfw@freebsd.org> Subject: Re: High intr CPU % and slow throughput Message-ID: <541B0B42.6050403@FreeBSD.org> In-Reply-To: <CAOjFWZ7DjjTUmk%2Ba9VdLuetwuTrZdQ9OkrrS3FX3c%2BWs18E-pQ@mail.gmail.com> References: <CAOjFWZ7DjjTUmk%2Ba9VdLuetwuTrZdQ9OkrrS3FX3c%2BWs18E-pQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 18.09.2014 20:26, Freddie Cash wrote:
> [Not sure if this is more appropriate for the -ipfw or -stable mailing
> lists.]
>
>
> 64-bit FreeBSD 10.0-p7
>
> Dual-core AMD Opteron 1218 CPU @ 2.6 GHz
> 2 GB of DDR2 RAM
> Intel i350-T4 quad-port gigabit NIC using igb(4)
>
> Each of the gigabit NIC ports are connected to gigabit links (we have a
> gigabit fibre link to our ISP, which has dual 10 Gbps links to the public
> Internet).
>
> Using the following simple ruleset (there are more rules, but these are the
> ones that match when we test transfers to/from the Internet):
Please show all the ruleset with counters.
>
> ipfw nat 8668 config ip 142.24.
> x.y
> same_ports
>
> 10 allow ip from any to any via lo0
> 12 allow carp from any to any
>
> 20 reject log logamount 10000 ip from 10.0.0.0/8 to any in recv igb0
> 22 reject log logamount 10000 ip from 127.0.0.0/8 to any in recv igb0
> 2
> 4 reject log logamount 10000 ip from 172.16.0.0/20 to any in recv igb0
> 26 reject log logamount 10000 ip from 192.168.0.0/16 to any in recv igb0
>
> 50 skipto 65000 ip from 192.168.0.0/24 to not 142.24.
> x.z
> /25 in recv igb2
> 52 skipto 65000 ip from not 142.24.13.128/25 to 142.24.
> x.y
> in recv igb0
>
> 65000 allow ip from 192.168.0.0/24 to any in recv igb2
> 65002 nat 8668 ip from 192.168.0.0/24 to any out xmit igb0
> 65004 allow ip from 142.24.
> x.y
> to any out xmit igb0
>
> 65006 nat 8668 ip from any to 142.24.
> x.y
> in recv igb0
> 65008 allow ip from any to 192.168.0.0/24 in recv igb0
> 65010 allow ip from any to 192.168.0.0/24 out xmit igb2
>
> When we start a large download or file transfer from the Internet (a single
> file from a single server), CPU usage for the [intr{irq256: igb0:que}]
> kernel thread jumps to over 90% (one CPU core) and causes all traffic
> through the firewall (even traffic that doesn't go through igb0) to grind
> to a standstill. Some TCP connections through other interfaces are even
> dropped. During this time, the other CPU core is under 50% usage.
can you do the following:
kldload hwpmc
sudo pmcstat -TS instructions -w 1
and show its output when the problem is observed?
>
> IIUIC, the [intr{irq256: igb0:que}] isn't showing actual CPU usage for
> processing hardware interrupts, but is showing the CPU usage used to
> process the packets going through IPFW. Correct? "vmstat -i" shows only
> 10-15 interrupts per second for each of the igb interfaces.
>
> The really depressing part is that throughput (as shown by "iftop -i igb0"
> and snmp graphing) never goes above 40 Mbps. :(
>
> What can I do to try and track down exactly why this is occurring?
>
> Is there anything I can do to reduce or mitigate this CPU usage?
>
> Or, is this simply a case of the CPU being too old?
>
> /boot/loader.conf currently has the following (been playing with most of
> these lately, without much change in CPU usage):
>
> ## Tune the igb(4) interfaces a little
> hw.igb.enable_aim="1"
> hw.igb.enable_msix="1"
> hw.igb.header_split="0"
> hw.igb.max_interrupt_rate="16000"
> hw.igb.num_queues="0"
> hw.igb.rx_process_limit="1000"
> hw.igb.rxd="4096"
> hw.igb.txd="4096"
>
> ## Configure kernel
> kern.hz="4000"
>
> ## Configure IPFW
> net.inet.ip.fw.default_to_accept="1"
> net.inet.ip.fw.verbose="1"
>
> ## Configure network threads
> net.isr.bindthreads="1"
> net.isr.direct="1"
> net.isr.maxthreads="2"
>
>
> /etc/sysctl.conf has the following (haven't changed these in a long time):
>
> # IPFW options
> net.inet.ip.fw.autoinc_step=2
> net.inet.ip.fw.enable=1
> net.inet.ip.fw.one_pass=1
> net.inet.ip.fw.verbose=1
> net.inet.ip.fw.verbose_limit=10000
>
>
> At lunch today, we'll be failing-over to the other firewall, which will be
> running without any /boot/loader.conf or /etc/sysctl.conf entries to see if
> my "optimisations" are actually "pessimisations".
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?541B0B42.6050403>