Date: Thu, 27 Jan 2011 10:57:14 +0100 From: Damien Fleuriot <ml@my.gd> To: "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org> Cc: freebsd-pf@freebsd.org Subject: High interrupt rate on a PF box + performance Message-ID: <4D41417A.20904@my.gd>
next in thread | raw e-mail | index | archive | help
Hello list,
I have a problem with interrupts, network cards, and PF performance.
We have 2 firewalls running FreeBSD 8.0 for the current master and
FreeBSD 8.1 for the backup host, which I upgraded just yesterday.
The servers use CARP for redundancy.
These are rather busy boxes which run PF and nginx as a reverse proxy.
As you will see below, we're getting a "high" %interrupt CPU usage,
which seems to come mostly from the NICs.
I'm wondering if there is any way to optimize the box's performance and
reduce the interrupts rate or the CPU usage ?
Also, we've noticed a sharp drop in CPU usage since we've disabled
pfsync, but we'd rather keep it now wouldn't we ?
Last, we seem to get input errors on the NICs, although the switch ports
report not a single layer 2 error in over a year.
I'm wondering what counts as a NIC input error ?
Hardware is as follows:
CPU
--
CPU: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz (2496.25-MHz
K8-class CPU)
Origin = "GenuineIntel" Id = 0x10676 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
ACPI APIC Table: <DELL PE_SC3 >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
MEM
--
real memory = 2147483648 (2048 MB)
avail memory = 2057293824 (1961 MB)
NICs
--
bce0: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem
0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci7
bce1: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem
0xf8000000-0xf9ffffff irq 16 at device 0.0 on pci3
igb0: <Intel(R) PRO/1000 Network Connection version - 1.7.3> port
0xdce0-0xdcff mem
0xfd0e0000-0xfd0fffff,0xfce00000-0xfcffffff,0xfd0dc000-0xfd0dffff irq 18
at device 0.0 on pci14
igb0: Using MSIX interrupts with 3 vectors
Find below different outputs from the current master running FreeBSD
8.0-RELEASE-p2
systat -v
---
3 users Load 0.41 0.31 0.29 Jan 26 18:59
Mem:KB REAL VIRTUAL VN PAGER SWAP
PAGER
Tot Share Tot Share Free in out in
out
Act 143036 8152 836392 11188 1262556 count
All 168224 10420 1074653k 31172 pages
Proc: Interrupts
r p d s w Csw Trp Sys Int Sof Flt cow 36163 total
47 105k 76 2077 28k 223 zfod
ata0 irq14
ozfod
mfi0 irq16
4.3%Sys 28.1%Intr 3.0%User 0.0%Nice 64.7%Idle %ozfod
uhci0 uhci
| | | | | | | | | | | daefr 1998
cpu0: time
==++++++++++++++>> prcfr 9428
bce0 256
33 dtbuf totfr 12931
igb0 257
Namei Name-cache Dir-cache 100000 desvn react 5791
igb0 258
Calls hits % hits % 70448 numvn pdwak
igb0 259
24988 frevn pdpgs
igb1 260
intrn 1
igb1 261
Disks mfid0 372392 wire
igb1 262
KB/t 0.00 62336 act 20
bce1 269
tps 0 323720 inact 1998
cpu1: time
MB/s 0.00 292 cache 1998
cpu2: time
%busy 0 1262264 free 1998
cpu3: time
218272 buf
vmstat -i
---
interrupt total rate
irq14: ata0 36 0
irq16: mfi0 353244 1
irq21: uhci0 uhci+ 461504 1
cpu0: timer 615183815 1996
irq256: bce0 1015412475 3295
irq257: igb0 1067318584 3464
irq258: igb0 695648752 2258
irq259: igb0 2 0
irq260: igb1 11503857 37
irq261: igb1 506598 1
irq262: igb1 69 0
irq269: bce1 790820 2
cpu1: timer 615183757 1996
cpu2: timer 615197165 1996
cpu3: timer 615197165 1996
Total 5252757843 17050
pf status (159 filter rules, 17 nat/rdr rules)
---
# pfctl -si
Status: Enabled for 3 days 13:34:56 Debug: Urgent
Interface Stats for igb0 IPv4 IPv6
Bytes In 487209136643 384
Bytes Out 687158173727 0
Packets In
Passed 1967249106 0
Blocked 6183860 6
Packets Out
Passed 2018192359 0
Blocked 686901 0
State Table Total Rate
current entries 25428
searches 9006187476 29231.8/s
inserts 679746853 2206.3/s
removals 679721425 2206.2/s
Counters
match 686988143 2229.8/s
bad-offset 0 0.0/s
fragment 56 0.0/s
short 0 0.0/s
normalize 171 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 1 0.0/s
proto-cksum 13916 0.0/s
state-mismatch 220169 0.7/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 1812 0.0/s
synproxy 0 0.0/s
Regards,
--
dfl
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D41417A.20904>