Date: Mon, 14 May 2012 10:49:28 -0600 From: "Kirk Davis" <kirk.davis@epsb.ca> To: <freebsd-net@freebsd.org> Subject: 10Gb/s ixgbe and mbuf performance tuning. Message-ID: <529374128DC1B04D9D037911B8E8F0531095BEF8@Exchange26.EDU.epsb.ca>
next in thread | raw e-mail | index | archive | help
Hi,
I am running FreeBSD 8.1p3 amd64 with 6Gb of RAM. This
is a router using Quagga for BGP. It has two 10Gb/s network connections
using the intel ixgbe 2.4.4 driver. It is handling routing for about
200 remote sites and about 300k simultaneous sessions through the box.
After a few hours at high traffic volumes (just over a 1Gb/s) it will
stop routing any traffic. It look like I am running out of mbuf's. I
have increased them but I am worried that there is a limit as to how
much I can increase them without starving the kernel of space. Console
is still responsive but no network.
=20
I have looked over Tuning(7) and some other great
articles on the net on performance tuning FreeBSD but it has left me a
little confused. Can someone help to explain the best way to
troubleshoot this issue and any formulas to tune it.
=20
Here is some information:
# netstat -m
66042/18063/84105 mbufs in use (current/cache/total)
66039/15931/81970/524288 mbuf clusters in use (current/cache/total/max)
66039/15881 mbuf+clusters out of packet secondary zone in use
(current/cache)
0/5/5/524288 4k (page size) jumbo clusters in use
(current/cache/total/max)
0/0/0/131072 9k jumbo clusters in use (current/cache/total/max)
0/0/0/65536 16k jumbo clusters in use (current/cache/total/max)
148588K/36397K/184986K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
=20
=20
# more /boot/loader.conf
geom_mirror_load=3D"YES"
loader_logo=3D"beastie"
vesa_load=3D"YES"
ixgbe_load=3D"YES"
hw.ixgbe.txd=3D4096
hw.ixgbe.rxd=3D4096
kern.ipc.nmbjumbop=3D524288
kern.ipc.nmbclusters=3D524288
kern.ipc.maxsockbuf=3D4194304
=20
# more /etc/sysctl.conf=20
#security.bsd.see_other_uids=3D0
kern.ipc.nmbjumbop=3D524288
kern.ipc.nmbclusters=3D524288
kern.ipc.maxsockbuf=3D4194304
hw.intr_storm_threshold=3D9000
net.inet.icmp.icmplim=3D1000
=20
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights
reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-RELEASE-p1 #3: Thu Mar 29 08:53:27 MDT 2012
root@SchoolGW2.epsb.ca:/usr/obj/usr/src/sys/DELLR710x64 amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU X5560 @ 2.80GHz (2793.02-MHz
K8-class CPU)
Origin =3D "GenuineIntel" Id =3D 0x106a5 Family =3D 6 Model =3D 1a
Stepping =3D 5
=20
Features=3D0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PG=
E
,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
=20
Features2=3D0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,P=
D
CM,DCA,SSE4.1,SSE4.2,POPCNT>
AMD Features=3D0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=3D0x1<LAHF>
TSC: P-state invariant
real memory =3D 6442450944 (6144 MB)
avail memory =3D 6170185728 (5884 MB)
ACPI APIC Table: <DELL PE_SC3 >
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads
cpu0 (BSP): APIC ID: 16
cpu1 (AP): APIC ID: 17
cpu2 (AP): APIC ID: 18
cpu3 (AP): APIC ID: 19
cpu4 (AP): APIC ID: 20
cpu5 (AP): APIC ID: 21
cpu6 (AP): APIC ID: 22
cpu7 (AP): APIC ID: 23
cpu8 (AP): APIC ID: 0
cpu9 (AP): APIC ID: 1
cpu10 (AP): APIC ID: 2
cpu11 (AP): APIC ID: 3
cpu12 (AP): APIC ID: 4
cpu13 (AP): APIC ID: 5
cpu14 (AP): APIC ID: 6
cpu15 (AP): APIC ID: 7
....
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.4>
port 0xbcc0-0xbcdf mem 0xdd800000-0xdd9fffff,0xdd7f8000-0xdd7fbfff irq
50 at device 0.0 on pci8
ix0: Using MSIX interrupts with 9 vectors
ix0: [ITHREAD]=20
ix0: [ITHREAD]=20
ix0: [ITHREAD]=20
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]=20
ix0: [ITHREAD]=20
ix0: [ITHREAD]=20
ix0: Ethernet address: 00:1b:21:d8:d8:be
ix0: PCI Express Bus: Speed 5.0Gb/s Width x8
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.4>
port 0xbce0-0xbcff mem 0xdda00000-0xddbfffff,0xdd7fc000-0xdd7fffff irq
40 at device 0.1 on pci8
ix1: Using MSIX interrupts with 9 vectors
ix1: [ITHREAD]=20
ix1: [ITHREAD]=20
ix1: [ITHREAD]=20
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]=20
ix1: Ethernet address: 00:1b:21:d8:d8:bf
ix1: PCI Express Bus: Speed 5.0Gb/s Width x8
=20
# ifconfig -a
ix0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
=20
options=3D405b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VL=
A
N_HWTSO>
ether 00:1b:21:d8:d8:be
inet 192.168.248.228 netmask 0xffffffe0 broadcast
192.168.248.255
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
ix1: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
=20
options=3D405b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VL=
A
N_HWTSO>
ether 00:1b:21:d8:d8:bf
inet 192.168.250.40 netmask 0xffffff00 broadcast 192.168.250.255
inet 192.168.250.23 netmask 0xffffff00 broadcast 192.168.250.255
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
=20
=20
---- Kirk
=20
Kirk Davis
Supervisor - Network Operations
ITS, Edmonton Public Schools
One Kingsway Ave. NW
Edmonton, Alberta
T5H 4G9
(780) 429-8308
=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?529374128DC1B04D9D037911B8E8F0531095BEF8>