Date: Mon, 14 May 2012 13:02:23 -0700 From: Jack Vogel <jfvogel@gmail.com> To: Kirk Davis <kirk.davis@epsb.ca> Cc: freebsd-net@freebsd.org Subject: Re: 10Gb/s ixgbe and mbuf performance tuning. Message-ID: <CAFOYbc=NsEy1-eT_XUvhrF7RQvkVbW0SDfHnh7bpCxKib96oWw@mail.gmail.com> In-Reply-To: <529374128DC1B04D9D037911B8E8F0531095BEF8@Exchange26.EDU.epsb.ca> References: <529374128DC1B04D9D037911B8E8F0531095BEF8@Exchange26.EDU.epsb.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Do you have LRO enabled? Also, you say it stops routing, do you mean it passes non-routing traffic, or does everything stop? Jack On Mon, May 14, 2012 at 9:49 AM, Kirk Davis <kirk.davis@epsb.ca> wrote: > Hi, > > I am running FreeBSD 8.1p3 amd64 with 6Gb of RAM. This > is a router using Quagga for BGP. It has two 10Gb/s network connections > using the intel ixgbe 2.4.4 driver. It is handling routing for about > 200 remote sites and about 300k simultaneous sessions through the box. > After a few hours at high traffic volumes (just over a 1Gb/s) it will > stop routing any traffic. It look like I am running out of mbuf's. I > have increased them but I am worried that there is a limit as to how > much I can increase them without starving the kernel of space. Console > is still responsive but no network. > > > > I have looked over Tuning(7) and some other great > articles on the net on performance tuning FreeBSD but it has left me a > little confused. Can someone help to explain the best way to > troubleshoot this issue and any formulas to tune it. > > > > Here is some information: > > # netstat -m > > 66042/18063/84105 mbufs in use (current/cache/total) > > 66039/15931/81970/524288 mbuf clusters in use (current/cache/total/max) > > 66039/15881 mbuf+clusters out of packet secondary zone in use > (current/cache) > > 0/5/5/524288 4k (page size) jumbo clusters in use > (current/cache/total/max) > > 0/0/0/131072 9k jumbo clusters in use (current/cache/total/max) > > 0/0/0/65536 16k jumbo clusters in use (current/cache/total/max) > > 148588K/36397K/184986K bytes allocated to network (current/cache/total) > > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > > 0/0/0 sfbufs in use (current/peak/max) > > 0 requests for sfbufs denied > > 0 requests for sfbufs delayed > > 0 requests for I/O initiated by sendfile > > 0 calls to protocol drain routines > > > > > > # more /boot/loader.conf > > geom_mirror_load="YES" > > loader_logo="beastie" > > vesa_load="YES" > > ixgbe_load="YES" > > hw.ixgbe.txd=4096 > > hw.ixgbe.rxd=4096 > > kern.ipc.nmbjumbop=524288 > > kern.ipc.nmbclusters=524288 > > kern.ipc.maxsockbuf=4194304 > > > > # more /etc/sysctl.conf > > #security.bsd.see_other_uids=0 > > kern.ipc.nmbjumbop=524288 > > kern.ipc.nmbclusters=524288 > > kern.ipc.maxsockbuf=4194304 > > hw.intr_storm_threshold=9000 > > net.inet.icmp.icmplim=1000 > > > > Copyright (c) 1992-2010 The FreeBSD Project. > > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > > The Regents of the University of California. All rights > reserved. > > FreeBSD is a registered trademark of The FreeBSD Foundation. > > FreeBSD 8.1-RELEASE-p1 #3: Thu Mar 29 08:53:27 MDT 2012 > > root@SchoolGW2.epsb.ca:/usr/obj/usr/src/sys/DELLR710x64 amd64 > > Timecounter "i8254" frequency 1193182 Hz quality 0 > > CPU: Intel(R) Xeon(R) CPU X5560 @ 2.80GHz (2793.02-MHz > K8-class CPU) > > Origin = "GenuineIntel" Id = 0x106a5 Family = 6 Model = 1a > Stepping = 5 > > > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE > ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > > > Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PD > CM,DCA,SSE4.1,SSE4.2,POPCNT> > > AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> > > AMD Features2=0x1<LAHF> > > TSC: P-state invariant > > real memory = 6442450944 (6144 MB) > > avail memory = 6170185728 (5884 MB) > > ACPI APIC Table: <DELL PE_SC3 > > > FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs > > FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads > > cpu0 (BSP): APIC ID: 16 > > cpu1 (AP): APIC ID: 17 > > cpu2 (AP): APIC ID: 18 > > cpu3 (AP): APIC ID: 19 > > cpu4 (AP): APIC ID: 20 > > cpu5 (AP): APIC ID: 21 > > cpu6 (AP): APIC ID: 22 > > cpu7 (AP): APIC ID: 23 > > cpu8 (AP): APIC ID: 0 > > cpu9 (AP): APIC ID: 1 > > cpu10 (AP): APIC ID: 2 > > cpu11 (AP): APIC ID: 3 > > cpu12 (AP): APIC ID: 4 > > cpu13 (AP): APIC ID: 5 > > cpu14 (AP): APIC ID: 6 > > cpu15 (AP): APIC ID: 7 > > .... > > ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.4> > port 0xbcc0-0xbcdf mem 0xdd800000-0xdd9fffff,0xdd7f8000-0xdd7fbfff irq > 50 at device 0.0 on pci8 > > ix0: Using MSIX interrupts with 9 vectors > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: Ethernet address: 00:1b:21:d8:d8:be > > ix0: PCI Express Bus: Speed 5.0Gb/s Width x8 > > ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.4> > port 0xbce0-0xbcff mem 0xdda00000-0xddbfffff,0xdd7fc000-0xdd7fffff irq > 40 at device 0.1 on pci8 > > ix1: Using MSIX interrupts with 9 vectors > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: Ethernet address: 00:1b:21:d8:d8:bf > > ix1: PCI Express Bus: Speed 5.0Gb/s Width x8 > > > > # ifconfig -a > > ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu > 1500 > > > options=405b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLA > N_HWTSO> > > ether 00:1b:21:d8:d8:be > > inet 192.168.248.228 netmask 0xffffffe0 broadcast > 192.168.248.255 > > media: Ethernet autoselect (10Gbase-T <full-duplex>) > > status: active > > ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu > 1500 > > > options=405b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLA > N_HWTSO> > > ether 00:1b:21:d8:d8:bf > > inet 192.168.250.40 netmask 0xffffff00 broadcast 192.168.250.255 > > inet 192.168.250.23 netmask 0xffffff00 broadcast 192.168.250.255 > > media: Ethernet autoselect (10Gbase-T <full-duplex>) > > status: active > > > > > > ---- Kirk > > > > Kirk Davis > > Supervisor - Network Operations > > ITS, Edmonton Public Schools > > One Kingsway Ave. NW > > Edmonton, Alberta > > T5H 4G9 > > (780) 429-8308 > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFOYbc=NsEy1-eT_XUvhrF7RQvkVbW0SDfHnh7bpCxKib96oWw>