From owner-freebsd-net@FreeBSD.ORG Mon May 14 16:50:29 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2D05A106564A for ; Mon, 14 May 2012 16:50:29 +0000 (UTC) (envelope-from kirk.davis@epsb.ca) Received: from OWA01.EDU.epsb.ca (owa01.epsb.ca [198.161.119.28]) by mx1.freebsd.org (Postfix) with ESMTP id E1D088FC0C for ; Mon, 14 May 2012 16:50:28 +0000 (UTC) Received: from Exchange26.EDU.epsb.ca ([10.0.5.123]) by OWA01.EDU.epsb.ca with Microsoft SMTPSVC(6.0.3790.4675); Mon, 14 May 2012 10:49:22 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Date: Mon, 14 May 2012 10:49:28 -0600 Message-ID: <529374128DC1B04D9D037911B8E8F0531095BEF8@Exchange26.EDU.epsb.ca> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: 10Gb/s ixgbe and mbuf performance tuning. Thread-Index: Ac0x725qHdZ6+4yiR/OOLwSecus7fg== From: "Kirk Davis" To: X-OriginalArrivalTime: 14 May 2012 16:49:22.0465 (UTC) FILETIME=[8360C910:01CD31F1] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: 10Gb/s ixgbe and mbuf performance tuning. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 May 2012 16:50:29 -0000 Hi, I am running FreeBSD 8.1p3 amd64 with 6Gb of RAM. This is a router using Quagga for BGP. It has two 10Gb/s network connections using the intel ixgbe 2.4.4 driver. It is handling routing for about 200 remote sites and about 300k simultaneous sessions through the box. After a few hours at high traffic volumes (just over a 1Gb/s) it will stop routing any traffic. It look like I am running out of mbuf's. I have increased them but I am worried that there is a limit as to how much I can increase them without starving the kernel of space. Console is still responsive but no network. =20 I have looked over Tuning(7) and some other great articles on the net on performance tuning FreeBSD but it has left me a little confused. Can someone help to explain the best way to troubleshoot this issue and any formulas to tune it. =20 Here is some information: # netstat -m 66042/18063/84105 mbufs in use (current/cache/total) 66039/15931/81970/524288 mbuf clusters in use (current/cache/total/max) 66039/15881 mbuf+clusters out of packet secondary zone in use (current/cache) 0/5/5/524288 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/131072 9k jumbo clusters in use (current/cache/total/max) 0/0/0/65536 16k jumbo clusters in use (current/cache/total/max) 148588K/36397K/184986K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines =20 =20 # more /boot/loader.conf geom_mirror_load=3D"YES" loader_logo=3D"beastie" vesa_load=3D"YES" ixgbe_load=3D"YES" hw.ixgbe.txd=3D4096 hw.ixgbe.rxd=3D4096 kern.ipc.nmbjumbop=3D524288 kern.ipc.nmbclusters=3D524288 kern.ipc.maxsockbuf=3D4194304 =20 # more /etc/sysctl.conf=20 #security.bsd.see_other_uids=3D0 kern.ipc.nmbjumbop=3D524288 kern.ipc.nmbclusters=3D524288 kern.ipc.maxsockbuf=3D4194304 hw.intr_storm_threshold=3D9000 net.inet.icmp.icmplim=3D1000 =20 Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-RELEASE-p1 #3: Thu Mar 29 08:53:27 MDT 2012 root@SchoolGW2.epsb.ca:/usr/obj/usr/src/sys/DELLR710x64 amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU X5560 @ 2.80GHz (2793.02-MHz K8-class CPU) Origin =3D "GenuineIntel" Id =3D 0x106a5 Family =3D 6 Model =3D 1a Stepping =3D 5 =20 Features=3D0xbfebfbff =20 Features2=3D0x9ce3bd AMD Features=3D0x28100800 AMD Features2=3D0x1 TSC: P-state invariant real memory =3D 6442450944 (6144 MB) avail memory =3D 6170185728 (5884 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 16 cpu1 (AP): APIC ID: 17 cpu2 (AP): APIC ID: 18 cpu3 (AP): APIC ID: 19 cpu4 (AP): APIC ID: 20 cpu5 (AP): APIC ID: 21 cpu6 (AP): APIC ID: 22 cpu7 (AP): APIC ID: 23 cpu8 (AP): APIC ID: 0 cpu9 (AP): APIC ID: 1 cpu10 (AP): APIC ID: 2 cpu11 (AP): APIC ID: 3 cpu12 (AP): APIC ID: 4 cpu13 (AP): APIC ID: 5 cpu14 (AP): APIC ID: 6 cpu15 (AP): APIC ID: 7 .... ix0: port 0xbcc0-0xbcdf mem 0xdd800000-0xdd9fffff,0xdd7f8000-0xdd7fbfff irq 50 at device 0.0 on pci8 ix0: Using MSIX interrupts with 9 vectors ix0: [ITHREAD]=20 ix0: [ITHREAD]=20 ix0: [ITHREAD]=20 ix0: [ITHREAD] ix0: [ITHREAD] ix0: [ITHREAD] ix0: [ITHREAD]=20 ix0: [ITHREAD]=20 ix0: [ITHREAD]=20 ix0: Ethernet address: 00:1b:21:d8:d8:be ix0: PCI Express Bus: Speed 5.0Gb/s Width x8 ix1: port 0xbce0-0xbcff mem 0xdda00000-0xddbfffff,0xdd7fc000-0xdd7fffff irq 40 at device 0.1 on pci8 ix1: Using MSIX interrupts with 9 vectors ix1: [ITHREAD]=20 ix1: [ITHREAD]=20 ix1: [ITHREAD]=20 ix1: [ITHREAD] ix1: [ITHREAD] ix1: [ITHREAD] ix1: [ITHREAD] ix1: [ITHREAD] ix1: [ITHREAD]=20 ix1: Ethernet address: 00:1b:21:d8:d8:bf ix1: PCI Express Bus: Speed 5.0Gb/s Width x8 =20 # ifconfig -a ix0: flags=3D8843 metric 0 mtu 1500 =20 options=3D405b8 ether 00:1b:21:d8:d8:be inet 192.168.248.228 netmask 0xffffffe0 broadcast 192.168.248.255 media: Ethernet autoselect (10Gbase-T ) status: active ix1: flags=3D8843 metric 0 mtu 1500 =20 options=3D405b8 ether 00:1b:21:d8:d8:bf inet 192.168.250.40 netmask 0xffffff00 broadcast 192.168.250.255 inet 192.168.250.23 netmask 0xffffff00 broadcast 192.168.250.255 media: Ethernet autoselect (10Gbase-T ) status: active =20 =20 ---- Kirk =20 Kirk Davis Supervisor - Network Operations ITS, Edmonton Public Schools One Kingsway Ave. NW Edmonton, Alberta T5H 4G9 (780) 429-8308 =20