From owner-freebsd-net@FreeBSD.ORG Mon May 14 20:02:24 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EF0BA106564A for ; Mon, 14 May 2012 20:02:24 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6FCDD8FC0A for ; Mon, 14 May 2012 20:02:24 +0000 (UTC) Received: by werg1 with SMTP id g1so3544284wer.13 for ; Mon, 14 May 2012 13:02:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Q5yc5y+ywRi5iv7cIfU7K13C2u4RXmlWNj2xY4k7CJk=; b=vGbenptSmYpF0iHS7MHeBKJWYH0nUsQyxzya20CX3ZmzFQfpbFdxdKs0hvfauHKzRA Qahoep9g39zQ5TpGYcuZ/lJW4VNhJMOKa+O7e+MmBe77HsQJG6VIZNTXpSLEgAX6qQ8p m1dGI10RT6+higVuZBc+mZC5i4OX+rv+eEZA2yoZaj+tAG/fDMRwcPnpyInDHd7wP+T3 l8Qv3/0EdhgqbKHUcFS9vMmrUE0Z8i8eKN/wGHFKXmQ1DCFqsbpE1y9HgM1kulvbO3p5 7l0P2POGRpQbkWpueqTX229yHm5FXu38FpJBBirKSjIQIrC5NmKFOJnskvD5zyzHY9CG QW9Q== MIME-Version: 1.0 Received: by 10.216.132.226 with SMTP id o76mr5912301wei.93.1337025743345; Mon, 14 May 2012 13:02:23 -0700 (PDT) Received: by 10.180.7.103 with HTTP; Mon, 14 May 2012 13:02:23 -0700 (PDT) In-Reply-To: <529374128DC1B04D9D037911B8E8F0531095BEF8@Exchange26.EDU.epsb.ca> References: <529374128DC1B04D9D037911B8E8F0531095BEF8@Exchange26.EDU.epsb.ca> Date: Mon, 14 May 2012 13:02:23 -0700 Message-ID: From: Jack Vogel To: Kirk Davis Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: 10Gb/s ixgbe and mbuf performance tuning. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 May 2012 20:02:25 -0000 Do you have LRO enabled? Also, you say it stops routing, do you mean it passes non-routing traffic, or does everything stop? Jack On Mon, May 14, 2012 at 9:49 AM, Kirk Davis wrote: > Hi, > > I am running FreeBSD 8.1p3 amd64 with 6Gb of RAM. This > is a router using Quagga for BGP. It has two 10Gb/s network connections > using the intel ixgbe 2.4.4 driver. It is handling routing for about > 200 remote sites and about 300k simultaneous sessions through the box. > After a few hours at high traffic volumes (just over a 1Gb/s) it will > stop routing any traffic. It look like I am running out of mbuf's. I > have increased them but I am worried that there is a limit as to how > much I can increase them without starving the kernel of space. Console > is still responsive but no network. > > > > I have looked over Tuning(7) and some other great > articles on the net on performance tuning FreeBSD but it has left me a > little confused. Can someone help to explain the best way to > troubleshoot this issue and any formulas to tune it. > > > > Here is some information: > > # netstat -m > > 66042/18063/84105 mbufs in use (current/cache/total) > > 66039/15931/81970/524288 mbuf clusters in use (current/cache/total/max) > > 66039/15881 mbuf+clusters out of packet secondary zone in use > (current/cache) > > 0/5/5/524288 4k (page size) jumbo clusters in use > (current/cache/total/max) > > 0/0/0/131072 9k jumbo clusters in use (current/cache/total/max) > > 0/0/0/65536 16k jumbo clusters in use (current/cache/total/max) > > 148588K/36397K/184986K bytes allocated to network (current/cache/total) > > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > > 0/0/0 sfbufs in use (current/peak/max) > > 0 requests for sfbufs denied > > 0 requests for sfbufs delayed > > 0 requests for I/O initiated by sendfile > > 0 calls to protocol drain routines > > > > > > # more /boot/loader.conf > > geom_mirror_load="YES" > > loader_logo="beastie" > > vesa_load="YES" > > ixgbe_load="YES" > > hw.ixgbe.txd=4096 > > hw.ixgbe.rxd=4096 > > kern.ipc.nmbjumbop=524288 > > kern.ipc.nmbclusters=524288 > > kern.ipc.maxsockbuf=4194304 > > > > # more /etc/sysctl.conf > > #security.bsd.see_other_uids=0 > > kern.ipc.nmbjumbop=524288 > > kern.ipc.nmbclusters=524288 > > kern.ipc.maxsockbuf=4194304 > > hw.intr_storm_threshold=9000 > > net.inet.icmp.icmplim=1000 > > > > Copyright (c) 1992-2010 The FreeBSD Project. > > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > > The Regents of the University of California. All rights > reserved. > > FreeBSD is a registered trademark of The FreeBSD Foundation. > > FreeBSD 8.1-RELEASE-p1 #3: Thu Mar 29 08:53:27 MDT 2012 > > root@SchoolGW2.epsb.ca:/usr/obj/usr/src/sys/DELLR710x64 amd64 > > Timecounter "i8254" frequency 1193182 Hz quality 0 > > CPU: Intel(R) Xeon(R) CPU X5560 @ 2.80GHz (2793.02-MHz > K8-class CPU) > > Origin = "GenuineIntel" Id = 0x106a5 Family = 6 Model = 1a > Stepping = 5 > > > Features=0xbfebfbff ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > > > Features2=0x9ce3bd CM,DCA,SSE4.1,SSE4.2,POPCNT> > > AMD Features=0x28100800 > > AMD Features2=0x1 > > TSC: P-state invariant > > real memory = 6442450944 (6144 MB) > > avail memory = 6170185728 (5884 MB) > > ACPI APIC Table: > > FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs > > FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads > > cpu0 (BSP): APIC ID: 16 > > cpu1 (AP): APIC ID: 17 > > cpu2 (AP): APIC ID: 18 > > cpu3 (AP): APIC ID: 19 > > cpu4 (AP): APIC ID: 20 > > cpu5 (AP): APIC ID: 21 > > cpu6 (AP): APIC ID: 22 > > cpu7 (AP): APIC ID: 23 > > cpu8 (AP): APIC ID: 0 > > cpu9 (AP): APIC ID: 1 > > cpu10 (AP): APIC ID: 2 > > cpu11 (AP): APIC ID: 3 > > cpu12 (AP): APIC ID: 4 > > cpu13 (AP): APIC ID: 5 > > cpu14 (AP): APIC ID: 6 > > cpu15 (AP): APIC ID: 7 > > .... > > ix0: > port 0xbcc0-0xbcdf mem 0xdd800000-0xdd9fffff,0xdd7f8000-0xdd7fbfff irq > 50 at device 0.0 on pci8 > > ix0: Using MSIX interrupts with 9 vectors > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: [ITHREAD] > > ix0: Ethernet address: 00:1b:21:d8:d8:be > > ix0: PCI Express Bus: Speed 5.0Gb/s Width x8 > > ix1: > port 0xbce0-0xbcff mem 0xdda00000-0xddbfffff,0xdd7fc000-0xdd7fffff irq > 40 at device 0.1 on pci8 > > ix1: Using MSIX interrupts with 9 vectors > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: [ITHREAD] > > ix1: Ethernet address: 00:1b:21:d8:d8:bf > > ix1: PCI Express Bus: Speed 5.0Gb/s Width x8 > > > > # ifconfig -a > > ix0: flags=8843 metric 0 mtu > 1500 > > > options=405b8 N_HWTSO> > > ether 00:1b:21:d8:d8:be > > inet 192.168.248.228 netmask 0xffffffe0 broadcast > 192.168.248.255 > > media: Ethernet autoselect (10Gbase-T ) > > status: active > > ix1: flags=8843 metric 0 mtu > 1500 > > > options=405b8 N_HWTSO> > > ether 00:1b:21:d8:d8:bf > > inet 192.168.250.40 netmask 0xffffff00 broadcast 192.168.250.255 > > inet 192.168.250.23 netmask 0xffffff00 broadcast 192.168.250.255 > > media: Ethernet autoselect (10Gbase-T ) > > status: active > > > > > > ---- Kirk > > > > Kirk Davis > > Supervisor - Network Operations > > ITS, Edmonton Public Schools > > One Kingsway Ave. NW > > Edmonton, Alberta > > T5H 4G9 > > (780) 429-8308 > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >