From owner-freebsd-net@FreeBSD.ORG Fri Dec 3 19:02:06 2010 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3FC371065670 for ; Fri, 3 Dec 2010 19:02:06 +0000 (UTC) (envelope-from tom@tomjudge.com) Received: from eu1sys200aob105.obsmtp.com (eu1sys200aob105.obsmtp.com [207.126.144.118]) by mx1.freebsd.org (Postfix) with SMTP id AB1618FC13 for ; Fri, 3 Dec 2010 19:02:05 +0000 (UTC) Received: from source ([63.174.175.251]) by eu1sys200aob105.postini.com ([207.126.147.11]) with SMTP ID DSNKTPk+qkM3Pmg/aEQv/AyBYDkVXBhfQOXG@postini.com; Fri, 03 Dec 2010 19:02:04 UTC Received: from [172.17.10.53] (unknown [172.17.10.53]) by bbbx3.usdmm.com (Postfix) with ESMTP id 2F596FD01C; Fri, 3 Dec 2010 19:02:02 +0000 (UTC) Message-ID: <4CF93E43.8010801@tomjudge.com> Date: Fri, 03 Dec 2010 13:00:19 -0600 From: Tom Judge User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Lightning/1.0b2 Thunderbird/3.1.6 MIME-Version: 1.0 To: freebsd-net@FreeBSD.org, jfvogel@gmail.com X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: igb and jumbo frames X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Dec 2010 19:02:06 -0000 Hi, So I have been playing around with some new hosts I have been deploying (Dell R710's). The systems have a single dual port card in them: igb0@pci0:5:0:0: class=0x020000 card=0xa04c8086 chip=0x10c98086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 10 messages in map 0x1c enabled cap 10[a0] = PCI-Express 2 endpoint max data 256(512) link x4(x4) igb1@pci0:5:0:1: class=0x020000 card=0xa04c8086 chip=0x10c98086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 10 messages in map 0x1c enabled cap 10[a0] = PCI-Express 2 endpoint max data 256(512) link x4(x4) Running 8.1 these cards panic the system at boot when initializing the jumbo mtu, so to solve this I back ported the stable/8 driver to 8.1 and booted with this kernel. So far so good. However when configuring the interfaces with and mtu of 8192 the system is unable to allocate the required mbufs for the receive queue. I believe the message was from here: http://fxr.watson.org/fxr/source/dev/e1000/if_igb.c#L1209 After a little digging and playing with just one interface i discovered that the default tuning for kern.ipc.nmbjumbo9 was insufficient to run a single interface with jumbo frames as it seemed just the TX queue consumed 90% of the available 9k jumbo clusters. So my question is (well 2 questions really): 1) Should igb be auto tuning kern.ipc.nmbjumbo9 and kern.ipc.nmbclusters up to suite its needs? 2) Should this be documented in igb(4)? Tom -- TJU13-ARIN