From owner-freebsd-net@FreeBSD.ORG Fri Feb 19 03:43:37 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 886FA106566C; Fri, 19 Feb 2010 03:43:37 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id E0E8F8FC0A; Fri, 19 Feb 2010 03:43:36 +0000 (UTC) Received: by vws14 with SMTP id 14so1034266vws.13 for ; Thu, 18 Feb 2010 19:43:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=nhp52rbB1zv6YcnoYDDpyul2eQw7N/Erc47mTnTIPTc=; b=uo9cJIR6hwUMRnC5KpCas3m0Cuvl2JKLuREF87gN2Rw+iuXkM7efx1jtvczTOAIzTX 0oZH0VSoS9BHvLhdq5jlFBBIR8fjskxAsvksQ8LT+NXxYOMlVsbBujYauAu56Mc3QC+g qvSgctcT/v7FvkRpgr/zjaie420KL4Lc3klTE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=geHJnWcQGOk3hPXobx7Sj0l4l6C/k1+ZO7HbsgCgkCR2Hy0vVAYfqNy6Hk9vdd6tSm yB+7axgQJ8eVnEGYD6u6bIj1d5hM/uRu4xT62ykLD8obR2zBQL9LEoc2CHAcfNlfpqaV BdlqseZ878HS7XTc5hIi1IssathshIsbsfbqo= Received: by 10.220.127.4 with SMTP id e4mr3740751vcs.79.1266551016030; Thu, 18 Feb 2010 19:43:36 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id 30sm33174334vws.1.2010.02.18.19.43.33 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 18 Feb 2010 19:43:34 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Thu, 18 Feb 2010 19:42:55 -0800 From: Pyun YongHyeon Date: Thu, 18 Feb 2010 19:42:55 -0800 To: Maxim Sobolev Message-ID: <20100219034255.GG11675@michelle.cdnetworks.com> References: <4B79297D.9080403@FreeBSD.org> <4B79205B.619A0A1A@verizon.net> <4B7ADFC6.7020202@FreeBSD.org> <4B7DE3CC.7040704@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="Md/poaVZ8hnGTzuv" Content-Disposition: inline In-Reply-To: <4B7DE3CC.7040704@FreeBSD.org> User-Agent: Mutt/1.4.2.3i Cc: Sergey Babkin , freebsd-net@freebsd.org, Alfred Perlstein , Jack Vogel , FreeBSD Hackers , "David G. Lawrence" Subject: Re: Sudden mbuf demand increase and shortage under the load (igb issue?) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Feb 2010 03:43:37 -0000 --Md/poaVZ8hnGTzuv Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Thu, Feb 18, 2010 at 05:05:16PM -0800, Maxim Sobolev wrote: > Folks, > > Indeed, it looks like igb(4) issue. Replacing the card with the > desktop-grade em(4)-supported card has fixed the problem for us. The > system has been happily pushing 110mbps worth of RTP traffic and 2000 > concurrent calls without any problems for two days now. > > em0@pci0:7:0:0: class=0x020000 card=0xa01f8086 chip=0x10d38086 rev=0x00 > hdr=0x00 > vendor = 'Intel Corporation' > class = network > subclass = ethernet > > em0: port 0xec00-0xec1f mem > 0xfbee0000-0xfbefffff,0xfbe00000-0xfbe7ffff,0xfbedc000-0xfbedffff irq 24 > at device 0.0 on pci7 > em0: Using MSIX interrupts > em0: [ITHREAD] > em0: [ITHREAD] > em0: [ITHREAD] > em0: Ethernet address: 00:1b:21:50:02:49 > > I really think that this has to be addressed before 7.3 release is out. > FreeBSD used to be famous for its excellent network performance and it's > shame to see that deteriorating due to sub-standard quality drivers. > Especially when there is a multi-billion vendor supporting the driver in > question. No finger pointing, but it really looks like either somebody > is not doing his job or the said vendor doesn't care so much about > supporting FreeBSD. I am pretty sure the vendor in question has access > to numerous load-testing tools, that should have catched this issue. > > This is the second time during the past 6 months I have issue with the > quality of the Intel-based drivers - the first one is filed as > kern/140326, which has stalled apparently despite me providing all > necessary debug information. > I can reproduce this bug on my box and I guess the root cause comes from PBA(Packet Buffer Allocation) configuration. Some controllers seems to require more TX buffer size to use 9000 MTU. The datasheet is not clear which controller has how much amount of Packet Buffer storage. This parameter seems to affect performance a lot because increasing TX buffer size results in decreasing RX buffer size. The attached patch seems to fix the issue for me but Jack may know better the hardware details as publicly available datasheet seems to be useless here. --Md/poaVZ8hnGTzuv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="em.ich10.jumbo.diff" Index: sys/dev/e1000/if_em.c =================================================================== --- sys/dev/e1000/if_em.c (revision 204011) +++ sys/dev/e1000/if_em.c (working copy) @@ -1384,7 +1384,12 @@ case e1000_ich9lan: case e1000_ich10lan: case e1000_pchlan: +#if 1 + /* Make 9000 MTU work. */ + pba = E1000_PBA_18K; +#else pba = E1000_PBA_10K; +#endif break; case e1000_ich8lan: pba = E1000_PBA_8K; --Md/poaVZ8hnGTzuv--