From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 19 17:52:38 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A3BEF106566B; Fri, 19 Feb 2010 17:52:38 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-qy0-f183.google.com (mail-qy0-f183.google.com [209.85.221.183]) by mx1.freebsd.org (Postfix) with ESMTP id 0DCF58FC1D; Fri, 19 Feb 2010 17:52:37 +0000 (UTC) Received: by qyk13 with SMTP id 13so195235qyk.13 for ; Fri, 19 Feb 2010 09:52:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=W8Zj/EEqIbdkgFJwYHKKxZFwq/gDt86PoOFCFUuFgEQ=; b=HKcGOGrvUor6k49wbis9rM2NF+z1vyZJigFxJ5cPPbhyvpa+/MSaqXN0jPST2gFSjr sXY16uM5wpii2r9HzWi6vNC+WbtXiBRgAkTtZi+EsJsO5Mmw27O39k3ut4lKq6WWYD2X 4sfTAYpLWeNF8XuY77Ss1GvDsMvZURT9HEzfs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=G+bAMMPYMd2QyLOQK5lbb1mzdi0ln1V8vwCgw4XIkP5XmiuG3ulfs7onpBfcyDxtP7 4tvffD23is3VsBreuxFjdYvPaHMhblJUH3OHzsIhg1QPr8LWPFvlPu6hLkp6Bhh/JVFt 4zmzZYEvU9+9KjGa2wcI0Fp5cBH8jP/xP9Otc= Received: by 10.224.73.76 with SMTP id p12mr3612694qaj.187.1266601957068; Fri, 19 Feb 2010 09:52:37 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id 6sm1221684qwk.10.2010.02.19.09.52.33 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 19 Feb 2010 09:52:35 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Fri, 19 Feb 2010 09:51:55 -0800 From: Pyun YongHyeon Date: Fri, 19 Feb 2010 09:51:55 -0800 To: Jack Vogel Message-ID: <20100219175155.GH11675@michelle.cdnetworks.com> References: <4B79297D.9080403@FreeBSD.org> <4B79205B.619A0A1A@verizon.net> <4B7ADFC6.7020202@FreeBSD.org> <4B7DE3CC.7040704@FreeBSD.org> <20100219034255.GG11675@michelle.cdnetworks.com> <2a41acea1002182343i1ef0124fm8e2f6cc56846454c@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2a41acea1002182343i1ef0124fm8e2f6cc56846454c@mail.gmail.com> User-Agent: Mutt/1.4.2.3i Cc: Maxim Sobolev , Sergey Babkin , FreeBSD Hackers , Alfred Perlstein , freebsd-net@freebsd.org, "David G. Lawrence" Subject: Re: Sudden mbuf demand increase and shortage under the load (igb issue?) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Feb 2010 17:52:38 -0000 On Thu, Feb 18, 2010 at 11:43:20PM -0800, Jack Vogel wrote: > This thread is confusing, first he says its an igb problem, then you offer > an em patch :) > > I have an important rev of igb that I am about ready to release, anyone that > wishes to > test against a problem they have would be welcome to have early access, just > let me > know. > > I am not sure about this ich10 change, there are client NICs that > specifically do NOT > support jumbo frames, I'll need to look into it tomorrow at work. Hmm, then how could I see advertised MSS 8960 on this controller? I guess em(4) already checks MAC type to limit jumbo frame support on these controllers. 09:44:18.701271 IP (tos 0x0, ttl 64, id 32888, offset 0, flags [DF], proto TCP (6), length 60) 192.168.30.2.55754 > 192.168.30.1.22: S, cksum 0xbd82 (incorrect (-> 0x3f62), 529388419:529388419(0) win 65535 09:44:18.701538 IP (tos 0x0, ttl 64, id 40268, offset 0, flags [DF], proto TCP (6), length 60) 192.168.30.1.22 > 192.168.30.2.55754: S, cksum 0xa6ce (correct), 3828990973:3828990973(0) ack 529388420 win 65535 09:44:18.701549 IP (tos 0x0, ttl 64, id 32889, offset 0, flags [DF], proto TCP (6), length 52) 192.168.30.2.55754 > 192.168.30.1.22: ., cksum 0xbd7a (incorrect (-> 0xd2e1), 1:1(0) ack 1 win 8192 em0@pci0:0:25:0: class=0x020000 card=0x027f1028 chip=0x10de8086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = 'Intel Gigabit network connection (83567LM-3 )' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xfdae0000, size 131072, enabled bar [14] = type Memory, range 32, base 0xfdad9000, size 4096, enabled bar [18] = type I/O Port, range 32, base 0xecc0, size 32, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 13[e0] = PCI Advanced Features: FLR TP > > Jack > > > On Thu, Feb 18, 2010 at 7:42 PM, Pyun YongHyeon wrote: > > > On Thu, Feb 18, 2010 at 05:05:16PM -0800, Maxim Sobolev wrote: > > > Folks, > > > > > > Indeed, it looks like igb(4) issue. Replacing the card with the > > > desktop-grade em(4)-supported card has fixed the problem for us. The > > > system has been happily pushing 110mbps worth of RTP traffic and 2000 > > > concurrent calls without any problems for two days now. > > > > > > em0@pci0:7:0:0: class=0x020000 card=0xa01f8086 chip=0x10d38086 rev=0x00 > > > hdr=0x00 > > > vendor = 'Intel Corporation' > > > class = network > > > subclass = ethernet > > > > > > em0: port 0xec00-0xec1f mem > > > 0xfbee0000-0xfbefffff,0xfbe00000-0xfbe7ffff,0xfbedc000-0xfbedffff irq 24 > > > at device 0.0 on pci7 > > > em0: Using MSIX interrupts > > > em0: [ITHREAD] > > > em0: [ITHREAD] > > > em0: [ITHREAD] > > > em0: Ethernet address: 00:1b:21:50:02:49 > > > > > > I really think that this has to be addressed before 7.3 release is out. > > > FreeBSD used to be famous for its excellent network performance and it's > > > shame to see that deteriorating due to sub-standard quality drivers. > > > Especially when there is a multi-billion vendor supporting the driver in > > > question. No finger pointing, but it really looks like either somebody > > > is not doing his job or the said vendor doesn't care so much about > > > supporting FreeBSD. I am pretty sure the vendor in question has access > > > to numerous load-testing tools, that should have catched this issue. > > > > > > This is the second time during the past 6 months I have issue with the > > > quality of the Intel-based drivers - the first one is filed as > > > kern/140326, which has stalled apparently despite me providing all > > > necessary debug information. > > > > > > > I can reproduce this bug on my box and I guess the root cause comes > > from PBA(Packet Buffer Allocation) configuration. Some controllers > > seems to require more TX buffer size to use 9000 MTU. The datasheet > > is not clear which controller has how much amount of Packet Buffer > > storage. This parameter seems to affect performance a lot because > > increasing TX buffer size results in decreasing RX buffer size. The > > attached patch seems to fix the issue for me but Jack may know > > better the hardware details as publicly available datasheet seems > > to be useless here. > >