From owner-freebsd-hackers@FreeBSD.ORG Thu Apr 10 14:40:57 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 564A137B401; Thu, 10 Apr 2003 14:40:57 -0700 (PDT) Received: from stork.mail.pas.earthlink.net (stork.mail.pas.earthlink.net [207.217.120.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id A3FA343F93; Thu, 10 Apr 2003 14:40:56 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from pool0274.cvx40-bradley.dialup.earthlink.net ([216.244.43.19] helo=mindspring.com) by stork.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 193jmo-0004lA-00; Thu, 10 Apr 2003 14:40:48 -0700 Message-ID: <3E95E446.73B7E510@mindspring.com> Date: Thu, 10 Apr 2003 14:38:14 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: bj@dc.luth.se References: <20030410171640.C44793B2@porter.dc.luth.se> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a477423e8d118ab8c71501f12832479819a7ce0e8f8d31aa3f350badd9bab72f9c350badd9bab72f9c cc: freebsd-hackers@freebsd.org cc: freebsd-performance@freebsd.org cc: "Jin Guojun \[DSD\]" cc: Eric Anderson cc: David Gilbert Subject: Re: tcp_output starving -- is due to mbuf get delay? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Apr 2003 21:40:57 -0000 Borje Josefsson wrote: > > It's interesting - I'm only getting about 320mb/s.. I must be hitting a > > similar problem. I'm not nearly as adept at hacking code to find bugs > > though. :( > > 320 Mbit/sec seems familiar, this was what I got when I first tried on a > system with "traditional" PCI bus. Changing the OS to NetBSD on that box > bumped that to 525 Mbit/sec. You need wide PCI (or preferrably PCI-X for > this). 32bit x 33Mhz = 1.03Gbit/S, burst rate 32bit x 66Mhz = 2.06Gbit/S, burst rate 64bit x 66MHz = 4.12Gbit/S, burst rate So it's entirely possible to keep up with 2 1Gbit ethernet cards in standard 64bit PCI bus slots, no problem, without running into bus limitations. PCI-X gets you to 8Gbit/S; you don't need PCI-X for Gbit, or even 2Gbit. > What happens in that case for me is that I run out of CPU resources. Try > running "top" in one window and "netstat 1" in another while bashing the > net with ttcp. This is incredibly bizarre. It's very hard to saturate the CPU at only 1Gbit: in all cases, you are I/O bound, not CPU bound, and not memory bandwidth bound. > IMPORTANT NOTE: Several tests here has shown that this is VERY BADLY > affected if You have too much LAN equipment (especially VLAN seems to be > harmful) at the edges. My speed of 960 Mbit/sec fell to 165 just by adding > 10 feet of cable and two switches :-( The products that Jeffrey Hsu and I and Alfred and Jon Mini worked on at a previous company had no problems at all on a 1Gbit/S saturating the link, even through a VLAN trunk through Cisco and one other less intelligent switch (i.e. two switches and a VLAN trunk). Maybe your network cards don't do hardware interrupt coelescing? Or maybe you are sending 1 byte packets instead of MTU-sized packets, or something? We were using Tigon III's, with 1K packets. -- Terry