Date: Thu, 10 Apr 2003 14:38:14 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: bj@dc.luth.se Cc: David Gilbert <dgilbert@velocet.ca> Subject: Re: tcp_output starving -- is due to mbuf get delay? Message-ID: <3E95E446.73B7E510@mindspring.com> References: <20030410171640.C44793B2@porter.dc.luth.se>
next in thread | previous in thread | raw e-mail | index | archive | help
Borje Josefsson wrote: > > It's interesting - I'm only getting about 320mb/s.. I must be hitting a > > similar problem. I'm not nearly as adept at hacking code to find bugs > > though. :( > > 320 Mbit/sec seems familiar, this was what I got when I first tried on a > system with "traditional" PCI bus. Changing the OS to NetBSD on that box > bumped that to 525 Mbit/sec. You need wide PCI (or preferrably PCI-X for > this). 32bit x 33Mhz = 1.03Gbit/S, burst rate 32bit x 66Mhz = 2.06Gbit/S, burst rate 64bit x 66MHz = 4.12Gbit/S, burst rate So it's entirely possible to keep up with 2 1Gbit ethernet cards in standard 64bit PCI bus slots, no problem, without running into bus limitations. PCI-X gets you to 8Gbit/S; you don't need PCI-X for Gbit, or even 2Gbit. > What happens in that case for me is that I run out of CPU resources. Try > running "top" in one window and "netstat 1" in another while bashing the > net with ttcp. This is incredibly bizarre. It's very hard to saturate the CPU at only 1Gbit: in all cases, you are I/O bound, not CPU bound, and not memory bandwidth bound. > IMPORTANT NOTE: Several tests here has shown that this is VERY BADLY > affected if You have too much LAN equipment (especially VLAN seems to be > harmful) at the edges. My speed of 960 Mbit/sec fell to 165 just by adding > 10 feet of cable and two switches :-( The products that Jeffrey Hsu and I and Alfred and Jon Mini worked on at a previous company had no problems at all on a 1Gbit/S saturating the link, even through a VLAN trunk through Cisco and one other less intelligent switch (i.e. two switches and a VLAN trunk). Maybe your network cards don't do hardware interrupt coelescing? Or maybe you are sending 1 byte packets instead of MTU-sized packets, or something? We were using Tigon III's, with 1K packets. -- Terry
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3E95E446.73B7E510>