From owner-freebsd-hackers Sun Dec 2 13:32:33 2001 Delivered-To: freebsd-hackers@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 927A737B417 for ; Sun, 2 Dec 2001 13:32:30 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.11.6/8.9.1) id fB2LWO093222; Sun, 2 Dec 2001 13:32:24 -0800 (PST) (envelope-from dillon) Date: Sun, 2 Dec 2001 13:32:24 -0800 (PST) From: Matthew Dillon Message-Id: <200112022132.fB2LWO093222@apollo.backplane.com> To: Lamont Granquist Cc: Richard Sharpe , Subject: Re: Patch #3 (TCP / Linux / Performance) References: <20011202121147.G92925-100000@coredump.scriptkiddie.org> Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG :On Sun, 2 Dec 2001, Matthew Dillon wrote: :> Throughput 47.2446 MB/sec (NB=59.0558 MB/sec 472.446 MBit/sec) 20 procs :> :> It seems to max-out at around 75,000 packets per second (input + output). :> :> I doubt these results could be duplicated on anything but a DELL2550. :> It dedicates an entire internal 64 bit 66MHz PCI bus just to the :> on-board gigabit ethernet. : :What is the remaining bottleneck in these tests? CPU? Interrupts? What :would you need to do to get that closer to the theoretical limit :(something around 920 Mbs for GigE IIRC)? The tbench test is mainly a transactional test (don't ask me why it only reports bandwidth). So the ultimate limiting factor is going to be cpu in regards to interrupt processing and transactional latency. It is fairly easy to saturated the GigE on the DELL if all you are doing is pushing a huge amount of data over a single TCP connection. This is connecting to inetd running a dd if=/dev/zero bs=32k on a machine with the rfc sysctl's turned on and 262144 byte send and receive buffers, without jumbo frames (my gigE switch doesn't support them :-( ). input (Total) output packets errs bytes packets errs bytes colls 82064 0 124238900 41033 0 2215718 0 82077 0 124263124 41039 0 2216142 0 82065 0 124244956 41033 0 2215818 0 82103 0 124302488 41052 0 2216836 0 82067 0 124247984 41035 0 2215872 0 81684 0 123668122 40842 0 2205558 0 82114 0 124319142 41057 0 2217122 0 Wow. 120,000 packets per second (In+Out) If I reduce the mtu to 250 it can sustain around 210,000 pps (In+Out). If I write a little program to do one-byte transactions.. i.e. send a byte, wait for a byte back, repeat, I get around 16,000 transactions/sec over localhost, 7100 transactions/sec over 100BaseTX, and oddly only 3400 transactions/sec over GigE. The minimum round trip over 100BaseTX for a 64 byte packet is around 0.125 mS, which comes to around 8000 transactions/sec per connection, so the 7100 number is reasonable. The minimum round trip over my GigE seems to average around 0.230mS, resulting in a theoretical transaction rate (per connection) of 4340 t/s. The 3400 number seems a little low but looks reasonable. -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message