From owner-freebsd-current@FreeBSD.ORG Thu Dec 8 14:22:44 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 228851065670 for ; Thu, 8 Dec 2011 14:22:44 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id 7FC798FC0A for ; Thu, 8 Dec 2011 14:22:43 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1RYeax-000F52-0B for freebsd-current@freebsd.org; Thu, 08 Dec 2011 18:05:07 +0400 Date: Thu, 8 Dec 2011 18:05:06 +0400 From: Slawa Olhovchenkov To: freebsd-current@freebsd.org Message-ID: <20111208140506.GA51886@zxy.spb.ru> References: <20111205192703.GA49118@onelab2.iet.unipi.it> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111205192703.GA49118@onelab2.iet.unipi.it> User-Agent: Mutt/1.5.21 (2010-09-15) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Subject: Re: datapoints on 10G throughput with TCP ? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Dec 2011 14:22:44 -0000 On Mon, Dec 05, 2011 at 08:27:03PM +0100, Luigi Rizzo wrote: > Hi, > I am trying to establish the baseline performance for 10G throughput > over TCP, and would like to collect some data points. As a testing > program i am using nuttcp from ports (as good as anything, i > guess -- it is reasonably flexible, and if you use it in > TCP with relatively large writes, the overhead for syscalls > and gettimeofday() shouldn't kill you). > > I'd be very grateful if you could do the following test: > > - have two machines connected by a 10G link > - on one run "nuttcp -S" > - on the other one run "nuttcp -t -T 5 -w 128 -v the.other.ip" > > and send me a dump of the output, such as the one(s) at the end of > the message. > > I am mostly interested in two configurations: > - one over loopback, which should tell how fast is the CPU+memory > As an example, one of my machines does about 15 Gbit/s, and > one of the faster ones does about 44 Gbit/s > > - one over the wire using 1500 byte mss. Here it really matters > how good is the handling of small MTUs. > > As a data point, on my machines i get 2..3.5 Gbit/s on the > "slow" machine with a 1500 byte mtu and default card setting. > Clearing the interrupt mitigation register (so no mitigation) > brings the rate to 5-6 Gbit/s. Same hardware with linux does > about 8 Gbit/s. HEAD seems 10-20% slower than RELENG_8 though i > am not sure who is at fault. > > The receive side is particularly critical - on FreeBSD > the receiver is woken up every two packets (do the math > below, between the number of rx calls and throughput and mss), > resulting in almost 200K activations per second, and despite > the fact that interrupt mitigation is set to a much lower > value (so incoming packets should be batched). > On linux, i see much fewer reads, presumably the process is > woken up only at the end of a burst. About relative performance FreeBSD and Linux I wrote in -performance@ at Jan'11 (Interrupt performance) > > ------------ EXAMPLES OF OUTPUT ---------------------- > > > nuttcp -t -T 5 -w 128 -v 10.0.1.2 > nuttcp-t: v6.1.2: socket > nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.0.1.2 > nuttcp-t: time limit = 5.00 seconds > nuttcp-t: connect to 10.0.1.2 with mss=1460, RTT=0.103 ms > nuttcp-t: send window size = 131400, receive window size = 65700 > nuttcp-t: 3095.0982 MB in 5.00 real seconds = 633785.85 KB/sec = 5191.9737 Mbps > nuttcp-t: host-retrans = 0 > nuttcp-t: 49522 I/O calls, msec/call = 0.10, calls/sec = 9902.99 > nuttcp-t: 0.0user 2.7sys 0:05real 54% 100i+2639d 752maxrss 0+3pf 258876+6csw > > nuttcp-r: v6.1.2: socket > nuttcp-r: buflen=65536, nstream=1, port=5001 tcp > nuttcp-r: accept from 10.0.1.4 > nuttcp-r: send window size = 33580, receive window size = 131400 > nuttcp-r: 3095.0982 MB in 5.17 real seconds = 613526.42 KB/sec = 5026.0084 Mbps > nuttcp-r: 1114794 I/O calls, msec/call = 0.00, calls/sec = 215801.03 > nuttcp-r: 0.1user 3.5sys 0:05real 69% 112i+1104d 626maxrss 0+15pf 507653+188csw > > > > > nuttcp -t -T 5 -w 128 -v localhost > nuttcp-t: v6.1.2: socket > nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost > nuttcp-t: time limit = 5.00 seconds > nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.051 ms > nuttcp-t: send window size = 143360, receive window size = 71680 > nuttcp-t: 26963.4375 MB in 5.00 real seconds = 5521440.59 KB/sec = 45231.6413 Mbps > nuttcp-t: host-retrans = 0 > nuttcp-t: 431415 I/O calls, msec/call = 0.01, calls/sec = 86272.51 > nuttcp-t: 0.0user 4.6sys 0:05real 93% 102i+2681d 774maxrss 0+3pf 2510+1csw > > nuttcp-r: v6.1.2: socket > nuttcp-r: buflen=65536, nstream=1, port=5001 tcp > nuttcp-r: accept from 127.0.0.1 > nuttcp-r: send window size = 43008, receive window size = 143360 > nuttcp-r: 26963.4375 MB in 5.20 real seconds = 5313135.74 KB/sec = 43525.2080 Mbps > nuttcp-r: 767807 I/O calls, msec/call = 0.01, calls/sec = 147750.09 > nuttcp-r: 0.1user 3.9sys 0:05real 79% 98i+2570d 772maxrss 0+16pf 311014+8csw > > > on the server, run " > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"