From owner-freebsd-current@FreeBSD.ORG Tue Dec 6 14:24:44 2011 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E863C106564A for ; Tue, 6 Dec 2011 14:24:43 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 0FD7A8FC08 for ; Tue, 6 Dec 2011 14:24:42 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id pB6EORTn045343 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 6 Dec 2011 16:24:33 +0200 (EET) (envelope-from daniel@digsys.bg) Message-ID: <4EDE259B.4010502@digsys.bg> Date: Tue, 06 Dec 2011 16:24:27 +0200 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Luigi Rizzo References: <20111205192703.GA49118@onelab2.iet.unipi.it> <2D87D847-A2B7-4E77-B6C1-61D73C9F582F@digsys.bg> <20111205222834.GA50285@onelab2.iet.unipi.it> <4EDDF9F4.9070508@digsys.bg> In-Reply-To: <4EDDF9F4.9070508@digsys.bg> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: current@freebsd.org Subject: Re: datapoints on 10G throughput with TCP ? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2011 14:24:44 -0000 Some tests with updated FreeBSD to 8-stable as of today, compared with the previous run On 06.12.11 13:18, Daniel Kalchev wrote: > > FreeBSD 8.2-STABLE #0: Wed Sep 28 11:23:59 EEST 2011 > CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2403.58-MHz > K8-class CPU) > real memory = 51539607552 (49152 MB) > blade 1: > > # nuttcp -S > # nuttcp -t -T 5 -w 128 -v localhost > nuttcp-t: v6.1.2: socket > nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost > nuttcp-t: time limit = 5.00 seconds > nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.044 ms > nuttcp-t: send window size = 143360, receive window size = 71680 > nuttcp-t: 8959.8750 MB in 5.02 real seconds = 1827635.67 KB/sec = > 14971.9914 Mbps > nuttcp-t: host-retrans = 0 > nuttcp-t: 143358 I/O calls, msec/call = 0.04, calls/sec = 28556.81 > nuttcp-t: 0.0user 4.9sys 0:05real 99% 106i+1428d 602maxrss 0+5pf 16+46csw > > nuttcp-r: v6.1.2: socket > nuttcp-r: buflen=65536, nstream=1, port=5001 tcp > nuttcp-r: accept from 127.0.0.1 > nuttcp-r: send window size = 43008, receive window size = 143360 > nuttcp-r: 8959.8750 MB in 5.17 real seconds = 1773171.07 KB/sec = > 14525.8174 Mbps > nuttcp-r: 219708 I/O calls, msec/call = 0.02, calls/sec = 42461.43 > nuttcp-r: 0.0user 3.8sys 0:05real 76% 105i+1407d 614maxrss 1+17pf > 95059+22csw New results: FreeBSD 8.2-STABLE #1: Tue Dec 6 13:51:01 EET 2011 # nuttcp -t -T 5 -w 128 -v localhost nuttcp-t: v6.1.2: socket nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost nuttcp-t: time limit = 5.00 seconds nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.030 ms nuttcp-t: send window size = 143360, receive window size = 71680 nuttcp-t: 12748.0625 MB in 5.02 real seconds = 2599947.38 KB/sec = 21298.7689 Mbps nuttcp-t: host-retrans = 0 nuttcp-t: 203969 I/O calls, msec/call = 0.03, calls/sec = 40624.18 nuttcp-t: 0.0user 4.9sys 0:05real 99% 106i+1428d 620maxrss 0+2pf 1+82csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=65536, nstream=1, port=5001 tcp nuttcp-r: accept from 127.0.0.1 nuttcp-r: send window size = 43008, receive window size = 143360 nuttcp-r: 12748.0625 MB in 5.15 real seconds = 2536511.81 KB/sec = 20779.1048 Mbps nuttcp-r: 297000 I/O calls, msec/call = 0.02, calls/sec = 57709.75 nuttcp-r: 0.1user 4.0sys 0:05real 81% 109i+1469d 626maxrss 0+15pf 121136+34csw Noticeable improvement. > > blade 2: > > # nuttcp -t -T 5 -w 128 -v 10.2.101.12 > nuttcp-t: v6.1.2: socket > nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.12 > nuttcp-t: time limit = 5.00 seconds > nuttcp-t: connect to 10.2.101.12 with mss=1448, RTT=0.059 ms > nuttcp-t: send window size = 131768, receive window size = 66608 > nuttcp-t: 1340.6469 MB in 5.02 real seconds = 273449.90 KB/sec = > 2240.1016 Mbps > nuttcp-t: host-retrans = 171 > nuttcp-t: 21451 I/O calls, msec/call = 0.24, calls/sec = 4272.78 > nuttcp-t: 0.0user 1.9sys 0:05real 39% 120i+1610d 600maxrss 2+3pf > 75658+0csw > > nuttcp-r: v6.1.2: socket > nuttcp-r: buflen=65536, nstream=1, port=5001 tcp > nuttcp-r: accept from 10.2.101.11 > nuttcp-r: send window size = 33304, receive window size = 131768 > nuttcp-r: 1340.6469 MB in 5.17 real seconds = 265292.92 KB/sec = > 2173.2796 Mbps > nuttcp-r: 408764 I/O calls, msec/call = 0.01, calls/sec = 78992.15 > nuttcp-r: 0.0user 3.3sys 0:05real 64% 105i+1413d 620maxrss 0+15pf > 105104+102csw # nuttcp -t -T 5 -w 128 -v 10.2.101.11 nuttcp-t: v6.1.2: socket nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.11 nuttcp-t: time limit = 5.00 seconds nuttcp-t: connect to 10.2.101.11 with mss=1448, RTT=0.055 ms nuttcp-t: send window size = 131768, receive window size = 66608 nuttcp-t: 1964.8640 MB in 5.02 real seconds = 400757.59 KB/sec = 3283.0062 Mbps nuttcp-t: host-retrans = 0 nuttcp-t: 31438 I/O calls, msec/call = 0.16, calls/sec = 6261.87 nuttcp-t: 0.0user 2.7sys 0:05real 55% 112i+1501d 1124maxrss 1+2pf 111165+112csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=65536, nstream=1, port=5001 tcp nuttcp-r: accept from 10.2.101.12 nuttcp-r: send window size = 33304, receive window size = 131768 nuttcp-r: 1964.8640 MB in 5.15 real seconds = 390972.20 KB/sec = 3202.8442 Mbps nuttcp-r: 560718 I/O calls, msec/call = 0.01, calls/sec = 108957.70 nuttcp-r: 0.1user 4.2sys 0:05real 84% 111i+1494d 626maxrss 0+15pf 151930+16csw Again, improvement. > > > Another pari of blades: > > FreeBSD 8.2-STABLE #0: Tue Aug 9 12:37:55 EEST 2011 > CPU: AMD Opteron(tm) Processor 6134 (2300.04-MHz K8-class CPU) > real memory = 68719476736 (65536 MB) > > first blade: > > # nuttcp -S > # nuttcp -t -T 5 -w 128 -v localhost > nuttcp-t: v6.1.2: socket > nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost > nuttcp-t: time limit = 5.00 seconds > nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.090 ms > nuttcp-t: send window size = 143360, receive window size = 71680 > nuttcp-t: 2695.0625 MB in 5.00 real seconds = 551756.90 KB/sec = > 4519.9925 Mbps > nuttcp-t: host-retrans = 0 > nuttcp-t: 43121 I/O calls, msec/call = 0.12, calls/sec = 8621.20 > nuttcp-t: 0.0user 4.9sys 0:05real 99% 106i+1428d 620maxrss 0+4pf 2+71csw > > nuttcp-r: v6.1.2: socket > nuttcp-r: buflen=65536, nstream=1, port=5001 tcp > nuttcp-r: accept from 127.0.0.1 > nuttcp-r: send window size = 43008, receive window size = 143360 > nuttcp-r: 2695.0625 MB in 5.14 real seconds = 536509.66 KB/sec = > 4395.0871 Mbps > nuttcp-r: 43126 I/O calls, msec/call = 0.12, calls/sec = 8383.94 > nuttcp-r: 0.0user 3.1sys 0:05real 61% 94i+1264d 624maxrss 1+16pf > 43019+0csw New result: FreeBSD 8.2-STABLE #1: Tue Dec 6 15:06:22 EET 2011 # nuttcp -t -T 5 -w 128 -v localhost nuttcp-t: v6.1.2: socket nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost nuttcp-t: time limit = 5.00 seconds nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.094 ms nuttcp-t: send window size = 143360, receive window size = 71680 nuttcp-t: 2659.1250 MB in 5.00 real seconds = 544493.51 KB/sec = 4460.4909 Mbps nuttcp-t: host-retrans = 0 nuttcp-t: 42546 I/O calls, msec/call = 0.12, calls/sec = 8507.71 nuttcp-t: 0.0user 4.9sys 0:05real 99% 106i+1428d 614maxrss 0+3pf 1+71csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=65536, nstream=1, port=5001 tcp nuttcp-r: accept from 127.0.0.1 nuttcp-r: send window size = 43008, receive window size = 143360 nuttcp-r: 2659.1250 MB in 5.14 real seconds = 529484.10 KB/sec = 4337.5338 Mbps nuttcp-r: 42560 I/O calls, msec/call = 0.12, calls/sec = 8275.91 nuttcp-r: 0.0user 3.1sys 0:05real 62% 103i+1391d 618maxrss 0+16pf 42454+0csw Same result.. something is wrong with this Opteron system. > > second blade: > > # nuttcp -t -T 5 -w 128 -v 10.2.101.13 > nuttcp-t: v6.1.2: socket > nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.13 > nuttcp-t: time limit = 5.00 seconds > nuttcp-t: connect to 10.2.101.13 with mss=1448, RTT=0.164 ms > nuttcp-t: send window size = 131768, receive window size = 66608 > nuttcp-t: 1290.3750 MB in 5.00 real seconds = 264173.96 KB/sec = > 2164.1131 Mbps > nuttcp-t: host-retrans = 0 > nuttcp-t: 20646 I/O calls, msec/call = 0.25, calls/sec = 4127.72 > nuttcp-t: 0.0user 3.8sys 0:05real 77% 96i+1299d 616maxrss 0+3pf > 27389+0csw > > nuttcp-r: v6.1.2: socket > nuttcp-r: buflen=65536, nstream=1, port=5001 tcp > nuttcp-r: accept from 10.2.101.14 > nuttcp-r: send window size = 33304, receive window size = 131768 > nuttcp-r: 1290.3750 MB in 5.14 real seconds = 256835.92 KB/sec = > 2103.9998 Mbps > nuttcp-r: 85668 I/O calls, msec/call = 0.06, calls/sec = 16651.70 > nuttcp-r: 0.0user 4.8sys 0:05real 94% 107i+1437d 624maxrss 0+15pf > 11848+0csw > # nuttcp -t -T 5 -w 128 -v 10.2.101.13 nuttcp-t: v6.1.2: socket nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.13 nuttcp-t: time limit = 5.00 seconds nuttcp-t: connect to 10.2.101.13 with mss=1448, RTT=0.130 ms nuttcp-t: send window size = 131768, receive window size = 66608 nuttcp-t: 1317.6875 MB in 5.00 real seconds = 269767.01 KB/sec = 2209.9313 Mbps nuttcp-t: host-retrans = 0 nuttcp-t: 21083 I/O calls, msec/call = 0.24, calls/sec = 4215.11 nuttcp-t: 0.0user 3.7sys 0:05real 75% 105i+1409d 616maxrss 0+3pf 39629+11csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=65536, nstream=1, port=5001 tcp nuttcp-r: accept from 10.2.101.14 nuttcp-r: send window size = 33304, receive window size = 131768 nuttcp-r: 1317.6875 MB in 5.14 real seconds = 262306.61 KB/sec = 2148.8157 Mbps nuttcp-r: 85980 I/O calls, msec/call = 0.06, calls/sec = 16714.53 nuttcp-r: 0.0user 4.8sys 0:05real 94% 107i+1446d 618maxrss 0+15pf 10783+0csw Same,but.. again interrupt storm: interrupt storm detected on "irq269:"; throttling interrupt source Also interesting, Intel (server) - AMD (client) # nuttcp -t -T 5 -w 128 -v 10.2.101.11 nuttcp-t: v6.1.2: socket nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.11 nuttcp-t: time limit = 5.00 seconds nuttcp-t: connect to 10.2.101.11 with mss=1448, RTT=0.087 ms nuttcp-t: send window size = 131768, receive window size = 66608 nuttcp-t: 1301.1125 MB in 5.00 real seconds = 266388.35 KB/sec = 2182.2534 Mbps nuttcp-t: host-retrans = 1 nuttcp-t: 20818 I/O calls, msec/call = 0.25, calls/sec = 4162.36 nuttcp-t: 0.0user 4.3sys 0:05real 87% 106i+1430d 612maxrss 0+3pf 88509+11csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=65536, nstream=1, port=5001 tcp nuttcp-r: accept from 10.2.101.13 nuttcp-r: send window size = 33304, receive window size = 131768 nuttcp-r: 1301.1125 MB in 5.14 real seconds = 259062.19 KB/sec = 2122.2374 Mbps nuttcp-r: 537176 I/O calls, msec/call = 0.01, calls/sec = 104449.37 nuttcp-r: 0.0user 3.2sys 0:05real 64% 122i+1643d 626maxrss 0+15pf 222593+0csw AMD (server) - Intel (client) # nuttcp -t -T 5 -w 128 -v 10.2.101.14 nuttcp-t: v6.1.2: socket nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.14 nuttcp-t: time limit = 5.00 seconds nuttcp-t: connect to 10.2.101.14 with mss=1448, RTT=0.099 ms nuttcp-t: send window size = 131768, receive window size = 66608 nuttcp-t: 750.8897 MB in 5.02 real seconds = 153161.09 KB/sec = 1254.6957 Mbps nuttcp-t: host-retrans = 25 nuttcp-t: 12015 I/O calls, msec/call = 0.43, calls/sec = 2393.29 nuttcp-t: 0.0user 0.9sys 0:05real 19% 112i+1507d 618maxrss 0+2pf 97668+4csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=65536, nstream=1, port=5001 tcp nuttcp-r: accept from 10.2.101.11 nuttcp-r: send window size = 33304, receive window size = 131768 nuttcp-r: 750.8897 MB in 5.15 real seconds = 149424.38 KB/sec = 1224.0845 Mbps nuttcp-r: 213009 I/O calls, msec/call = 0.02, calls/sec = 41394.56 nuttcp-r: 0.0user 4.6sys 0:05real 91% 107i+1445d 618maxrss 0+16pf 18273+0csw It seems performance measurements are more dependent on the server (nuttcp -S) machine. We will have to rule out the interrupt storms first of course, any advice? Daniel