From owner-freebsd-current@FreeBSD.ORG Wed Dec 7 19:58:49 2011 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C0764106564A for ; Wed, 7 Dec 2011 19:58:49 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 2E6308FC15 for ; Wed, 7 Dec 2011 19:58:48 +0000 (UTC) Received: from [192.92.129.101] ([192.92.129.101]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id pB7JwVts057027 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 7 Dec 2011 21:58:39 +0200 (EET) (envelope-from daniel@digsys.bg) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=us-ascii From: Daniel Kalchev In-Reply-To: <20111207180807.GA71878@onelab2.iet.unipi.it> Date: Wed, 7 Dec 2011 21:58:31 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20111205192703.GA49118@onelab2.iet.unipi.it> <2D87D847-A2B7-4E77-B6C1-61D73C9F582F@digsys.bg> <20111205222834.GA50285@onelab2.iet.unipi.it> <4EDDF9F4.9070508@digsys.bg> <4EDE259B.4010502@digsys.bg> <20111206210625.GB62605@onelab2.iet.unipi.it> <4EDF471F.1030202@freebsd.org> <20111207180807.GA71878@onelab2.iet.unipi.it> To: Luigi Rizzo X-Mailer: Apple Mail (2.1251.1) Cc: Andre Oppermann , Jack Vogel , current@freebsd.org Subject: Re: quick summary results with ixgbe (was Re: datapoints on 10G throughput with TCP ? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Dec 2011 19:58:49 -0000 On Dec 7, 2011, at 8:08 PM, Luigi Rizzo wrote: > Summary: >=20 > - with default interrupt mitigation, the fastest configuration > is with checksums enabled on both sender and receiver, lro > enabled on the receiver. This gets about 8.0 Gbit/s I do not observe this. With defaults: # nuttcp -t -T 5 -w 128 -v 10.2.101.11 nuttcp-t: v6.1.2: socket nuttcp-t: buflen=3D65536, nstream=3D1, port=3D5001 tcp -> 10.2.101.11 nuttcp-t: time limit =3D 5.00 seconds nuttcp-t: connect to 10.2.101.11 with mss=3D1448, RTT=3D0.053 ms nuttcp-t: send window size =3D 131768, receive window size =3D 66608 nuttcp-t: 1857.4978 MB in 5.02 real seconds =3D 378856.02 KB/sec =3D = 3103.5885 Mbps nuttcp-t: host-retrans =3D 0 nuttcp-t: 29720 I/O calls, msec/call =3D 0.17, calls/sec =3D 5919.63 nuttcp-t: 0.0user 2.5sys 0:05real 52% 115i+1544d 630maxrss 0+2pf = 107264+1csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=3D65536, nstream=3D1, port=3D5001 tcp nuttcp-r: accept from 10.2.101.12 nuttcp-r: send window size =3D 33304, receive window size =3D 131768 nuttcp-r: 1857.4978 MB in 5.15 real seconds =3D 369617.39 KB/sec =3D = 3027.9057 Mbps nuttcp-r: 543991 I/O calls, msec/call =3D 0.01, calls/sec =3D 105709.95 nuttcp-r: 0.1user 4.1sys 0:05real 83% 110i+1482d 618maxrss 0+15pf = 158432+0csw On receiver: ifconfig ix1 lro # nuttcp -t -T 5 -w 128 -v 10.2.101.11 nuttcp-t: v6.1.2: socket nuttcp-t: buflen=3D65536, nstream=3D1, port=3D5001 tcp -> 10.2.101.11 nuttcp-t: time limit =3D 5.00 seconds nuttcp-t: connect to 10.2.101.11 with mss=3D1448, RTT=3D0.068 ms nuttcp-t: send window size =3D 131768, receive window size =3D 66608 nuttcp-t: 1673.3125 MB in 5.02 real seconds =3D 341312.36 KB/sec =3D = 2796.0308 Mbps nuttcp-t: host-retrans =3D 1 nuttcp-t: 26773 I/O calls, msec/call =3D 0.19, calls/sec =3D 5333.01 nuttcp-t: 0.0user 1.0sys 0:05real 21% 113i+1518d 630maxrss 0+2pf = 12772+1csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=3D65536, nstream=3D1, port=3D5001 tcp nuttcp-r: accept from 10.2.101.12 nuttcp-r: send window size =3D 33304, receive window size =3D 131768 nuttcp-r: 1673.3125 MB in 5.15 real seconds =3D 332975.19 KB/sec =3D = 2727.7327 Mbps nuttcp-r: 106268 I/O calls, msec/call =3D 0.05, calls/sec =3D 20650.82 nuttcp-r: 0.0user 1.3sys 0:05real 28% 101i+1354d 618maxrss 0+15pf = 64567+0csw On sender: ifconfig ix1 lro (now both receiver and sender have LRO enabled) # nuttcp -t -T 5 -w 128 -v 10.2.101.11 nuttcp-t: v6.1.2: socket nuttcp-t: buflen=3D65536, nstream=3D1, port=3D5001 tcp -> 10.2.101.11 nuttcp-t: time limit =3D 5.00 seconds nuttcp-t: connect to 10.2.101.11 with mss=3D1448, RTT=3D0.063 ms nuttcp-t: send window size =3D 131768, receive window size =3D 66608 nuttcp-t: 1611.7805 MB in 5.02 real seconds =3D 328716.18 KB/sec =3D = 2692.8430 Mbps nuttcp-t: host-retrans =3D 1 nuttcp-t: 25789 I/O calls, msec/call =3D 0.20, calls/sec =3D 5136.29 nuttcp-t: 0.0user 1.0sys 0:05real 21% 109i+1465d 630maxrss 0+2pf = 12697+0csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=3D65536, nstream=3D1, port=3D5001 tcp nuttcp-r: accept from 10.2.101.12 nuttcp-r: send window size =3D 33304, receive window size =3D 131768 nuttcp-r: 1611.7805 MB in 5.15 real seconds =3D 320694.82 KB/sec =3D = 2627.1319 Mbps nuttcp-r: 104346 I/O calls, msec/call =3D 0.05, calls/sec =3D 20275.05 nuttcp-r: 0.0user 1.3sys 0:05real 27% 113i+1516d 618maxrss 0+15pf = 63510+0csw remove LRO from receiver (only sender has LRO): # nuttcp -t -T 5 -w 128 -v 10.2.101.11 nuttcp-t: v6.1.2: socket nuttcp-t: buflen=3D65536, nstream=3D1, port=3D5001 tcp -> 10.2.101.11 nuttcp-t: time limit =3D 5.00 seconds nuttcp-t: connect to 10.2.101.11 with mss=3D1448, RTT=3D0.065 ms nuttcp-t: send window size =3D 131768, receive window size =3D 66608 nuttcp-t: 1884.8702 MB in 5.02 real seconds =3D 384464.57 KB/sec =3D = 3149.5338 Mbps nuttcp-t: host-retrans =3D 0 nuttcp-t: 30158 I/O calls, msec/call =3D 0.17, calls/sec =3D 6007.27 nuttcp-t: 0.0user 2.7sys 0:05real 55% 104i+1403d 630maxrss 0+2pf = 106046+0csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=3D65536, nstream=3D1, port=3D5001 tcp nuttcp-r: accept from 10.2.101.12 nuttcp-r: send window size =3D 33304, receive window size =3D 131768 nuttcp-r: 1884.8702 MB in 5.15 real seconds =3D 375093.52 KB/sec =3D = 3072.7661 Mbps nuttcp-r: 540237 I/O calls, msec/call =3D 0.01, calls/sec =3D 104988.68 nuttcp-r: 0.1user 4.2sys 0:05real 84% 110i+1483d 618maxrss 0+15pf = 156340+0csw Strange enough, setting hw.ixgbe.max_interrupt_rate=3D0 does not have = any effect.. Daniel