Date: Wed, 7 Dec 2011 21:58:31 +0200 From: Daniel Kalchev <daniel@digsys.bg> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: Andre Oppermann <andre@freebsd.org>, Jack Vogel <jfvogel@gmail.com>, current@freebsd.org Subject: Re: quick summary results with ixgbe (was Re: datapoints on 10G throughput with TCP ? Message-ID: <A4BC7995-4F11-41BD-8583-3731694B404D@digsys.bg> In-Reply-To: <20111207180807.GA71878@onelab2.iet.unipi.it> References: <20111205192703.GA49118@onelab2.iet.unipi.it> <2D87D847-A2B7-4E77-B6C1-61D73C9F582F@digsys.bg> <20111205222834.GA50285@onelab2.iet.unipi.it> <4EDDF9F4.9070508@digsys.bg> <4EDE259B.4010502@digsys.bg> <CAFOYbcmVR_K0iZU_Z4TxDVzPzx6-GZuzfCxUZbf6KQn4siF2UA@mail.gmail.com> <F5BCA7E9-6A61-4492-9F18-423178E9C9B4@digsys.bg> <20111206210625.GB62605@onelab2.iet.unipi.it> <4EDF471F.1030202@freebsd.org> <20111207180807.GA71878@onelab2.iet.unipi.it>
next in thread | previous in thread | raw e-mail | index | archive | help
On Dec 7, 2011, at 8:08 PM, Luigi Rizzo wrote: > Summary: >=20 > - with default interrupt mitigation, the fastest configuration > is with checksums enabled on both sender and receiver, lro > enabled on the receiver. This gets about 8.0 Gbit/s I do not observe this. With defaults: # nuttcp -t -T 5 -w 128 -v 10.2.101.11 nuttcp-t: v6.1.2: socket nuttcp-t: buflen=3D65536, nstream=3D1, port=3D5001 tcp -> 10.2.101.11 nuttcp-t: time limit =3D 5.00 seconds nuttcp-t: connect to 10.2.101.11 with mss=3D1448, RTT=3D0.053 ms nuttcp-t: send window size =3D 131768, receive window size =3D 66608 nuttcp-t: 1857.4978 MB in 5.02 real seconds =3D 378856.02 KB/sec =3D = 3103.5885 Mbps nuttcp-t: host-retrans =3D 0 nuttcp-t: 29720 I/O calls, msec/call =3D 0.17, calls/sec =3D 5919.63 nuttcp-t: 0.0user 2.5sys 0:05real 52% 115i+1544d 630maxrss 0+2pf = 107264+1csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=3D65536, nstream=3D1, port=3D5001 tcp nuttcp-r: accept from 10.2.101.12 nuttcp-r: send window size =3D 33304, receive window size =3D 131768 nuttcp-r: 1857.4978 MB in 5.15 real seconds =3D 369617.39 KB/sec =3D = 3027.9057 Mbps nuttcp-r: 543991 I/O calls, msec/call =3D 0.01, calls/sec =3D 105709.95 nuttcp-r: 0.1user 4.1sys 0:05real 83% 110i+1482d 618maxrss 0+15pf = 158432+0csw On receiver: ifconfig ix1 lro # nuttcp -t -T 5 -w 128 -v 10.2.101.11 nuttcp-t: v6.1.2: socket nuttcp-t: buflen=3D65536, nstream=3D1, port=3D5001 tcp -> 10.2.101.11 nuttcp-t: time limit =3D 5.00 seconds nuttcp-t: connect to 10.2.101.11 with mss=3D1448, RTT=3D0.068 ms nuttcp-t: send window size =3D 131768, receive window size =3D 66608 nuttcp-t: 1673.3125 MB in 5.02 real seconds =3D 341312.36 KB/sec =3D = 2796.0308 Mbps nuttcp-t: host-retrans =3D 1 nuttcp-t: 26773 I/O calls, msec/call =3D 0.19, calls/sec =3D 5333.01 nuttcp-t: 0.0user 1.0sys 0:05real 21% 113i+1518d 630maxrss 0+2pf = 12772+1csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=3D65536, nstream=3D1, port=3D5001 tcp nuttcp-r: accept from 10.2.101.12 nuttcp-r: send window size =3D 33304, receive window size =3D 131768 nuttcp-r: 1673.3125 MB in 5.15 real seconds =3D 332975.19 KB/sec =3D = 2727.7327 Mbps nuttcp-r: 106268 I/O calls, msec/call =3D 0.05, calls/sec =3D 20650.82 nuttcp-r: 0.0user 1.3sys 0:05real 28% 101i+1354d 618maxrss 0+15pf = 64567+0csw On sender: ifconfig ix1 lro (now both receiver and sender have LRO enabled) # nuttcp -t -T 5 -w 128 -v 10.2.101.11 nuttcp-t: v6.1.2: socket nuttcp-t: buflen=3D65536, nstream=3D1, port=3D5001 tcp -> 10.2.101.11 nuttcp-t: time limit =3D 5.00 seconds nuttcp-t: connect to 10.2.101.11 with mss=3D1448, RTT=3D0.063 ms nuttcp-t: send window size =3D 131768, receive window size =3D 66608 nuttcp-t: 1611.7805 MB in 5.02 real seconds =3D 328716.18 KB/sec =3D = 2692.8430 Mbps nuttcp-t: host-retrans =3D 1 nuttcp-t: 25789 I/O calls, msec/call =3D 0.20, calls/sec =3D 5136.29 nuttcp-t: 0.0user 1.0sys 0:05real 21% 109i+1465d 630maxrss 0+2pf = 12697+0csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=3D65536, nstream=3D1, port=3D5001 tcp nuttcp-r: accept from 10.2.101.12 nuttcp-r: send window size =3D 33304, receive window size =3D 131768 nuttcp-r: 1611.7805 MB in 5.15 real seconds =3D 320694.82 KB/sec =3D = 2627.1319 Mbps nuttcp-r: 104346 I/O calls, msec/call =3D 0.05, calls/sec =3D 20275.05 nuttcp-r: 0.0user 1.3sys 0:05real 27% 113i+1516d 618maxrss 0+15pf = 63510+0csw remove LRO from receiver (only sender has LRO): # nuttcp -t -T 5 -w 128 -v 10.2.101.11 nuttcp-t: v6.1.2: socket nuttcp-t: buflen=3D65536, nstream=3D1, port=3D5001 tcp -> 10.2.101.11 nuttcp-t: time limit =3D 5.00 seconds nuttcp-t: connect to 10.2.101.11 with mss=3D1448, RTT=3D0.065 ms nuttcp-t: send window size =3D 131768, receive window size =3D 66608 nuttcp-t: 1884.8702 MB in 5.02 real seconds =3D 384464.57 KB/sec =3D = 3149.5338 Mbps nuttcp-t: host-retrans =3D 0 nuttcp-t: 30158 I/O calls, msec/call =3D 0.17, calls/sec =3D 6007.27 nuttcp-t: 0.0user 2.7sys 0:05real 55% 104i+1403d 630maxrss 0+2pf = 106046+0csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=3D65536, nstream=3D1, port=3D5001 tcp nuttcp-r: accept from 10.2.101.12 nuttcp-r: send window size =3D 33304, receive window size =3D 131768 nuttcp-r: 1884.8702 MB in 5.15 real seconds =3D 375093.52 KB/sec =3D = 3072.7661 Mbps nuttcp-r: 540237 I/O calls, msec/call =3D 0.01, calls/sec =3D 104988.68 nuttcp-r: 0.1user 4.2sys 0:05real 84% 110i+1483d 618maxrss 0+15pf = 156340+0csw Strange enough, setting hw.ixgbe.max_interrupt_rate=3D0 does not have = any effect.. Daniel
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A4BC7995-4F11-41BD-8583-3731694B404D>