Date: Sun, 10 Apr 2016 20:44:22 +0000 From: "Cui, Cheng" <Cheng.Cui@netapp.com> To: Hans Petter Selasky <hps@selasky.org> Cc: "svn-src-head@freebsd.org" <svn-src-head@freebsd.org> Subject: Re: question about trimning data "len" conditions in TSO in tcp_output.c Message-ID: <D3300112.FE63%Cheng.Cui@netapp.com> In-Reply-To: <D26BC410.B4AD%Cheng.Cui@netapp.com> References: <C44A2900-40E0-41EF-83B1-6DD4B31DABD5@netapp.com> <563D1892.3050406@selasky.org> <E6DBCD15-A41C-49A4-88C5-FB79EC969716@netapp.com> <563D2C26.2070300@selasky.org> <D26BC410.B4AD%Cheng.Cui@netapp.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--_006_D3300112FE63ChengCuinetappcom_
Content-Type: text/plain; charset="iso-8859-1"
Content-ID: <F8ED6A7E76AB6C44911474A1D22D688D@hq.netapp.com>
Content-Transfer-Encoding: quoted-printable
Hi Hans,
I would continue this discussion with a different change. The piece of
change is
here and also I attached the patch "change.patch" against the FreeBSD HEAD
code-line.
diff --git a/sys/netinet/tcp_output.c b/sys/netinet/tcp_output.c
index 2043fc9..fa124f1 100644
--- a/sys/netinet/tcp_output.c
+++ b/sys/netinet/tcp_output.c
@@ -938,25 +938,16 @@ send:
* fractional unless the send sockbuf can be
* emptied:
*/
- max_len =3D (tp->t_maxseg - optlen);
- if ((off + len) < sbavail(&so->so_snd)) {
+ max_len =3D (tp->t_maxopd - optlen);
+ if (len > (max_len << 1)) {
moff =3D len % max_len;
if (moff !=3D 0) {
len -=3D moff;
sendalot =3D 1;
}
}
-
- /*
- * In case there are too many small fragments
- * don't use TSO:
- */
- if (len <=3D max_len) {
- len =3D max_len;
- sendalot =3D 1;
- tso =3D 0;
- }
-
+ KASSERT(len > max_len,
+ ("[%s:%d]: len <=3D max_len", __func__,
__LINE__));
/*
* Send the FIN in a separate segment
* after the bulk sending is done.
I think this change could save additional loops that send single MSS-size
packets. So I think some CPU cycles can be saved as well, due to this
change=20
reduced software sends and pushed more data to offloading sends.
Here is my test. The iperf command I choose pushes 100Mbytes data to the
wire by setting the default TCP sendspace to 1MB and recvspace to 2MB. I
tested this TCP connection performance on a pair of 10Gbps FreeBSD 10.2
nodes=20
(s1 and r1) with a switch in between. Both nodes have TSO and delayed ACK
enabled.=20
root@s1:~ # ping -c 3 r1
PING r1-link1 (10.1.2.3): 56 data bytes
64 bytes from 10.1.2.3: icmp_seq=3D0 ttl=3D64 time=3D0.045 ms
64 bytes from 10.1.2.3: icmp_seq=3D1 ttl=3D64 time=3D0.037 ms
64 bytes from 10.1.2.3: icmp_seq=3D2 ttl=3D64 time=3D0.038 ms
--- r1-link1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev =3D 0.037/0.040/0.045/0.004 ms
1M snd buffer/2M rcv buffer
sysctl -w net.inet.tcp.hostcache.expire=3D1
sysctl -w net.inet.tcp.sendspace=3D1048576
sysctl -w net.inet.tcp.recvspace=3D2097152
iperf -s <=3D=3D iperf command@receiver
iperf -c r1 -m -n 100M <=3D=3D iperf command@sender
root@s1:~ # iperf -c r1 -m -n 100M
------------------------------------------------------------
Client connecting to r1, TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[ 3] local 10.1.2.2 port 22491 connected with 10.1.2.3 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 0.3 sec 100 MBytes 2.69 Gbits/sec
[ 3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
root@r1:~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 2.00 MByte (default)
------------------------------------------------------------
[ 4] local 10.1.2.3 port 5001 connected with 10.1.2.2 port 22491
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 0.3 sec 100 MBytes 2.62 Gbits/sec
Each test sent 100MBytes of data, and I collected the packet trace from
both=20
nodes by tcpdump. I did this test twice to confirm the result can be
reproduced.
>From the trace files of both nodes before my code change, I see a lot of
single-MSS size packets. See the attached trace files in
"before_change.zip".
For example, in a sender trace file I see 43480 single-MSS size
packets(tcp.len=3D=3D1448) out of 57005 packets that contain data(tcp.len >
0).=20
That's 76.2%.
And I did the same iperf test and gathered trace files. I did not find
many single-MSS packets this time. See the attached trace files in
"after_change.zip". For example, in a sender trace file I see zero
single-MSS=20
size packets(tcp.len=3D=3D1448) out of 35729 data packets(tcp.len > 0).
Compared with the receiver traces, I did not see significant more
fractional=20
packets received after change.
I also did tests using netperf, although I did not get enough 95%
confidence for
every test on snd/rcv buffer size. Attached are my netperf result on
different
snd/rcv buffer size before and after the change (netperf_before_change.txt
and=20
netperf_after_change.txt), which also look good.
used netperf command:
netperf -H s1 -t TCP_STREAM -C -c -l 400 -i 10,3 -I 95,10 -- -s
${LocalSndBuf} -S ${RemoteSndBuf}
Thanks,
--Cheng Cui
NetApp Scale Out Networking
--_006_D3300112FE63ChengCuinetappcom_
Content-Type: application/octet-stream; name="change.patch"
Content-Description: change.patch
Content-Disposition: attachment; filename="change.patch"; size=864;
creation-date="Sun, 10 Apr 2016 20:44:22 GMT";
modification-date="Sun, 10 Apr 2016 20:44:22 GMT"
Content-ID: <951E33D4E6C172408F958B2B6FB247EA@hq.netapp.com>
Content-Transfer-Encoding: base64
ZGlmZiAtLWdpdCBhL3N5cy9uZXRpbmV0L3RjcF9vdXRwdXQuYyBiL3N5cy9uZXRpbmV0L3RjcF9v
dXRwdXQuYwppbmRleCAyMDQzZmM5Li5mYTEyNGYxIDEwMDY0NAotLS0gYS9zeXMvbmV0aW5ldC90
Y3Bfb3V0cHV0LmMKKysrIGIvc3lzL25ldGluZXQvdGNwX291dHB1dC5jCkBAIC05MzgsMjUgKzkz
OCwxNiBAQCBzZW5kOgogCQkJICogZnJhY3Rpb25hbCB1bmxlc3MgdGhlIHNlbmQgc29ja2J1ZiBj
YW4gYmUKIAkJCSAqIGVtcHRpZWQ6CiAJCQkgKi8KLQkJCW1heF9sZW4gPSAodHAtPnRfbWF4c2Vn
IC0gb3B0bGVuKTsKLQkJCWlmICgob2ZmICsgbGVuKSA8IHNiYXZhaWwoJnNvLT5zb19zbmQpKSB7
CisJCQltYXhfbGVuID0gKHRwLT50X21heG9wZCAtIG9wdGxlbik7CisJCQlpZiAobGVuID4gKG1h
eF9sZW4gPDwgMSkpIHsKIAkJCQltb2ZmID0gbGVuICUgbWF4X2xlbjsKIAkJCQlpZiAobW9mZiAh
PSAwKSB7CiAJCQkJCWxlbiAtPSBtb2ZmOwogCQkJCQlzZW5kYWxvdCA9IDE7CiAJCQkJfQogCQkJ
fQotCi0JCQkvKgotCQkJICogSW4gY2FzZSB0aGVyZSBhcmUgdG9vIG1hbnkgc21hbGwgZnJhZ21l
bnRzCi0JCQkgKiBkb24ndCB1c2UgVFNPOgotCQkJICovCi0JCQlpZiAobGVuIDw9IG1heF9sZW4p
IHsKLQkJCQlsZW4gPSBtYXhfbGVuOwotCQkJCXNlbmRhbG90ID0gMTsKLQkJCQl0c28gPSAwOwot
CQkJfQotCisJCQlLQVNTRVJUKGxlbiA+IG1heF9sZW4sCisJCQkgICAgKCJbJXM6JWRdOiBsZW4g
PD0gbWF4X2xlbiIsIF9fZnVuY19fLCBfX0xJTkVfXykpOwogCQkJLyoKIAkJCSAqIFNlbmQgdGhl
IEZJTiBpbiBhIHNlcGFyYXRlIHNlZ21lbnQKIAkJCSAqIGFmdGVyIHRoZSBidWxrIHNlbmRpbmcg
aXMgZG9uZS4K
--_006_D3300112FE63ChengCuinetappcom_
Content-Type: text/plain; name="netperf_before_change.txt"
Content-Description: netperf_before_change.txt
Content-Disposition: attachment; filename="netperf_before_change.txt";
size=6702; creation-date="Sun, 10 Apr 2016 20:44:22 GMT";
modification-date="Sun, 10 Apr 2016 20:44:22 GMT"
Content-ID: <1887B0ABE050584E9A69FB7D4D43EBD8@hq.netapp.com>
Content-Transfer-Encoding: base64
VGh1IEFwciAgNyAxNDo0MjoyMSBNRFQgMjAxNgpNSUdSQVRFRCBUQ1AgU1RSRUFNIFRFU1QgZnJv
bSAwLjAuMC4wICgwLjAuMC4wKSBwb3J0IDAgQUZfSU5FVCB0byByMS1saW5rMSAoKSBwb3J0IDAg
QUZfSU5FVCA6ICsvLTUuMDAwJSBAIDk1JSBjb25mLiAgOiBoaXN0b2dyYW0gOiBpbnRlcnZhbCA6
IGRpcnR5IGRhdGEgOiBkZW1vCiEhISBXQVJOSU5HCiEhISBEZXNpcmVkIGNvbmZpZGVuY2Ugd2Fz
IG5vdCBhY2hpZXZlZCB3aXRoaW4gdGhlIHNwZWNpZmllZCBpdGVyYXRpb25zLgohISEgVGhpcyBp
bXBsaWVzIHRoYXQgdGhlcmUgd2FzIHZhcmlhYmlsaXR5IGluIHRoZSB0ZXN0IGVudmlyb25tZW50
IHRoYXQKISEhIG11c3QgYmUgaW52ZXN0aWdhdGVkIGJlZm9yZSBnb2luZyBmdXJ0aGVyLgohISEg
Q29uZmlkZW5jZSBpbnRlcnZhbHM6IFRocm91Z2hwdXQgICAgICA6IDEwLjc4MyUKISEhICAgICAg
ICAgICAgICAgICAgICAgICBMb2NhbCBDUFUgdXRpbCAgOiA2LjI3NyUKISEhICAgICAgICAgICAg
ICAgICAgICAgICBSZW1vdGUgQ1BVIHV0aWwgOiAxLjE1MyUKClJlY3YgICBTZW5kICAgIFNlbmQg
ICAgICAgICAgICAgICAgICAgICAgICAgIFV0aWxpemF0aW9uICAgICAgIFNlcnZpY2UgRGVtYW5k
ClNvY2tldCBTb2NrZXQgIE1lc3NhZ2UgIEVsYXBzZWQgICAgICAgICAgICAgIFNlbmQgICAgIFJl
Y3YgICAgIFNlbmQgICAgUmVjdgpTaXplICAgU2l6ZSAgICBTaXplICAgICBUaW1lICAgICBUaHJv
dWdocHV0ICBsb2NhbCAgICByZW1vdGUgICBsb2NhbCAgIHJlbW90ZQpieXRlcyAgYnl0ZXMgICBi
eXRlcyAgICBzZWNzLiAgICAxMF42Yml0cy9zICAlIEMgICAgICAlIEMgICAgICB1cy9LQiAgIHVz
L0tCCgogNjU1MzYgIDMyNzY4ICAzMjc2OCAgICA0MDAuMDEgICAgIDQ2NzAuMzEgICA0Ljg3ICAg
ICA1LjQ4ICAgICAyLjc0NyAgIDMuMDkxICAKVGh1IEFwciAgNyAxNTo0OTowMiBNRFQgMjAxNgoK
CgpUaHUgQXByICA3IDE1OjQ5OjEyIE1EVCAyMDE2Ck1JR1JBVEVEIFRDUCBTVFJFQU0gVEVTVCBm
cm9tIDAuMC4wLjAgKDAuMC4wLjApIHBvcnQgMCBBRl9JTkVUIHRvIHIxLWxpbmsxICgpIHBvcnQg
MCBBRl9JTkVUIDogKy8tNS4wMDAlIEAgOTUlIGNvbmYuICA6IGhpc3RvZ3JhbSA6IGludGVydmFs
IDogZGlydHkgZGF0YSA6IGRlbW8KISEhIFdBUk5JTkcKISEhIERlc2lyZWQgY29uZmlkZW5jZSB3
YXMgbm90IGFjaGlldmVkIHdpdGhpbiB0aGUgc3BlY2lmaWVkIGl0ZXJhdGlvbnMuCiEhISBUaGlz
IGltcGxpZXMgdGhhdCB0aGVyZSB3YXMgdmFyaWFiaWxpdHkgaW4gdGhlIHRlc3QgZW52aXJvbm1l
bnQgdGhhdAohISEgbXVzdCBiZSBpbnZlc3RpZ2F0ZWQgYmVmb3JlIGdvaW5nIGZ1cnRoZXIuCiEh
ISBDb25maWRlbmNlIGludGVydmFsczogVGhyb3VnaHB1dCAgICAgIDogNy4zNDclCiEhISAgICAg
ICAgICAgICAgICAgICAgICAgTG9jYWwgQ1BVIHV0aWwgIDogMTEuNjU4JQohISEgICAgICAgICAg
ICAgICAgICAgICAgIFJlbW90ZSBDUFUgdXRpbCA6IDEuNTI0JQoKUmVjdiAgIFNlbmQgICAgU2Vu
ZCAgICAgICAgICAgICAgICAgICAgICAgICAgVXRpbGl6YXRpb24gICAgICAgU2VydmljZSBEZW1h
bmQKU29ja2V0IFNvY2tldCAgTWVzc2FnZSAgRWxhcHNlZCAgICAgICAgICAgICAgU2VuZCAgICAg
UmVjdiAgICAgU2VuZCAgICBSZWN2ClNpemUgICBTaXplICAgIFNpemUgICAgIFRpbWUgICAgIFRo
cm91Z2hwdXQgIGxvY2FsICAgIHJlbW90ZSAgIGxvY2FsICAgcmVtb3RlCmJ5dGVzICBieXRlcyAg
IGJ5dGVzICAgIHNlY3MuICAgIDEwXjZiaXRzL3MgICUgQyAgICAgICUgQyAgICAgIHVzL0tCICAg
dXMvS0IKCjEzMTA3MiAgNjU1MzYgIDY1NTM2ICAgIDQwMC4wMSAgICAgNDc0Mi40NSAgIDQuOTkg
ICAgIDUuNTMgICAgIDIuNzU5ICAgMy4wNjQgIApUaHUgQXByICA3IDE2OjU1OjUyIE1EVCAyMDE2
CgoKClRodSBBcHIgIDcgMTY6NTY6MDIgTURUIDIwMTYKTUlHUkFURUQgVENQIFNUUkVBTSBURVNU
IGZyb20gMC4wLjAuMCAoMC4wLjAuMCkgcG9ydCAwIEFGX0lORVQgdG8gcjEtbGluazEgKCkgcG9y
dCAwIEFGX0lORVQgOiArLy01LjAwMCUgQCA5NSUgY29uZi4gIDogaGlzdG9ncmFtIDogaW50ZXJ2
YWwgOiBkaXJ0eSBkYXRhIDogZGVtbwohISEgV0FSTklORwohISEgRGVzaXJlZCBjb25maWRlbmNl
IHdhcyBub3QgYWNoaWV2ZWQgd2l0aGluIHRoZSBzcGVjaWZpZWQgaXRlcmF0aW9ucy4KISEhIFRo
aXMgaW1wbGllcyB0aGF0IHRoZXJlIHdhcyB2YXJpYWJpbGl0eSBpbiB0aGUgdGVzdCBlbnZpcm9u
bWVudCB0aGF0CiEhISBtdXN0IGJlIGludmVzdGlnYXRlZCBiZWZvcmUgZ29pbmcgZnVydGhlci4K
ISEhIENvbmZpZGVuY2UgaW50ZXJ2YWxzOiBUaHJvdWdocHV0ICAgICAgOiAxMC4yMTIlCiEhISAg
ICAgICAgICAgICAgICAgICAgICAgTG9jYWwgQ1BVIHV0aWwgIDogMTIuODUwJQohISEgICAgICAg
ICAgICAgICAgICAgICAgIFJlbW90ZSBDUFUgdXRpbCA6IDAuODc0JQoKUmVjdiAgIFNlbmQgICAg
U2VuZCAgICAgICAgICAgICAgICAgICAgICAgICAgVXRpbGl6YXRpb24gICAgICAgU2VydmljZSBE
ZW1hbmQKU29ja2V0IFNvY2tldCAgTWVzc2FnZSAgRWxhcHNlZCAgICAgICAgICAgICAgU2VuZCAg
ICAgUmVjdiAgICAgU2VuZCAgICBSZWN2ClNpemUgICBTaXplICAgIFNpemUgICAgIFRpbWUgICAg
IFRocm91Z2hwdXQgIGxvY2FsICAgIHJlbW90ZSAgIGxvY2FsICAgcmVtb3RlCmJ5dGVzICBieXRl
cyAgIGJ5dGVzICAgIHNlY3MuICAgIDEwXjZiaXRzL3MgICUgQyAgICAgICUgQyAgICAgIHVzL0tC
ICAgdXMvS0IKCjI2MjE0NCAxMzEwNzIgMTMxMDcyICAgIDQwMC4wMiAgICAgNDg4MS40OSAgIDUu
NDIgICAgIDUuNTMgICAgIDIuOTE1ICAgMi45ODEgIApUaHUgQXByICA3IDE4OjAyOjQyIE1EVCAy
MDE2CgoKClRodSBBcHIgIDcgMTg6MDI6NTIgTURUIDIwMTYKTUlHUkFURUQgVENQIFNUUkVBTSBU
RVNUIGZyb20gMC4wLjAuMCAoMC4wLjAuMCkgcG9ydCAwIEFGX0lORVQgdG8gcjEtbGluazEgKCkg
cG9ydCAwIEFGX0lORVQgOiArLy01LjAwMCUgQCA5NSUgY29uZi4gIDogaGlzdG9ncmFtIDogaW50
ZXJ2YWwgOiBkaXJ0eSBkYXRhIDogZGVtbwohISEgV0FSTklORwohISEgRGVzaXJlZCBjb25maWRl
bmNlIHdhcyBub3QgYWNoaWV2ZWQgd2l0aGluIHRoZSBzcGVjaWZpZWQgaXRlcmF0aW9ucy4KISEh
IFRoaXMgaW1wbGllcyB0aGF0IHRoZXJlIHdhcyB2YXJpYWJpbGl0eSBpbiB0aGUgdGVzdCBlbnZp
cm9ubWVudCB0aGF0CiEhISBtdXN0IGJlIGludmVzdGlnYXRlZCBiZWZvcmUgZ29pbmcgZnVydGhl
ci4KISEhIENvbmZpZGVuY2UgaW50ZXJ2YWxzOiBUaHJvdWdocHV0ICAgICAgOiAzNi42ODYlCiEh
ISAgICAgICAgICAgICAgICAgICAgICAgTG9jYWwgQ1BVIHV0aWwgIDogMTIuNjQxJQohISEgICAg
ICAgICAgICAgICAgICAgICAgIFJlbW90ZSBDUFUgdXRpbCA6IDEyLjMyMiUKClJlY3YgICBTZW5k
ICAgIFNlbmQgICAgICAgICAgICAgICAgICAgICAgICAgIFV0aWxpemF0aW9uICAgICAgIFNlcnZp
Y2UgRGVtYW5kClNvY2tldCBTb2NrZXQgIE1lc3NhZ2UgIEVsYXBzZWQgICAgICAgICAgICAgIFNl
bmQgICAgIFJlY3YgICAgIFNlbmQgICAgUmVjdgpTaXplICAgU2l6ZSAgICBTaXplICAgICBUaW1l
ICAgICBUaHJvdWdocHV0ICBsb2NhbCAgICByZW1vdGUgICBsb2NhbCAgIHJlbW90ZQpieXRlcyAg
Ynl0ZXMgICBieXRlcyAgICBzZWNzLiAgICAxMF42Yml0cy9zICAlIEMgICAgICAlIEMgICAgICB1
cy9LQiAgIHVzL0tCCgo1MjQyODggMjYyMTQ0IDI2MjE0NCAgICA0MDAuMDIgICAgIDM3MzQuMjkg
ICA1LjAxICAgICA1LjAzICAgICAzLjY3OCAgIDMuNjcxICAKVGh1IEFwciAgNyAxOTowOTozMyBN
RFQgMjAxNgoKCgpUaHUgQXByICA3IDE5OjA5OjQzIE1EVCAyMDE2Ck1JR1JBVEVEIFRDUCBTVFJF
QU0gVEVTVCBmcm9tIDAuMC4wLjAgKDAuMC4wLjApIHBvcnQgMCBBRl9JTkVUIHRvIHIxLWxpbmsx
ICgpIHBvcnQgMCBBRl9JTkVUIDogKy8tNS4wMDAlIEAgOTUlIGNvbmYuICA6IGhpc3RvZ3JhbSA6
IGludGVydmFsIDogZGlydHkgZGF0YSA6IGRlbW8KUmVjdiAgIFNlbmQgICAgU2VuZCAgICAgICAg
ICAgICAgICAgICAgICAgICAgVXRpbGl6YXRpb24gICAgICAgU2VydmljZSBEZW1hbmQKU29ja2V0
IFNvY2tldCAgTWVzc2FnZSAgRWxhcHNlZCAgICAgICAgICAgICAgU2VuZCAgICAgUmVjdiAgICAg
U2VuZCAgICBSZWN2ClNpemUgICBTaXplICAgIFNpemUgICAgIFRpbWUgICAgIFRocm91Z2hwdXQg
IGxvY2FsICAgIHJlbW90ZSAgIGxvY2FsICAgcmVtb3RlCmJ5dGVzICBieXRlcyAgIGJ5dGVzICAg
IHNlY3MuICAgIDEwXjZiaXRzL3MgICUgQyAgICAgICUgQyAgICAgIHVzL0tCICAgdXMvS0IKCjEw
NDg1NzYgNTI0Mjg4IDUyNDI4OCAgICA0MDAuMDIgICAgIDI4OTEuMTAgICA0LjU4ICAgICA0LjY0
ICAgICA0LjE1NSAgIDQuMjEwICAKVGh1IEFwciAgNyAxOTo0MzowMyBNRFQgMjAxNgoKCgpUaHUg
QXByICA3IDE5OjQzOjEzIE1EVCAyMDE2Ck1JR1JBVEVEIFRDUCBTVFJFQU0gVEVTVCBmcm9tIDAu
MC4wLjAgKDAuMC4wLjApIHBvcnQgMCBBRl9JTkVUIHRvIHIxLWxpbmsxICgpIHBvcnQgMCBBRl9J
TkVUIDogKy8tNS4wMDAlIEAgOTUlIGNvbmYuICA6IGhpc3RvZ3JhbSA6IGludGVydmFsIDogZGly
dHkgZGF0YSA6IGRlbW8KISEhIFdBUk5JTkcKISEhIERlc2lyZWQgY29uZmlkZW5jZSB3YXMgbm90
IGFjaGlldmVkIHdpdGhpbiB0aGUgc3BlY2lmaWVkIGl0ZXJhdGlvbnMuCiEhISBUaGlzIGltcGxp
ZXMgdGhhdCB0aGVyZSB3YXMgdmFyaWFiaWxpdHkgaW4gdGhlIHRlc3QgZW52aXJvbm1lbnQgdGhh
dAohISEgbXVzdCBiZSBpbnZlc3RpZ2F0ZWQgYmVmb3JlIGdvaW5nIGZ1cnRoZXIuCiEhISBDb25m
aWRlbmNlIGludGVydmFsczogVGhyb3VnaHB1dCAgICAgIDogMTEuNjkyJQohISEgICAgICAgICAg
ICAgICAgICAgICAgIExvY2FsIENQVSB1dGlsICA6IDEwLjgwMCUKISEhICAgICAgICAgICAgICAg
ICAgICAgICBSZW1vdGUgQ1BVIHV0aWwgOiA3Ljc5MiUKClJlY3YgICBTZW5kICAgIFNlbmQgICAg
ICAgICAgICAgICAgICAgICAgICAgIFV0aWxpemF0aW9uICAgICAgIFNlcnZpY2UgRGVtYW5kClNv
Y2tldCBTb2NrZXQgIE1lc3NhZ2UgIEVsYXBzZWQgICAgICAgICAgICAgIFNlbmQgICAgIFJlY3Yg
ICAgIFNlbmQgICAgUmVjdgpTaXplICAgU2l6ZSAgICBTaXplICAgICBUaW1lICAgICBUaHJvdWdo
cHV0ICBsb2NhbCAgICByZW1vdGUgICBsb2NhbCAgIHJlbW90ZQpieXRlcyAgYnl0ZXMgICBieXRl
cyAgICBzZWNzLiAgICAxMF42Yml0cy9zICAlIEMgICAgICAlIEMgICAgICB1cy9LQiAgIHVzL0tC
CgoyMDk3MTUyIDEwNDg1NzYgMTA0ODU3NiAgICA0MDAuMDQgICAgIDI5ODQuNzcgICA0Ljc4ICAg
ICA0LjgwICAgICA0LjIwMSAgIDQuMjIxICAKVGh1IEFwciAgNyAyMDo0OTo1NCBNRFQgMjAxNgoK
CgpUaHUgQXByICA3IDIwOjUwOjA1IE1EVCAyMDE2Ck1JR1JBVEVEIFRDUCBTVFJFQU0gVEVTVCBm
cm9tIDAuMC4wLjAgKDAuMC4wLjApIHBvcnQgMCBBRl9JTkVUIHRvIHIxLWxpbmsxICgpIHBvcnQg
MCBBRl9JTkVUIDogKy8tNS4wMDAlIEAgOTUlIGNvbmYuICA6IGhpc3RvZ3JhbSA6IGludGVydmFs
IDogZGlydHkgZGF0YSA6IGRlbW8KUmVjdiAgIFNlbmQgICAgU2VuZCAgICAgICAgICAgICAgICAg
ICAgICAgICAgVXRpbGl6YXRpb24gICAgICAgU2VydmljZSBEZW1hbmQKU29ja2V0IFNvY2tldCAg
TWVzc2FnZSAgRWxhcHNlZCAgICAgICAgICAgICAgU2VuZCAgICAgUmVjdiAgICAgU2VuZCAgICBS
ZWN2ClNpemUgICBTaXplICAgIFNpemUgICAgIFRpbWUgICAgIFRocm91Z2hwdXQgIGxvY2FsICAg
IHJlbW90ZSAgIGxvY2FsICAgcmVtb3RlCmJ5dGVzICBieXRlcyAgIGJ5dGVzICAgIHNlY3MuICAg
IDEwXjZiaXRzL3MgICUgQyAgICAgICUgQyAgICAgIHVzL0tCICAgdXMvS0IKCjQxOTQzMDQgMjA5
NzE1MiAyMDk3MTUyICAgIDQwMC4zNCAgICAgMjkwOC45NyAgIDQuNjYgICAgIDQuNjggICAgIDQu
MTk2ICAgNC4yMTMgIApUaHUgQXByICA3IDIxOjE2OjQ3IE1EVCAyMDE2CgoKClRodSBBcHIgIDcg
MjE6MTY6NTcgTURUIDIwMTYKTUlHUkFURUQgVENQIFNUUkVBTSBURVNUIGZyb20gMC4wLjAuMCAo
MC4wLjAuMCkgcG9ydCAwIEFGX0lORVQgdG8gcjEtbGluazEgKCkgcG9ydCAwIEFGX0lORVQgOiAr
Ly01LjAwMCUgQCA5NSUgY29uZi4gIDogaGlzdG9ncmFtIDogaW50ZXJ2YWwgOiBkaXJ0eSBkYXRh
IDogZGVtbwpSZWN2ICAgU2VuZCAgICBTZW5kICAgICAgICAgICAgICAgICAgICAgICAgICBVdGls
aXphdGlvbiAgICAgICBTZXJ2aWNlIERlbWFuZApTb2NrZXQgU29ja2V0ICBNZXNzYWdlICBFbGFw
c2VkICAgICAgICAgICAgICBTZW5kICAgICBSZWN2ICAgICBTZW5kICAgIFJlY3YKU2l6ZSAgIFNp
emUgICAgU2l6ZSAgICAgVGltZSAgICAgVGhyb3VnaHB1dCAgbG9jYWwgICAgcmVtb3RlICAgbG9j
YWwgICByZW1vdGUKYnl0ZXMgIGJ5dGVzICAgYnl0ZXMgICAgc2Vjcy4gICAgMTBeNmJpdHMvcyAg
JSBDICAgICAgJSBDICAgICAgdXMvS0IgICB1cy9LQgoKODM4ODYwOCA0MTk0MzA0IDQxOTQzMDQg
ICAgNDAwLjI4ICAgICAyOTIyLjU0ICAgMy44MiAgICAgNC42OSAgICAgMy40MzEgICA0LjIwNSAg
ClRodSBBcHIgIDcgMjE6NTc6MDEgTURUIDIwMTYKCgoK
--_006_D3300112FE63ChengCuinetappcom_
Content-Type: text/plain; name="netperf_after_change.txt"
Content-Description: netperf_after_change.txt
Content-Disposition: attachment; filename="netperf_after_change.txt";
size=5982; creation-date="Sun, 10 Apr 2016 20:44:22 GMT";
modification-date="Sun, 10 Apr 2016 20:44:22 GMT"
Content-ID: <13F37D885E29F4478604178500D5FD97@hq.netapp.com>
Content-Transfer-Encoding: base64
U2F0IEFwciAgOSAwOTo1NjoyOCBNRFQgMjAxNgpNSUdSQVRFRCBUQ1AgU1RSRUFNIFRFU1QgZnJv
bSAwLjAuMC4wICgwLjAuMC4wKSBwb3J0IDAgQUZfSU5FVCB0byBzMS1saW5rMSAoKSBwb3J0IDAg
QUZfSU5FVCA6ICsvLTUuMDAwJSBAIDk1JSBjb25mLiAgOiBoaXN0b2dyYW0gOiBpbnRlcnZhbCA6
IGRpcnR5IGRhdGEgOiBkZW1vCiEhISBXQVJOSU5HCiEhISBEZXNpcmVkIGNvbmZpZGVuY2Ugd2Fz
IG5vdCBhY2hpZXZlZCB3aXRoaW4gdGhlIHNwZWNpZmllZCBpdGVyYXRpb25zLgohISEgVGhpcyBp
bXBsaWVzIHRoYXQgdGhlcmUgd2FzIHZhcmlhYmlsaXR5IGluIHRoZSB0ZXN0IGVudmlyb25tZW50
IHRoYXQKISEhIG11c3QgYmUgaW52ZXN0aWdhdGVkIGJlZm9yZSBnb2luZyBmdXJ0aGVyLgohISEg
Q29uZmlkZW5jZSBpbnRlcnZhbHM6IFRocm91Z2hwdXQgICAgICA6IDIxLjgyMCUKISEhICAgICAg
ICAgICAgICAgICAgICAgICBMb2NhbCBDUFUgdXRpbCAgOiAxMC4xNzQlCiEhISAgICAgICAgICAg
ICAgICAgICAgICAgUmVtb3RlIENQVSB1dGlsIDogMTQuMjUwJQoKUmVjdiAgIFNlbmQgICAgU2Vu
ZCAgICAgICAgICAgICAgICAgICAgICAgICAgVXRpbGl6YXRpb24gICAgICAgU2VydmljZSBEZW1h
bmQKU29ja2V0IFNvY2tldCAgTWVzc2FnZSAgRWxhcHNlZCAgICAgICAgICAgICAgU2VuZCAgICAg
UmVjdiAgICAgU2VuZCAgICBSZWN2ClNpemUgICBTaXplICAgIFNpemUgICAgIFRpbWUgICAgIFRo
cm91Z2hwdXQgIGxvY2FsICAgIHJlbW90ZSAgIGxvY2FsICAgcmVtb3RlCmJ5dGVzICBieXRlcyAg
IGJ5dGVzICAgIHNlY3MuICAgIDEwXjZiaXRzL3MgICUgQyAgICAgICUgQyAgICAgIHVzL0tCICAg
dXMvS0IKCiA2NTUzNiAgMzI3NjggIDMyNzY4ICAgIDQwMC4wMSAgICAgNDUzNy4yNCAgIDUuMTcg
ICAgIDUuMjggICAgIDMuMDM5ICAgMy4wODUgIApTYXQgQXByICA5IDExOjAzOjA4IE1EVCAyMDE2
CgoKClNhdCBBcHIgIDkgMTE6MDM6MTkgTURUIDIwMTYKTUlHUkFURUQgVENQIFNUUkVBTSBURVNU
IGZyb20gMC4wLjAuMCAoMC4wLjAuMCkgcG9ydCAwIEFGX0lORVQgdG8gczEtbGluazEgKCkgcG9y
dCAwIEFGX0lORVQgOiArLy01LjAwMCUgQCA5NSUgY29uZi4gIDogaGlzdG9ncmFtIDogaW50ZXJ2
YWwgOiBkaXJ0eSBkYXRhIDogZGVtbwpSZWN2ICAgU2VuZCAgICBTZW5kICAgICAgICAgICAgICAg
ICAgICAgICAgICBVdGlsaXphdGlvbiAgICAgICBTZXJ2aWNlIERlbWFuZApTb2NrZXQgU29ja2V0
ICBNZXNzYWdlICBFbGFwc2VkICAgICAgICAgICAgICBTZW5kICAgICBSZWN2ICAgICBTZW5kICAg
IFJlY3YKU2l6ZSAgIFNpemUgICAgU2l6ZSAgICAgVGltZSAgICAgVGhyb3VnaHB1dCAgbG9jYWwg
ICAgcmVtb3RlICAgbG9jYWwgICByZW1vdGUKYnl0ZXMgIGJ5dGVzICAgYnl0ZXMgICAgc2Vjcy4g
ICAgMTBeNmJpdHMvcyAgJSBDICAgICAgJSBDICAgICAgdXMvS0IgICB1cy9LQgoKMTMxMDcyICA2
NTUzNiAgNjU1MzYgICAgNDAwLjAyICAgICA0NjI4LjQzICAgNS43MCAgICAgNS40MiAgICAgMy4y
MzEgICAzLjA3MSAgClNhdCBBcHIgIDkgMTE6NDk6NTkgTURUIDIwMTYKCgoKU2F0IEFwciAgOSAx
MTo1MDowOSBNRFQgMjAxNgpNSUdSQVRFRCBUQ1AgU1RSRUFNIFRFU1QgZnJvbSAwLjAuMC4wICgw
LjAuMC4wKSBwb3J0IDAgQUZfSU5FVCB0byBzMS1saW5rMSAoKSBwb3J0IDAgQUZfSU5FVCA6ICsv
LTUuMDAwJSBAIDk1JSBjb25mLiAgOiBoaXN0b2dyYW0gOiBpbnRlcnZhbCA6IGRpcnR5IGRhdGEg
OiBkZW1vCiEhISBXQVJOSU5HCiEhISBEZXNpcmVkIGNvbmZpZGVuY2Ugd2FzIG5vdCBhY2hpZXZl
ZCB3aXRoaW4gdGhlIHNwZWNpZmllZCBpdGVyYXRpb25zLgohISEgVGhpcyBpbXBsaWVzIHRoYXQg
dGhlcmUgd2FzIHZhcmlhYmlsaXR5IGluIHRoZSB0ZXN0IGVudmlyb25tZW50IHRoYXQKISEhIG11
c3QgYmUgaW52ZXN0aWdhdGVkIGJlZm9yZSBnb2luZyBmdXJ0aGVyLgohISEgQ29uZmlkZW5jZSBp
bnRlcnZhbHM6IFRocm91Z2hwdXQgICAgICA6IDMuNTUxJQohISEgICAgICAgICAgICAgICAgICAg
ICAgIExvY2FsIENQVSB1dGlsICA6IDEwLjIxNiUKISEhICAgICAgICAgICAgICAgICAgICAgICBS
ZW1vdGUgQ1BVIHV0aWwgOiAyLjk2MSUKClJlY3YgICBTZW5kICAgIFNlbmQgICAgICAgICAgICAg
ICAgICAgICAgICAgIFV0aWxpemF0aW9uICAgICAgIFNlcnZpY2UgRGVtYW5kClNvY2tldCBTb2Nr
ZXQgIE1lc3NhZ2UgIEVsYXBzZWQgICAgICAgICAgICAgIFNlbmQgICAgIFJlY3YgICAgIFNlbmQg
ICAgUmVjdgpTaXplICAgU2l6ZSAgICBTaXplICAgICBUaW1lICAgICBUaHJvdWdocHV0ICBsb2Nh
bCAgICByZW1vdGUgICBsb2NhbCAgIHJlbW90ZQpieXRlcyAgYnl0ZXMgICBieXRlcyAgICBzZWNz
LiAgICAxMF42Yml0cy9zICAlIEMgICAgICAlIEMgICAgICB1cy9LQiAgIHVzL0tCCgoyNjIxNDQg
MTMxMDcyIDEzMTA3MiAgICA0MDAuMDEgICAgIDQ1NTEuNjEgICA1LjMwICAgICA1LjQ2ICAgICAz
LjA1NyAgIDMuMTQ4ICAKU2F0IEFwciAgOSAxMjo1Njo0OSBNRFQgMjAxNgoKCgpTYXQgQXByICA5
IDEyOjU2OjU5IE1EVCAyMDE2Ck1JR1JBVEVEIFRDUCBTVFJFQU0gVEVTVCBmcm9tIDAuMC4wLjAg
KDAuMC4wLjApIHBvcnQgMCBBRl9JTkVUIHRvIHMxLWxpbmsxICgpIHBvcnQgMCBBRl9JTkVUIDog
Ky8tNS4wMDAlIEAgOTUlIGNvbmYuICA6IGhpc3RvZ3JhbSA6IGludGVydmFsIDogZGlydHkgZGF0
YSA6IGRlbW8KISEhIFdBUk5JTkcKISEhIERlc2lyZWQgY29uZmlkZW5jZSB3YXMgbm90IGFjaGll
dmVkIHdpdGhpbiB0aGUgc3BlY2lmaWVkIGl0ZXJhdGlvbnMuCiEhISBUaGlzIGltcGxpZXMgdGhh
dCB0aGVyZSB3YXMgdmFyaWFiaWxpdHkgaW4gdGhlIHRlc3QgZW52aXJvbm1lbnQgdGhhdAohISEg
bXVzdCBiZSBpbnZlc3RpZ2F0ZWQgYmVmb3JlIGdvaW5nIGZ1cnRoZXIuCiEhISBDb25maWRlbmNl
IGludGVydmFsczogVGhyb3VnaHB1dCAgICAgIDogMjUuOTE4JQohISEgICAgICAgICAgICAgICAg
ICAgICAgIExvY2FsIENQVSB1dGlsICA6IDE0LjAzMCUKISEhICAgICAgICAgICAgICAgICAgICAg
ICBSZW1vdGUgQ1BVIHV0aWwgOiAxMS42MDglCgpSZWN2ICAgU2VuZCAgICBTZW5kICAgICAgICAg
ICAgICAgICAgICAgICAgICBVdGlsaXphdGlvbiAgICAgICBTZXJ2aWNlIERlbWFuZApTb2NrZXQg
U29ja2V0ICBNZXNzYWdlICBFbGFwc2VkICAgICAgICAgICAgICBTZW5kICAgICBSZWN2ICAgICBT
ZW5kICAgIFJlY3YKU2l6ZSAgIFNpemUgICAgU2l6ZSAgICAgVGltZSAgICAgVGhyb3VnaHB1dCAg
bG9jYWwgICAgcmVtb3RlICAgbG9jYWwgICByZW1vdGUKYnl0ZXMgIGJ5dGVzICAgYnl0ZXMgICAg
c2Vjcy4gICAgMTBeNmJpdHMvcyAgJSBDICAgICAgJSBDICAgICAgdXMvS0IgICB1cy9LQgoKNTI0
Mjg4IDI2MjE0NCAyNjIxNDQgICAgNDAwLjAyICAgICA0MTM3LjIzICAgNS41MSAgICAgNS4xOSAg
ICAgMy41ODcgICAzLjM1NiAgClNhdCBBcHIgIDkgMTQ6MDM6NDAgTURUIDIwMTYKCgoKU2F0IEFw
ciAgOSAxNDowMzo1MCBNRFQgMjAxNgpNSUdSQVRFRCBUQ1AgU1RSRUFNIFRFU1QgZnJvbSAwLjAu
MC4wICgwLjAuMC4wKSBwb3J0IDAgQUZfSU5FVCB0byBzMS1saW5rMSAoKSBwb3J0IDAgQUZfSU5F
VCA6ICsvLTUuMDAwJSBAIDk1JSBjb25mLiAgOiBoaXN0b2dyYW0gOiBpbnRlcnZhbCA6IGRpcnR5
IGRhdGEgOiBkZW1vClJlY3YgICBTZW5kICAgIFNlbmQgICAgICAgICAgICAgICAgICAgICAgICAg
IFV0aWxpemF0aW9uICAgICAgIFNlcnZpY2UgRGVtYW5kClNvY2tldCBTb2NrZXQgIE1lc3NhZ2Ug
IEVsYXBzZWQgICAgICAgICAgICAgIFNlbmQgICAgIFJlY3YgICAgIFNlbmQgICAgUmVjdgpTaXpl
ICAgU2l6ZSAgICBTaXplICAgICBUaW1lICAgICBUaHJvdWdocHV0ICBsb2NhbCAgICByZW1vdGUg
ICBsb2NhbCAgIHJlbW90ZQpieXRlcyAgYnl0ZXMgICBieXRlcyAgICBzZWNzLiAgICAxMF42Yml0
cy9zICAlIEMgICAgICAlIEMgICAgICB1cy9LQiAgIHVzL0tCCgoxMDQ4NTc2IDUyNDI4OCA1MjQy
ODggICAgNDAwLjAyICAgICAyOTUyLjE0ICAgNC43NSAgICAgNC43MyAgICAgNC4yMTYgICA0LjE5
NiAgClNhdCBBcHIgIDkgMTQ6NDM6NTAgTURUIDIwMTYKCgoKU2F0IEFwciAgOSAxNDo0NDowMSBN
RFQgMjAxNgpNSUdSQVRFRCBUQ1AgU1RSRUFNIFRFU1QgZnJvbSAwLjAuMC4wICgwLjAuMC4wKSBw
b3J0IDAgQUZfSU5FVCB0byBzMS1saW5rMSAoKSBwb3J0IDAgQUZfSU5FVCA6ICsvLTUuMDAwJSBA
IDk1JSBjb25mLiAgOiBoaXN0b2dyYW0gOiBpbnRlcnZhbCA6IGRpcnR5IGRhdGEgOiBkZW1vClJl
Y3YgICBTZW5kICAgIFNlbmQgICAgICAgICAgICAgICAgICAgICAgICAgIFV0aWxpemF0aW9uICAg
ICAgIFNlcnZpY2UgRGVtYW5kClNvY2tldCBTb2NrZXQgIE1lc3NhZ2UgIEVsYXBzZWQgICAgICAg
ICAgICAgIFNlbmQgICAgIFJlY3YgICAgIFNlbmQgICAgUmVjdgpTaXplICAgU2l6ZSAgICBTaXpl
ICAgICBUaW1lICAgICBUaHJvdWdocHV0ICBsb2NhbCAgICByZW1vdGUgICBsb2NhbCAgIHJlbW90
ZQpieXRlcyAgYnl0ZXMgICBieXRlcyAgICBzZWNzLiAgICAxMF42Yml0cy9zICAlIEMgICAgICAl
IEMgICAgICB1cy9LQiAgIHVzL0tCCgoyMDk3MTUyIDEwNDg1NzYgMTA0ODU3NiAgICA0MDAuMDMg
ICAgIDMwMDEuNDQgICA0Ljk0ICAgICA0Ljg0ICAgICA0LjMxMCAgIDQuMjMxICAKU2F0IEFwciAg
OSAxNTo0NDowMiBNRFQgMjAxNgoKCgpTYXQgQXByICA5IDE1OjQ0OjEyIE1EVCAyMDE2Ck1JR1JB
VEVEIFRDUCBTVFJFQU0gVEVTVCBmcm9tIDAuMC4wLjAgKDAuMC4wLjApIHBvcnQgMCBBRl9JTkVU
IHRvIHMxLWxpbmsxICgpIHBvcnQgMCBBRl9JTkVUIDogKy8tNS4wMDAlIEAgOTUlIGNvbmYuICA6
IGhpc3RvZ3JhbSA6IGludGVydmFsIDogZGlydHkgZGF0YSA6IGRlbW8KUmVjdiAgIFNlbmQgICAg
U2VuZCAgICAgICAgICAgICAgICAgICAgICAgICAgVXRpbGl6YXRpb24gICAgICAgU2VydmljZSBE
ZW1hbmQKU29ja2V0IFNvY2tldCAgTWVzc2FnZSAgRWxhcHNlZCAgICAgICAgICAgICAgU2VuZCAg
ICAgUmVjdiAgICAgU2VuZCAgICBSZWN2ClNpemUgICBTaXplICAgIFNpemUgICAgIFRpbWUgICAg
IFRocm91Z2hwdXQgIGxvY2FsICAgIHJlbW90ZSAgIGxvY2FsICAgcmVtb3RlCmJ5dGVzICBieXRl
cyAgIGJ5dGVzICAgIHNlY3MuICAgIDEwXjZiaXRzL3MgICUgQyAgICAgICUgQyAgICAgIHVzL0tC
ICAgdXMvS0IKCjQxOTQzMDQgMjA5NzE1MiAyMDk3MTUyICAgIDQwMC4zNCAgICAgMjk0OC41NyAg
IDQuNzkgICAgIDQuNzkgICAgIDQuMjU5ICAgNC4yNjIgIApTYXQgQXByICA5IDE2OjEwOjU0IE1E
VCAyMDE2CgoKClNhdCBBcHIgIDkgMTY6MTE6MDQgTURUIDIwMTYKTUlHUkFURUQgVENQIFNUUkVB
TSBURVNUIGZyb20gMC4wLjAuMCAoMC4wLjAuMCkgcG9ydCAwIEFGX0lORVQgdG8gczEtbGluazEg
KCkgcG9ydCAwIEFGX0lORVQgOiArLy01LjAwMCUgQCA5NSUgY29uZi4gIDogaGlzdG9ncmFtIDog
aW50ZXJ2YWwgOiBkaXJ0eSBkYXRhIDogZGVtbwpSZWN2ICAgU2VuZCAgICBTZW5kICAgICAgICAg
ICAgICAgICAgICAgICAgICBVdGlsaXphdGlvbiAgICAgICBTZXJ2aWNlIERlbWFuZApTb2NrZXQg
U29ja2V0ICBNZXNzYWdlICBFbGFwc2VkICAgICAgICAgICAgICBTZW5kICAgICBSZWN2ICAgICBT
ZW5kICAgIFJlY3YKU2l6ZSAgIFNpemUgICAgU2l6ZSAgICAgVGltZSAgICAgVGhyb3VnaHB1dCAg
bG9jYWwgICAgcmVtb3RlICAgbG9jYWwgICByZW1vdGUKYnl0ZXMgIGJ5dGVzICAgYnl0ZXMgICAg
c2Vjcy4gICAgMTBeNmJpdHMvcyAgJSBDICAgICAgJSBDICAgICAgdXMvS0IgICB1cy9LQgoKODM4
ODYwOCA0MTk0MzA0IDQxOTQzMDQgICAgNDAwLjM0ICAgICAyOTQwLjI4ICAgNC4yOCAgICAgNC43
MCAgICAgMy44MTEgICA0LjE5NCAgClNhdCBBcHIgIDkgMTc6MDQ6MzAgTURUIDIwMTYKCgoK
--_006_D3300112FE63ChengCuinetappcom_--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?D3300112.FE63%Cheng.Cui>