Date: Fri, 2 Feb 2024 10:21:46 +0100 From: "Scheffenegger, Richard" <rscheff@freebsd.org> To: "freebsd-net@FreeBSD.org" <freebsd-net@freebsd.org>, FreeBSD Transport <freebsd-transport@freebsd.org>, rmacklem@freebsd.org, gallatin@FreeBSD.org, kp@FreeBSD.org Subject: Increasing TCP TSO size support Message-ID: <2c31ac44-b34b-469c-a6de-fdd927ec2f9e@freebsd.org>
next in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--------------cFlXxpdNPSvFDnJ2Q7jJvaOb
Content-Type: multipart/mixed; boundary="------------imiNeDBqBOWYIblw01WKAtFz";
protected-headers="v1"
From: "Scheffenegger, Richard" <rscheff@freebsd.org>
To: "freebsd-net@FreeBSD.org" <freebsd-net@freebsd.org>,
FreeBSD Transport <freebsd-transport@freebsd.org>, rmacklem@freebsd.org,
gallatin@FreeBSD.org, kp@FreeBSD.org
Message-ID: <2c31ac44-b34b-469c-a6de-fdd927ec2f9e@freebsd.org>
Subject: Increasing TCP TSO size support
--------------imiNeDBqBOWYIblw01WKAtFz
Content-Type: multipart/mixed; boundary="------------f8bM96TDLFPthBSFcPuWJDK5"
--------------f8bM96TDLFPthBSFcPuWJDK5
Content-Type: multipart/alternative;
boundary="------------g30wiijxOrV2m8hMPa7cSCrN"
--------------g30wiijxOrV2m8hMPa7cSCrN
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: base64
DQpIaSwNCg0KV2UgaGF2ZSBydW4gYSB0ZXN0IGZvciBhIFJQQyB3b3JrbG9hZCB3aXRoIDFN
QiBJTyBzaXplcywgYW5kIGNvbGxlY3RlZCANCnRoZSB0Y3BfZGVmYXVsdF9vdXRwdXQoKSBs
ZW4oZ3RoKSBkdXJpbmcgdGhlIGZpcnN0IHBhc3MgaW4gdGhlIG91dHB1dCBsb29wLg0KDQpJ
biBzdWNoIGEgc2NlbmFyaW8sIHdoZXJlIHRoZSBhcHBsaWNhdGlvbiBmcmVxdWVudGx5IGlu
dHJvZHVjZXMgc21hbGwgDQpwYXVzZXMgKHNpbmNlIHRoZSBuZXh0IGxhcmdlIElPIGlzIG9u
bHkgc2VudCBhZnRlciB0aGUgY29ycmVzcG9uZGluZyANCnJlcXVlc3QgZnJvbSB0aGUgY2xp
ZW50IGhhcyBiZWVuIHJlY2VpdmVkIGFuZCBwcm9jZXNzZWQpIGJldHdlZW4gc2VuZGluZyAN
CmFkZGl0aW9uYWwgZGF0YSwgdGhlIGN1cnJlbnQgVFNPIGxpbWl0IG9mIDY0a0IgVFNPIG1h
eGltdW0gKDQ1KjE0NDggaW4gDQplZmZlY3QpIHJlcXVpcmVzIG11bHRpcGxlIHBhc3NlcyBp
biB0aGUgb3V0cHV0IHJvdXRpbmUgdG8gc2VuZCBhbGwgdGhlIA0KYWxsb3dhYmxlIChjd25k
IGxpbWl0ZWQpIGRhdGEuDQoNCkknbGwgdHJ5IHRvIGdldCBhIGRhdGEgY29sbGVjdGlvbiB3
aXRoIGJldHRlciBncmFudWxhcml5IGFib3ZlIDkwIDAwMCANCmJ5dGVzIC0gYnV0IGV2ZW4g
aGVyZSB0aGUgYXZlcmFnZSBzdHJvbmdseSBpbmRpY2F0ZXMgdGhhdCBhIG1ham9yaXR5IG9m
IA0KdHJhbnNtaXNzaW9uIG9wcG9ydHVuaXRpZXMgYXJlIGluIHRoZSA1MTIga0IgYXJlYSAt
IHByb2JhYmx5IGFsc28gaGF2aW5nIA0KdG8gZG8gd2l0aCBMUk8gYW5kIEFDSyB0aGlubmlu
ZyBlZmZlY3RzIGJ5IHRoZSBjbGllbnQuDQoNCldpdGggb3RoZXIgd29yZHMsIHRoZSB0Y3Ag
b3V0cHV0IGhhcyB0byBydW4gYWJvdXQgOSB0aW1lcyB3aXRoIFRTTywgdG8gDQp0cmFuc21p
dCBhbGwgZWxlZ2libGUgZGF0YSAtIGluY3JlYXNpbmcgdGhlIEZyZWVCU0Qgc3VwcG9ydGVk
IG1heGltdW0gDQpUU08gc2l6ZSB0byB3aGF0IGN1cnJlbnQgaGFyZHdhcmUgY291bGQgaGFu
ZGxlICgyNTZrQi4uMU1CKSB3b3VsZCByZWR1Y2UgDQp0aGUgQ1BVIGJ1cmRlbiBoZXJlLg0K
DQoNCklzIGluY3JlYXNpbmcgdGhlIHNvZndhcmUgc3VwcG9ydGVkIFRTTyBzaXplIHRvIGFs
bG93IGZvciB3aGF0IHRoZSBOSUNzIA0KY291bGQgbm93YWRheXMgZG8gc29tZXRoaW5nIGFu
eW9uZSBhcGFydCBmcm9tIHVzIHdvdWxkIGJlIGludGVyZXN0ZWQgaW4gDQooaW4gcGFydGlj
dWxhciwgdGhvc2Ugd2hvIHdvcmsgd2l0aCB0aGUgZHJpdmVycyk/DQoNCg0KQmVzdCByZWdh
cmRzLA0KDQogwqAgUmljaGFyZA0KDQoNCg0KDQp0c28gc2l6ZSAodHJhbnNtaXNzaW9ucyA8
IDE0NDggd291bGQgbm90IGJlIGFjY291bnRlZCBoZXJlIGF0IGFsbCkNCg0KIMKgwqDCoCDC
oMKgwqAgwqDCoMKgIMKgwqDCoCDCoMKgwqAgIyBjb3VudA0KDQo8MTAwMCAJMA0KPDIwMDAg
CTIzDQo8MzAwMCAJMTExDQo8NDAwMCAJNDANCjw1MDAwIAkzMA0KPDcwMDAgCTE0DQo8ODAw
MCAJMTM0DQo8OTAwMCAJNDQyDQo8MTAwMDAgCTkzOTYNCjwyMDAwMCAJNDYyMjcNCjwzMDAw
MCAJMjU2NDYNCjw0MDAwMCAJMzMwNjANCjw2MDAwMCAJMjMxNjINCjw3MDAwMCAJMjQzNjgN
Cjw4MDAwMCAJMTk3NzINCjw5MDAwMCAJNDAxMDENCiA+PTkwMDAwIAk3NTM4NDE2OQ0KQXZl
cmFnZTogCTU3ODg0NC40NA0KDQo=
--------------g30wiijxOrV2m8hMPa7cSCrN
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE html>
<html>
<head>
<meta http-equiv=3D"content-type" content=3D"text/html; charset=3DUTF=
-8">
</head>
<body>
<p><br>
</p>
<p>Hi,</p>
<p>We have run a test for a RPC workload with 1MB IO sizes, and
collected the tcp_default_output() len(gth) during the first pass
in the output loop.</p>
<p>In such a scenario, where the application frequently introduces
small pauses (since the next large IO is only sent after the
corresponding request from the client has been received and
processed) between sending additional data, the current TSO limit
of 64kB TSO maximum (45*1448 in effect) requires multiple passes
in the output routine to send all the allowable (cwnd limited)
data.<br>
</p>
<p>I'll try to get a data collection with better granulariy above 90
000 bytes - but even here the average strongly indicates that a
majority of transmission opportunities are in the 512 kB area -
probably also having to do with LRO and ACK thinning effects by
the client.<br>
</p>
<p>With other words, the tcp output has to run about 9 times with
TSO, to transmit all elegible data - increasing the FreeBSD
supported maximum TSO size to what current hardware could handle
(256kB..1MB) would reduce the CPU burden here.</p>
<p><br>
</p>
<p>Is increasing the sofware supported TSO size to allow for what
the NICs could nowadays do something anyone apart from us would be
interested in (in particular, those who work with the drivers)?</p>=
<p><br>
</p>
<p>Best regards,</p>
<p>=C2=A0 Richard</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p>tso size (transmissions < 1448 would not be accounted here at
all)</p>
<p>=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=
=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # count<br>
</p>
<p> </p>
<table width=3D"128" cellspacing=3D"0" cellpadding=3D"0" border=3D"0"=
>
<colgroup><col style=3D"width:48pt" width=3D"64" span=3D"2"> </colg=
roup><tbody>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt;width:48pt" width=3D"64" height=3D"1=
9"><1000</td>
<td style=3D"width:48pt" width=3D"64" align=3D"right">0</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><2000</td>
<td align=3D"right">23</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><3000</td>
<td align=3D"right">111</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><4000</td>
<td align=3D"right">40</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><5000</td>
<td align=3D"right">30</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><7000</td>
<td align=3D"right">14</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><8000</td>
<td align=3D"right">134</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><9000</td>
<td align=3D"right">442</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><10000</td>
<td align=3D"right">9396</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><20000</td>
<td align=3D"right">46227</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><30000</td>
<td align=3D"right">25646</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><40000</td>
<td align=3D"right">33060</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><60000</td>
<td align=3D"right">23162</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><70000</td>
<td align=3D"right">24368</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><80000</td>
<td align=3D"right">19772</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19"><90000</td>
<td align=3D"right">40101</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19">>=3D90000</td>
<td align=3D"right">75384169</td>
</tr>
<tr style=3D"height:14.4pt" height=3D"19">
<td style=3D"height:14.4pt" height=3D"19">Average:</td>
<td>578844.44</td>
</tr>
</tbody>
</table>
</body>
</html>
--------------g30wiijxOrV2m8hMPa7cSCrN--
--------------f8bM96TDLFPthBSFcPuWJDK5
Content-Type: application/pgp-keys; name="OpenPGP_0x17BE5899E0B1439B.asc"
Content-Disposition: attachment; filename="OpenPGP_0x17BE5899E0B1439B.asc"
Content-Description: OpenPGP public key
Content-Transfer-Encoding: quoted-printable
-----BEGIN PGP PUBLIC KEY BLOCK-----
xjMEY/i74RYJKwYBBAHaRw8BAQdAwtnvjlFVnnzNXO9hjHtB6MPGSY19L/BHh/iz
iPF0FzrNK1JpY2hhcmQgU2NoZWZmZW5lZ2dlciA8cnNjaGVmZkBmcmVlYnNkLm9y
Zz7CmgQTFgoAQhYhBDZLt5msg0Ras820cRe+WJngsUObBQJj+LvhAhsDBQkJZgGA
BQsJCAcCAyICAQYVCgkICwIEFgIDAQIeBwIXgAAKCRAXvliZ4LFDm4ylAQCSw2/n
vht8kExJ31M+3qpjOqdVypMp+/Ojvh5Zlsk96QEA5HCBkteJcrohwRA7llZvLH3m
25hcJdzmDh39mc0cSgPOOARj+LvhEgorBgEEAZdVAQUBAQdA1Dim8ZWpXRS5i9hb
3O4RNHub8XvqTTkYyiZ2lSkXDwYDAQgHwn4EGBYKACYWIQQ2S7eZrINEWrPNtHEX
vliZ4LFDmwUCY/i74QIbDAUJCWYBgAAKCRAXvliZ4LFDm2TGAQDcg+bAEPqOH+JC
IND8wZ62MwnjFyXFv73qevXkUHHNSgEApUgpHW9f6UaIAQpc3R185xjz6tk8XXBx
eYpxKgIAeQ8=3D
=3DBwxS
-----END PGP PUBLIC KEY BLOCK-----
--------------f8bM96TDLFPthBSFcPuWJDK5--
--------------imiNeDBqBOWYIblw01WKAtFz--
--------------cFlXxpdNPSvFDnJ2Q7jJvaOb
Content-Type: application/pgp-signature; name="OpenPGP_signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="OpenPGP_signature.asc"
-----BEGIN PGP SIGNATURE-----
wnsEABYIACMWIQQ2S7eZrINEWrPNtHEXvliZ4LFDmwUCZby0KgUDAAAAAAAKCRAXvliZ4LFDm73n
AQC00+GZ7IDDzaPlORXmsNmMrrKgmFRT/rH6+MIpnbvA8AEA+130h9m/ksyrLjAOFWgXqG68ByH7
+gaZN4mhV5Q13Q0=
=QnaZ
-----END PGP SIGNATURE-----
--------------cFlXxpdNPSvFDnJ2Q7jJvaOb--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2c31ac44-b34b-469c-a6de-fdd927ec2f9e>