From nobody Fri Feb 2 22:14:15 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TRVSF2spcz58Nrp; Fri, 2 Feb 2024 22:14:37 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TRVSF25c5z40RM; Fri, 2 Feb 2024 22:14:37 +0000 (UTC) (envelope-from gallatin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706912077; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=DjtbGOYAruVAO+vh5MF8LOAoRii6LS0uXXBjX+ehR1U=; b=QzX/5UC0ozDjLhqFyBtpK2PejmbCytLfc+jP15nI2VcItz4Rb8qldzOS95535nl56A7H37 R2hxR9DHdloUNnzXe0KpCR1CHJuCOgeLn9pFkIrwui4ZcSqmREQl1As1iZNvlRKzBV2A2m yAzpCefwfOo0eT9ousAVC/MrVAle7mHB56g3hnzOa1sovUdObE0R9zDlZ+CqjwNdTm2mTt +wOBOnDtTic4OloKFGKBjYZZV2rmjhrfwjdSh07no5sbnZHI5Ynb/LSFqBrXcEdoRnmkpb b2JB6UNf1pxNCyHRzH88u4TbfpnU6Z4D7vykclLsHCKQsFznl6AtFq9FjTEOXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706912077; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=DjtbGOYAruVAO+vh5MF8LOAoRii6LS0uXXBjX+ehR1U=; b=QfQkV0Y0i/C9rOBGyHM417YMKxTWJN78+rDZS2sCsUOkH5CyIdvrAyjXLWKkWx3gjszWdJ 90mUAdNtq0HzulpjjVwe526JqwKHqdUwH+vlo4z9LvDF3oFHym3kfT/smLEkrJhzlKNkx7 txGs1tgs21MV+efLxZOjjDtVIU5nmpoyMoGDjikD+1s6N6BOhwayRbt3NN7LUHv7+RNids jOMmXGOQOgA/ANYchsutY3ml386XN35fWCjqU3MuiRdWG04mnmdIrmkKCP4v3mfMyn+38Y Y4xvMPkEenxV1fsuXgh1wtQKAx1e4Ryd4JomkTEdbKoDL9FE1cXAzj0qxmWXuA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706912077; a=rsa-sha256; cv=none; b=ak3YWdgJ1XQ9UQtVnnDDBqsFKN3uDwEL6W/y0tNlmqz9l7BH5ip+fiHqP3cjzWGKdvegep 4/jgAOCOG1xPuI5GDqlKbnLqTCrdHBBtN4fQrSWHwuS+msHESDIXfT7spc29OQTmfqj/Zj eyembYZQD5SHsjzAy9RwkxqdYTFiEnmx1ODllRLP+3qgRlQqc3rS4bfMk+SasrF6rJZHjd WhJlHUDjwxJs/CoAPGyG9cxx3JBrCVfd+055U8yEmA4qdeh4qdp74b6JZNYShG9XdGtH20 lPJmiojrfMA4LG/Uo8f0LS+ufsYOB+SSQRRRrEtAaAZ3++IQT6IWMEuAbo1SyA== Received: from auth1-smtp.messagingengine.com (auth1-smtp.messagingengine.com [66.111.4.227]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: gallatin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TRVSF0qxMz11tq; Fri, 2 Feb 2024 22:14:37 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailauth.nyi.internal (Postfix) with ESMTP id 2010D27C005B; Fri, 2 Feb 2024 17:14:36 -0500 (EST) Received: from imap53 ([10.202.2.103]) by compute5.internal (MEProxy); Fri, 02 Feb 2024 17:14:36 -0500 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedugedgudehjecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefofgggkfgjfhffhffvufgtsegrtderreerredtnecuhfhrohhmpedfffhr vgifucfirghllhgrthhinhdfuceoghgrlhhlrghtihhnsehfrhgvvggsshgurdhorhhgqe enucggtffrrghtthgvrhhnpeehjeevleeigeefffdtgedtudegheetteeigfeileejuedv gefhvdekjedvhefhveenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehgrghllhgrthhinhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihht hidqudeffeehledvvdduiedqvdelhedtgedukeegqdhgrghllhgrthhinheppehfrhgvvg gsshgurdhorhhgsehfrghsthhmrghilhdrtghomh X-ME-Proxy: Feedback-ID: i41414658:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id D7AF6364006B; Fri, 2 Feb 2024 17:14:35 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-144-ge5821d614e-fm-20240125.002-ge5821d61 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Message-Id: <95e76a2c-44c8-4fbb-ab45-8bcffe80d4a3@app.fastmail.com> In-Reply-To: <2c31ac44-b34b-469c-a6de-fdd927ec2f9e@freebsd.org> References: <2c31ac44-b34b-469c-a6de-fdd927ec2f9e@freebsd.org> Date: Fri, 02 Feb 2024 17:14:15 -0500 From: "Drew Gallatin" To: "Richard Scheffenegger" , "freebsd-net@FreeBSD.org" , "FreeBSD Transport" , rmacklem@freebsd.org, kp@FreeBSD.org Subject: Re: Increasing TCP TSO size support Content-Type: multipart/alternative; boundary=16179d38f0f74410b648a6fe3220c987 --16179d38f0f74410b648a6fe3220c987 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable What is the link speed that you're working with? A long time ago, when I worked for a now-defunct 10GbE NIC vendor, I exp= erimented with the benefits of TSO as we varied the max TSO size. I ca= nnot recall the platform (it could have been OSX, Solaris, FreeBSD or Li= nux). At the time (~2006?) the CPU saving benefits of increasing the ma= x TSO size from 8k to 64k was fairly minimal. In fact, I seem to reca= ll that there was almost no benefit to TSO sizes larger than 16K. I was wondering if you see any difference in your benchmark if you cap m= ax TSO size to 8k, 16k,32k, and the default of 64k. Any change in CPU = use, or in your benchmark's performance would be interesting to hear abo= ut. Naively, I'd expect the benchmark performance to remain unchanged until = you'd reduced the TSO size so much as to make the host slower than the w= ire, thereby inserting gaps between TSOs. That would be reflected in th= e CPU use as well.. Drew On Fri, Feb 2, 2024, at 4:21 AM, Scheffenegger, Richard wrote: >=20 >=20 > Hi, >=20 > We have run a test for a RPC workload with 1MB IO sizes, and collected= the tcp_default_output() len(gth) during the first pass in the output l= oop. >=20 > In such a scenario, where the application frequently introduces small = pauses (since the next large IO is only sent after the corresponding req= uest from the client has been received and processed) between sending ad= ditional data, the current TSO limit of 64kB TSO maximum (45*1448 in eff= ect) requires multiple passes in the output routine to send all the allo= wable (cwnd limited) data. >=20 > I'll try to get a data collection with better granulariy above 90 000 = bytes - but even here the average strongly indicates that a majority of = transmission opportunities are in the 512 kB area - probably also having= to do with LRO and ACK thinning effects by the client. >=20 > With other words, the tcp output has to run about 9 times with TSO, to= transmit all elegible data - increasing the FreeBSD supported maximum T= SO size to what current hardware could handle (256kB..1MB) would reduce = the CPU burden here. >=20 >=20 >=20 > Is increasing the sofware supported TSO size to allow for what the NIC= s could nowadays do something anyone apart from us would be interested i= n (in particular, those who work with the drivers)? >=20 >=20 >=20 > Best regards, >=20 > Richard >=20 >=20 >=20 >=20 >=20 >=20 >=20 > tso size (transmissions < 1448 would not be accounted here at all) >=20 > # count >=20 >=20 >=20 > <1000 > 0 > <2000 > 23 > <3000 > 111 > <4000 > 40 > <5000 > 30 > <7000 > 14 > <8000 > 134 > <9000 > 442 > <10000 > 9396 > <20000 > 46227 > <30000 > 25646 > <40000 > 33060 > <60000 > 23162 > <70000 > 24368 > <80000 > 19772 > <90000 > 40101 > >=3D90000 > 75384169 > Average: > 578844.44 >=20 > *Attachments:* > =E2=80=A2 OpenPGP_0x17BE5899E0B1439B.asc > =E2=80=A2 OpenPGP_signature.asc --16179d38f0f74410b648a6fe3220c987 Content-Type: text/html Content-Transfer-Encoding: quoted-printable
What is the lin= k speed that you're working with?

A long ti= me ago, when I worked for a now-defunct 10GbE NIC vendor, I experimented= with the benefits of  TSO as we varied the max TSO size.  I c= annot recall the platform (it could have been OSX, Solaris, FreeBSD or L= inux).  At the time (~2006?) the CPU saving benefits of increasing = the max TSO size from 8k to 64k was fairly minimal.    In= fact, I seem to recall that there was almost no benefit to TSO sizes la= rger than 16K.

I was wondering if you see a= ny difference in your benchmark if you cap max TSO size to 8k,  16k= ,32k, and the default of 64k.  Any change in CPU use, or in your be= nchmark's performance would be interesting to hear about.
=
Naively, I'd expect the benchmark performance to remain u= nchanged until you'd reduced the TSO size so much as to make the host sl= ower than the wire, thereby inserting gaps between TSOs.  That woul= d be reflected in the CPU use as well..

Dre= w

On Fri, Feb 2, 2024, at 4:21 AM, Scheffen= egger, Richard wrote:


Hi,

We have run a test for a RPC workload = with 1MB IO sizes, and collected the tcp_default_output() len(gth) during the first pass in the output loop.

In such a scenario, where the applic= ation frequently introduces small pauses (since the next large IO is only sent after the corresponding request from the client has been received and processed) between sending additional data, the current TSO limit of 64kB TSO maximum (45*1448 in effect) requires multiple passes in the output routine to send all the allowable (cwnd limited) data.

I'll try to get a data collection with better gran= ulariy above 90 000 bytes - but even here the average strongly indicates that a majority of transmission opportunities are in the 512 kB area - probably also having to do with LRO and ACK thinning effects by the client.

With other words, the tcp output has to run = about 9 times with TSO, to transmit all elegible data - increasing the FreeBSD supported maximum TSO size to what current hardware could handle (256kB..1MB) would reduce the CPU burden here.


<= p>Is increasing the sofware supported TSO size to allow for what the NICs could nowadays do something anyone apart from us would be interested in (in particular, those who work with the drivers)?


Best regards,

  Richard


=



tso size (transmissions < 1448 would not= be accounted here at all)

          &= nbsp;         # count


<1000
0
&l= t;2000
23
<300= 0
111
<400040
<5000
30
<7000
14
<8000
134
<= td style=3D"height:14.4pt;" height=3D"19"><9000
442
<10000
9396
<20000
46227
<30000
25646
<40000
= 33060
<60000
231= 62
<70000
2436= 8
<80000
19772=
<90000
40101<= br>>=3D90000
7538= 4169
Average:
578844.44

Attachments:
    =
  • OpenPGP_0x17BE5899E0B1439B.asc
  • OpenPGP_signature.asc
    =

--16179d38f0f74410b648a6fe3220c987--