From nobody Wed May 24 23:55:26 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRSk55fSbz4V9Wl for ; Wed, 24 May 2023 23:55:41 +0000 (UTC) (envelope-from ccfreebsd@gmail.com) Received: from mail-lf1-f42.google.com (mail-lf1-f42.google.com [209.85.167.42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRSk53v1pz3jj0 for ; Wed, 24 May 2023 23:55:41 +0000 (UTC) (envelope-from ccfreebsd@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-f42.google.com with SMTP id 2adb3069b0e04-4f3a873476bso1613605e87.1 for ; Wed, 24 May 2023 16:55:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684972539; x=1687564539; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=AN/JC1NdyKdS4585xwSn/go7N11udpH3YBxCZcdwtqg=; b=VyFz2weR7ORci55Kk8zIqGEK8Hhw57fUsJuHz3jBH8sQCq/EsLDKMM4a2Dz9Mdmjbq IOYeBYbS5+oPJcJ740ffdmQI1GBlMhzT8D95emGQ1cuvMSQbVPllOmfjS77ObumCmSC9 8wFPA5577o6lXDZjo71lcAhUKOobWtlnuChtjttLxkomzipakOsDdyFIR83jmwPQVLWs jdLzae/MLtniJmJ7kG91QeTJB5FO2C3c2DSgETWwkFBPOwCzutarqrLDxEwdDy1++BqL jS5li0JnRkKQ8ChXZ0dtPfQJmH08jCOxGJXwxOKajNNRxh1MBdo+ib+EndD9MDEFUYBi kwgQ== X-Gm-Message-State: AC+VfDwmOWaMUavVeRF5SAGY/V93GXA2TcuRNBbkEriyiFgzHbiJHjO4 qcub8Xnclv1pi0ghXgIcWsuHs6CYrFF4WQ== X-Google-Smtp-Source: ACHHUZ4wzLnhCLFZgDu5xwgyWMGuk2CDU7feA8CODwG5vp12Qxp5eFMRexBILjcnwnhW2O4qzczRWw== X-Received: by 2002:ac2:5232:0:b0:4f3:a51f:54fb with SMTP id i18-20020ac25232000000b004f3a51f54fbmr4788976lfl.59.1684972538710; Wed, 24 May 2023 16:55:38 -0700 (PDT) Received: from mail-lj1-f171.google.com (mail-lj1-f171.google.com. [209.85.208.171]) by smtp.gmail.com with ESMTPSA id m7-20020ac24ac7000000b004f24bc0cbf9sm1876406lfp.154.2023.05.24.16.55.38 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 24 May 2023 16:55:38 -0700 (PDT) Received: by mail-lj1-f171.google.com with SMTP id 38308e7fff4ca-2af31dc49f9so19300681fa.0 for ; Wed, 24 May 2023 16:55:38 -0700 (PDT) X-Received: by 2002:a05:6512:11e7:b0:4dd:af29:92c1 with SMTP id p7-20020a05651211e700b004ddaf2992c1mr5388378lfs.44.1684972538377; Wed, 24 May 2023 16:55:38 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 References: <20230523231533.32c1d022@x25> In-Reply-To: From: Cheng Cui Date: Wed, 24 May 2023 19:55:26 -0400 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: how to increase the vnet speed? To: Benoit Chesneau Cc: Marko Zec , "freebsd-net@FreeBSD.org" Content-Type: multipart/alternative; boundary="0000000000009b3c3c05fc793d88" X-Rspamd-Queue-Id: 4QRSk53v1pz3jj0 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000009b3c3c05fc793d88 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable >From TCP's point of view, your iperf result shows "Retr" as retransmissions. Well, that tells packet drops or something else which makes TCP's congestion control into playing throttling the throughput to a certain degree. Try a different TCP congestion control. If the default is CUBIC, try to use NewReno. I don't know your FreeBSD version. So if it is FreeBSD14-current, I would like to see the difference as there is improvement discussion on CUBIC. cc@fbsd ~$ sysctl net.inet.tcp.cc net.inet.tcp.cc.newreno.beta_ecn: 80 net.inet.tcp.cc.newreno.beta: 50 net.inet.tcp.cc.abe_frlossreduce: 0 net.inet.tcp.cc.abe: 0 net.inet.tcp.cc.available: newreno net.inet.tcp.cc.algorithm: newreno cc@fbsd ~$ sysctl net.inet.tcp.cc.available net.inet.tcp.cc.available: newreno cc@fbsd ~$ sudo kldload cc_cubic << load TCP congestion control module cc_cubic cc@fbsd ~$ sysctl net.inet.tcp.cc.available net.inet.tcp.cc.available: newreno, cubic cc@fbsd ~$ sudo sysctl net.inet.tcp.cc.algorithm=3Dcubic net.inet.tcp.cc.algorithm: newreno -> cubic cc@fbsd ~$ sudo sysctl net.inet.tcp.cc.algorithm net.inet.tcp.cc.algorithm: cubic cc@fbsd ~$ uname -a FreeBSD fbsd.cc.home 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64 cc@fbsd ~$ Best Regards, Cheng Cui On Wed, May 24, 2023 at 2:19=E2=80=AFAM Benoit Chesneau wrote: > Sorry, I thought I posted it but it's a bridge: > > ``` > vlan200: flags=3D8943 met= ric > 0 mtu 9000 > > options=3D1c280401 > ether 9c:dc:71:4c:84:f0 > groups: vlan > vlan: 200 vlanproto: 802.1q vlanpcp: 0 parent interface: mce0 > media: Ethernet 25GBase-SR > status: active > nd6 options=3D29 > vlan200bridge: flags=3D8843 metri= c 0 > mtu 9000 > ether 58:9c:fc:10:ff:95 > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: e0a_bastille0 flags=3D143 > ifmaxaddr 0 port 10 priority 128 path cost 2000 > member: vlan200 flags=3D143 > ifmaxaddr 0 port 8 priority 128 path cost 800 > groups: bridge > nd6 options=3D9 > e0a_bastille0: flags=3D8963 > metric 0 mtu 9000 > description: vnet host interface for Bastille jail testing > options=3D8 > ether 02:20:98:4c:84:f0 > hwaddr 02:68:8a:24:67:0a > groups: epair > media: Ethernet 10Gbase-T (10Gbase-T ) > status: active > nd6 options=3D29 > ``` > > After relaunching the machine and removed filtering: > > ``` > net.link.bridge.pfil_bridge=3D0 > net.link.bridge.pfil_onlyip=3D0 > net.link.bridge.pfil_member=3D0 > ``` > > I get better results. Still not dull speed but since it's in a bridge > seems normal. Unsure what was the issue... > > ``` > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.01 sec 1.75 GBytes 14.8 Gbits/sec 74 936 KBytes > [ 5] 1.01-2.00 sec 1.31 GBytes 11.3 Gbits/sec 27 1.76 MBytes > [ 5] 2.00-3.00 sec 2.12 GBytes 18.2 Gbits/sec 34 1.74 MBytes > [ 5] 3.00-4.00 sec 2.08 GBytes 17.9 Gbits/sec 85 1.75 MBytes > [ 5] 4.00-5.00 sec 2.11 GBytes 18.2 Gbits/sec 37 1.75 MBytes > [ 5] 5.00-6.00 sec 2.09 GBytes 18.0 Gbits/sec 60 1.75 MBytes > [ 5] 6.00-7.00 sec 2.11 GBytes 18.2 Gbits/sec 10 1.50 MBytes > [ 5] 7.00-8.00 sec 1.51 GBytes 13.0 Gbits/sec 27 1.75 MBytes > [ 5] 8.00-9.00 sec 1.48 GBytes 12.7 Gbits/sec 75 1.50 MBytes > [ 5] 9.00-10.00 sec 2.09 GBytes 17.9 Gbits/sec 52 1.58 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 18.7 GBytes 16.0 Gbits/sec 481 > sender > [ 5] 0.00-10.00 sec 18.7 GBytes 16.0 Gbits/sec > receiver > ``` > > > > Beno=C3=AEt > > > ------- Original Message ------- > On Tuesday, May 23rd, 2023 at 23:15, Marko Zec wrote: > > > > On Tue, 23 May 2023 19:58:07 +0000 > > Benoit Chesneau benoitc@enki-multimedia.eu wrote: > > > > > Hi all, > > > > > > I've created a jail using bastille and setup network. The mainin > > > terface is a 25Gbps nic and between hosts I get 24.6 Gbits/sec : > > > > > > [...] > > > > > But between one host and the jail I only get 3.96 Gbits/sec > > > > > > [...] > > > > > Is there a way to increase the performance of the of the jail? The > > > nice is a mellannox ConnectX-4 Lx, mce(4) . > > > > > > Modern NICs offload a lot of the protocol stack processing (checksum, > > segmentation, and / or reassembly) from the CPU to dedicated silicon, > > whereas inter-vnet traffic needs to be handled completely in software, > > that's where the difference comes from. > > > > Perhaps we could gain some speed by abusing mbuf flags to skip RXCSUM > > for epair traffic, maybe even skip and fake TXCSUM... > > > > Marko > > --0000000000009b3c3c05fc793d88 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
From TCP's point of view, your iperf result shows= "Retr" as retransmissions. Well, that tells packet drops or some= thing else which makes TCP's congestion control into playing throttling= the throughput to a certain degree.

Try a differe= nt TCP congestion control. If the default is CUBIC, try to use NewReno. I d= on't know your FreeBSD version. So if it is FreeBSD14-current, I would = like to see the difference as there is improvement discussion on CUBIC.
=

cc@fbsd ~$ sysctl net.inet.tcp.cc
net.inet.tcp.cc.newreno.beta_ecn: 80
net.inet= .tcp.cc.newreno.beta: 50
net.inet.tcp.cc.abe_frlossreduce: 0
net.inet= .tcp.cc.abe: 0
net.inet.tcp.cc.available: newreno
net.inet.tcp.cc.alg= orithm: newreno
cc@fbsd ~$ sysctl net.inet.tcp.cc.available
ne= t.inet.tcp.cc.available: newreno
cc@fbsd ~$ sudo kldload cc_cubic=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 << load TCP= congestion control module cc_cubic
cc@fbsd ~$ sysctl net.inet.tcp.cc.av= ailable
net.inet.tcp.cc.available: newreno, cubic
cc@fbsd ~$ sudo sys= ctl net.inet.tcp.cc.algorithm=3Dcubic
net.inet.tcp.cc.algorithm: newreno= -> cubic
=C2=A0=C2=A0 =C2=A0
cc@fbsd ~$ sudo sysctl net.inet.tcp= .cc.algorithm =C2=A0 =C2=A0 =C2=A0
net.inet.tcp.cc.algorithm: cubic
c= c@fbsd ~$ uname -a
FreeBSD fbsd.cc.home 13.1-RELEASE FreeBSD 13.1-RELEAS= E releng/13.1-n250148-fc952ac2212 GENERIC amd64
cc@fbsd ~$


Best Regards,
Cheng Cui


On Wed, May 24, 2023 at 2:19=E2=80=AFAM Benoit Chesneau <benoitc@enki-multimedia.eu&= gt; wrote:
Sorry= , I thought I posted it but it's a bridge:

```
vlan200: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>= metric 0 mtu 9000
=C2=A0 =C2=A0 =C2=A0 =C2=A0 options=3D1c280401<RXCSUM,LRO,LINKSTATE,RXCS= UM_IPV6,NOMAP,TXTLS4,TXTLS6>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ether 9c:dc:71:4c:84:f0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 groups: vlan
=C2=A0 =C2=A0 =C2=A0 =C2=A0 vlan: 200 vlanproto: 802.1q vlanpcp: 0 parent i= nterface: mce0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 media: Ethernet 25GBase-SR <full-duplex,rxpa= use,txpause>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 status: active
=C2=A0 =C2=A0 =C2=A0 =C2=A0 nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_= LINKLOCAL>
vlan200bridge: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> m= etric 0 mtu 9000
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ether 58:9c:fc:10:ff:95
=C2=A0 =C2=A0 =C2=A0 =C2=A0 id 00:00:00:00:00:00 priority 32768 hellotime 2= fwddelay 15
=C2=A0 =C2=A0 =C2=A0 =C2=A0 maxage 20 holdcnt 6 proto rstp maxaddr 2000 tim= eout 1200
=C2=A0 =C2=A0 =C2=A0 =C2=A0 root id 00:00:00:00:00:00 priority 32768 ifcost= 0 port 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 member: e0a_bastille0 flags=3D143<LEARNING,D= ISCOVER,AUTOEDGE,AUTOPTP>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ifmaxaddr 0 port 10= priority 128 path cost 2000
=C2=A0 =C2=A0 =C2=A0 =C2=A0 member: vlan200 flags=3D143<LEARNING,DISCOVE= R,AUTOEDGE,AUTOPTP>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ifmaxaddr 0 port 8 = priority 128 path cost 800
=C2=A0 =C2=A0 =C2=A0 =C2=A0 groups: bridge
=C2=A0 =C2=A0 =C2=A0 =C2=A0 nd6 options=3D9<PERFORMNUD,IFDISABLED> e0a_bastille0: flags=3D8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICA= ST> metric 0 mtu 9000
=C2=A0 =C2=A0 =C2=A0 =C2=A0 description: vnet host interface for Bastille j= ail testing
=C2=A0 =C2=A0 =C2=A0 =C2=A0 options=3D8<VLAN_MTU>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ether 02:20:98:4c:84:f0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 hwaddr 02:68:8a:24:67:0a
=C2=A0 =C2=A0 =C2=A0 =C2=A0 groups: epair
=C2=A0 =C2=A0 =C2=A0 =C2=A0 media: Ethernet 10Gbase-T (10Gbase-T <full-d= uplex>)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 status: active
=C2=A0 =C2=A0 =C2=A0 =C2=A0 nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_= LINKLOCAL>
```

After relaunching the machine and removed filtering:

```
net.link.bridge.pfil_bridge=3D0
net.link.bridge.pfil_onlyip=3D0
net.link.bridge.pfil_member=3D0
```

I get better results. Still not dull speed but since it's in a bridge s= eems normal. Unsure what was the issue...

```
[ ID] Interval=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Transfer=C2=A0 =C2= =A0 =C2=A0Bitrate=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Retr=C2=A0 Cwnd
[=C2=A0 5]=C2=A0 =C2=A00.00-1.01=C2=A0 =C2=A0sec=C2=A0 1.75 GBytes=C2=A0 14= .8 Gbits/sec=C2=A0 =C2=A074=C2=A0 =C2=A0 936 KBytes
[=C2=A0 5]=C2=A0 =C2=A01.01-2.00=C2=A0 =C2=A0sec=C2=A0 1.31 GBytes=C2=A0 11= .3 Gbits/sec=C2=A0 =C2=A027=C2=A0 =C2=A01.76 MBytes
[=C2=A0 5]=C2=A0 =C2=A02.00-3.00=C2=A0 =C2=A0sec=C2=A0 2.12 GBytes=C2=A0 18= .2 Gbits/sec=C2=A0 =C2=A034=C2=A0 =C2=A01.74 MBytes
[=C2=A0 5]=C2=A0 =C2=A03.00-4.00=C2=A0 =C2=A0sec=C2=A0 2.08 GBytes=C2=A0 17= .9 Gbits/sec=C2=A0 =C2=A085=C2=A0 =C2=A01.75 MBytes
[=C2=A0 5]=C2=A0 =C2=A04.00-5.00=C2=A0 =C2=A0sec=C2=A0 2.11 GBytes=C2=A0 18= .2 Gbits/sec=C2=A0 =C2=A037=C2=A0 =C2=A01.75 MBytes
[=C2=A0 5]=C2=A0 =C2=A05.00-6.00=C2=A0 =C2=A0sec=C2=A0 2.09 GBytes=C2=A0 18= .0 Gbits/sec=C2=A0 =C2=A060=C2=A0 =C2=A01.75 MBytes
[=C2=A0 5]=C2=A0 =C2=A06.00-7.00=C2=A0 =C2=A0sec=C2=A0 2.11 GBytes=C2=A0 18= .2 Gbits/sec=C2=A0 =C2=A010=C2=A0 =C2=A01.50 MBytes
[=C2=A0 5]=C2=A0 =C2=A07.00-8.00=C2=A0 =C2=A0sec=C2=A0 1.51 GBytes=C2=A0 13= .0 Gbits/sec=C2=A0 =C2=A027=C2=A0 =C2=A01.75 MBytes
[=C2=A0 5]=C2=A0 =C2=A08.00-9.00=C2=A0 =C2=A0sec=C2=A0 1.48 GBytes=C2=A0 12= .7 Gbits/sec=C2=A0 =C2=A075=C2=A0 =C2=A01.50 MBytes
[=C2=A0 5]=C2=A0 =C2=A09.00-10.00=C2=A0 sec=C2=A0 2.09 GBytes=C2=A0 17.9 Gb= its/sec=C2=A0 =C2=A052=C2=A0 =C2=A01.58 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Transfer=C2=A0 =C2= =A0 =C2=A0Bitrate=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Retr
[=C2=A0 5]=C2=A0 =C2=A00.00-10.00=C2=A0 sec=C2=A0 18.7 GBytes=C2=A0 16.0 Gb= its/sec=C2=A0 481=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0sender
[=C2=A0 5]=C2=A0 =C2=A00.00-10.00=C2=A0 sec=C2=A0 18.7 GBytes=C2=A0 16.0 Gb= its/sec=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 recei= ver
```



Beno=C3=AEt


------- Original Message -------
On Tuesday, May 23rd, 2023 at 23:15, Marko Zec <zec@fer.hr> wrote:


> On Tue, 23 May 2023 19:58:07 +0000
> Benoit Chesneau benoitc@enki-multimedia.eu wrote:
>
> > Hi all,
> >
> > I've created a jail using bastille and setup network. The mai= nin
> > terface is a 25Gbps nic and between hosts I get 24.6 Gbits/sec :<= br> >
>
> [...]
>
> > But between one host and the jail I only get 3.96 Gbits/sec
>
>
> [...]
>
> > Is there a way to increase the performance of the of the jail? Th= e
> > nice is a mellannox ConnectX-4 Lx, mce(4) .
>
>
> Modern NICs offload a lot of the protocol stack processing (checksum,<= br> > segmentation, and / or reassembly) from the CPU to dedicated silicon,<= br> > whereas inter-vnet traffic needs to be handled completely in software,=
> that's where the difference comes from.
>
> Perhaps we could gain some speed by abusing mbuf flags to skip RXCSUM<= br> > for epair traffic, maybe even skip and fake TXCSUM...
>
> Marko

--0000000000009b3c3c05fc793d88--