Date: Wed, 24 May 2023 19:55:26 -0400 From: Cheng Cui <cc@freebsd.org> To: Benoit Chesneau <benoitc@enki-multimedia.eu> Cc: Marko Zec <zec@fer.hr>, "freebsd-net@FreeBSD.org" <freebsd-net@freebsd.org> Subject: Re: how to increase the vnet speed? Message-ID: <CAGaXuiKxQTZae9dsF1Wm%2BdQPu6qZVHPosFUU4_xKvQv3=okJ9g@mail.gmail.com> In-Reply-To: <N-GhfYpIpKX6Wn1AneSq4X8yM0WO1KPW-E0H0sOW-HznJLdzK3gFSQeVHh2lugmzBlVWUaRZg2ikg8uo0TIAf-r_rGdSCahSOpKJ5a8GMN0=@enki-multimedia.eu> References: <TbR2mwTT9IpAlR-ds87h3qzhRbj1mVmWUGVVsmxQU2qq6Z1Bv52fsaB5LqUxaZT9ooHdXo2VIhcYQu2zkT7eVMxwbbfxIv6q2JNHp7l-ydM=@enki-multimedia.eu> <20230523231533.32c1d022@x25> <N-GhfYpIpKX6Wn1AneSq4X8yM0WO1KPW-E0H0sOW-HznJLdzK3gFSQeVHh2lugmzBlVWUaRZg2ikg8uo0TIAf-r_rGdSCahSOpKJ5a8GMN0=@enki-multimedia.eu>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000009b3c3c05fc793d88 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable >From TCP's point of view, your iperf result shows "Retr" as retransmissions. Well, that tells packet drops or something else which makes TCP's congestion control into playing throttling the throughput to a certain degree. Try a different TCP congestion control. If the default is CUBIC, try to use NewReno. I don't know your FreeBSD version. So if it is FreeBSD14-current, I would like to see the difference as there is improvement discussion on CUBIC. cc@fbsd ~$ sysctl net.inet.tcp.cc net.inet.tcp.cc.newreno.beta_ecn: 80 net.inet.tcp.cc.newreno.beta: 50 net.inet.tcp.cc.abe_frlossreduce: 0 net.inet.tcp.cc.abe: 0 net.inet.tcp.cc.available: newreno net.inet.tcp.cc.algorithm: newreno cc@fbsd ~$ sysctl net.inet.tcp.cc.available net.inet.tcp.cc.available: newreno cc@fbsd ~$ sudo kldload cc_cubic << load TCP congestion control module cc_cubic cc@fbsd ~$ sysctl net.inet.tcp.cc.available net.inet.tcp.cc.available: newreno, cubic cc@fbsd ~$ sudo sysctl net.inet.tcp.cc.algorithm=3Dcubic net.inet.tcp.cc.algorithm: newreno -> cubic cc@fbsd ~$ sudo sysctl net.inet.tcp.cc.algorithm net.inet.tcp.cc.algorithm: cubic cc@fbsd ~$ uname -a FreeBSD fbsd.cc.home 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64 cc@fbsd ~$ Best Regards, Cheng Cui On Wed, May 24, 2023 at 2:19=E2=80=AFAM Benoit Chesneau <benoitc@enki-multi= media.eu> wrote: > Sorry, I thought I posted it but it's a bridge: > > ``` > vlan200: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> met= ric > 0 mtu 9000 > > options=3D1c280401<RXCSUM,LRO,LINKSTATE,RXCSUM_IPV6,NOMAP,TXTLS4,TXTLS6> > ether 9c:dc:71:4c:84:f0 > groups: vlan > vlan: 200 vlanproto: 802.1q vlanpcp: 0 parent interface: mce0 > media: Ethernet 25GBase-SR <full-duplex,rxpause,txpause> > status: active > nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > vlan200bridge: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metri= c 0 > mtu 9000 > ether 58:9c:fc:10:ff:95 > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: e0a_bastille0 flags=3D143<LEARNING,DISCOVER,AUTOEDGE,AUTO= PTP> > ifmaxaddr 0 port 10 priority 128 path cost 2000 > member: vlan200 flags=3D143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> > ifmaxaddr 0 port 8 priority 128 path cost 800 > groups: bridge > nd6 options=3D9<PERFORMNUD,IFDISABLED> > e0a_bastille0: flags=3D8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAS= T> > metric 0 mtu 9000 > description: vnet host interface for Bastille jail testing > options=3D8<VLAN_MTU> > ether 02:20:98:4c:84:f0 > hwaddr 02:68:8a:24:67:0a > groups: epair > media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) > status: active > nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > ``` > > After relaunching the machine and removed filtering: > > ``` > net.link.bridge.pfil_bridge=3D0 > net.link.bridge.pfil_onlyip=3D0 > net.link.bridge.pfil_member=3D0 > ``` > > I get better results. Still not dull speed but since it's in a bridge > seems normal. Unsure what was the issue... > > ``` > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.01 sec 1.75 GBytes 14.8 Gbits/sec 74 936 KBytes > [ 5] 1.01-2.00 sec 1.31 GBytes 11.3 Gbits/sec 27 1.76 MBytes > [ 5] 2.00-3.00 sec 2.12 GBytes 18.2 Gbits/sec 34 1.74 MBytes > [ 5] 3.00-4.00 sec 2.08 GBytes 17.9 Gbits/sec 85 1.75 MBytes > [ 5] 4.00-5.00 sec 2.11 GBytes 18.2 Gbits/sec 37 1.75 MBytes > [ 5] 5.00-6.00 sec 2.09 GBytes 18.0 Gbits/sec 60 1.75 MBytes > [ 5] 6.00-7.00 sec 2.11 GBytes 18.2 Gbits/sec 10 1.50 MBytes > [ 5] 7.00-8.00 sec 1.51 GBytes 13.0 Gbits/sec 27 1.75 MBytes > [ 5] 8.00-9.00 sec 1.48 GBytes 12.7 Gbits/sec 75 1.50 MBytes > [ 5] 9.00-10.00 sec 2.09 GBytes 17.9 Gbits/sec 52 1.58 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 18.7 GBytes 16.0 Gbits/sec 481 > sender > [ 5] 0.00-10.00 sec 18.7 GBytes 16.0 Gbits/sec > receiver > ``` > > > > Beno=C3=AEt > > > ------- Original Message ------- > On Tuesday, May 23rd, 2023 at 23:15, Marko Zec <zec@fer.hr> wrote: > > > > On Tue, 23 May 2023 19:58:07 +0000 > > Benoit Chesneau benoitc@enki-multimedia.eu wrote: > > > > > Hi all, > > > > > > I've created a jail using bastille and setup network. The mainin > > > terface is a 25Gbps nic and between hosts I get 24.6 Gbits/sec : > > > > > > [...] > > > > > But between one host and the jail I only get 3.96 Gbits/sec > > > > > > [...] > > > > > Is there a way to increase the performance of the of the jail? The > > > nice is a mellannox ConnectX-4 Lx, mce(4) . > > > > > > Modern NICs offload a lot of the protocol stack processing (checksum, > > segmentation, and / or reassembly) from the CPU to dedicated silicon, > > whereas inter-vnet traffic needs to be handled completely in software, > > that's where the difference comes from. > > > > Perhaps we could gain some speed by abusing mbuf flags to skip RXCSUM > > for epair traffic, maybe even skip and fake TXCSUM... > > > > Marko > > --0000000000009b3c3c05fc793d88 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>From TCP's point of view, your iperf result shows= "Retr" as retransmissions. Well, that tells packet drops or some= thing else which makes TCP's congestion control into playing throttling= the throughput to a certain degree.</div><div><br></div><div>Try a differe= nt TCP congestion control. If the default is CUBIC, try to use NewReno. I d= on't know your FreeBSD version. So if it is FreeBSD14-current, I would = like to see the difference as there is improvement discussion on CUBIC.<br>= </div><div><br></div><div>cc@fbsd ~$ sysctl <a href=3D"http://net.inet.tcp.= cc">net.inet.tcp.cc</a><br>net.inet.tcp.cc.newreno.beta_ecn: 80<br>net.inet= .tcp.cc.newreno.beta: 50<br>net.inet.tcp.cc.abe_frlossreduce: 0<br>net.inet= .tcp.cc.abe: 0<br>net.inet.tcp.cc.available: newreno<br>net.inet.tcp.cc.alg= orithm: newreno</div><div>cc@fbsd ~$ sysctl net.inet.tcp.cc.available<br>ne= t.inet.tcp.cc.available: newreno<br>cc@fbsd ~$ sudo kldload cc_cubic=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 << load TCP= congestion control module cc_cubic<br>cc@fbsd ~$ sysctl net.inet.tcp.cc.av= ailable<br>net.inet.tcp.cc.available: newreno, cubic<br>cc@fbsd ~$ sudo sys= ctl net.inet.tcp.cc.algorithm=3Dcubic<br>net.inet.tcp.cc.algorithm: newreno= -> cubic<br>=C2=A0=C2=A0 =C2=A0 <br>cc@fbsd ~$ sudo sysctl net.inet.tcp= .cc.algorithm =C2=A0 =C2=A0 =C2=A0<br>net.inet.tcp.cc.algorithm: cubic<br>c= c@fbsd ~$ uname -a<br>FreeBSD fbsd.cc.home 13.1-RELEASE FreeBSD 13.1-RELEAS= E releng/13.1-n250148-fc952ac2212 GENERIC amd64<br>cc@fbsd ~$ </div><div di= r=3D"ltr"><br clear=3D"all"><div><div dir=3D"ltr" class=3D"gmail_signature"= ><div dir=3D"ltr"><div><br></div>Best Regards,<div>Cheng Cui</div></div></d= iv></div><br></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D= "gmail_attr">On Wed, May 24, 2023 at 2:19=E2=80=AFAM Benoit Chesneau <<a= href=3D"mailto:benoitc@enki-multimedia.eu">benoitc@enki-multimedia.eu</a>&= gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0= px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Sorry= , I thought I posted it but it's a bridge:<br> <br> ```<br> vlan200: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>= metric 0 mtu 9000<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 options=3D1c280401<RXCSUM,LRO,LINKSTATE,RXCS= UM_IPV6,NOMAP,TXTLS4,TXTLS6><br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 ether 9c:dc:71:4c:84:f0<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 groups: vlan<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 vlan: 200 vlanproto: 802.1q vlanpcp: 0 parent i= nterface: mce0<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 media: Ethernet 25GBase-SR <full-duplex,rxpa= use,txpause><br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 status: active<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_= LINKLOCAL><br> vlan200bridge: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> m= etric 0 mtu 9000<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 ether 58:9c:fc:10:ff:95<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 id 00:00:00:00:00:00 priority 32768 hellotime 2= fwddelay 15<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 maxage 20 holdcnt 6 proto rstp maxaddr 2000 tim= eout 1200<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 root id 00:00:00:00:00:00 priority 32768 ifcost= 0 port 0<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 member: e0a_bastille0 flags=3D143<LEARNING,D= ISCOVER,AUTOEDGE,AUTOPTP><br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ifmaxaddr 0 port 10= priority 128 path cost 2000<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 member: vlan200 flags=3D143<LEARNING,DISCOVE= R,AUTOEDGE,AUTOPTP><br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ifmaxaddr 0 port 8 = priority 128 path cost 800<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 groups: bridge<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 nd6 options=3D9<PERFORMNUD,IFDISABLED><br= > e0a_bastille0: flags=3D8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICA= ST> metric 0 mtu 9000<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 description: vnet host interface for Bastille j= ail testing<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 options=3D8<VLAN_MTU><br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 ether 02:20:98:4c:84:f0<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 hwaddr 02:68:8a:24:67:0a<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 groups: epair<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 media: Ethernet 10Gbase-T (10Gbase-T <full-d= uplex>)<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 status: active<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_= LINKLOCAL><br> ```<br> <br> After relaunching the machine and removed filtering:<br> <br> ```<br> net.link.bridge.pfil_bridge=3D0<br> net.link.bridge.pfil_onlyip=3D0<br> net.link.bridge.pfil_member=3D0<br> ```<br> <br> I get better results. Still not dull speed but since it's in a bridge s= eems normal. Unsure what was the issue...<br> <br> ```<br> [ ID] Interval=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Transfer=C2=A0 =C2= =A0 =C2=A0Bitrate=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Retr=C2=A0 Cwnd<br> [=C2=A0 5]=C2=A0 =C2=A00.00-1.01=C2=A0 =C2=A0sec=C2=A0 1.75 GBytes=C2=A0 14= .8 Gbits/sec=C2=A0 =C2=A074=C2=A0 =C2=A0 936 KBytes<br> [=C2=A0 5]=C2=A0 =C2=A01.01-2.00=C2=A0 =C2=A0sec=C2=A0 1.31 GBytes=C2=A0 11= .3 Gbits/sec=C2=A0 =C2=A027=C2=A0 =C2=A01.76 MBytes<br> [=C2=A0 5]=C2=A0 =C2=A02.00-3.00=C2=A0 =C2=A0sec=C2=A0 2.12 GBytes=C2=A0 18= .2 Gbits/sec=C2=A0 =C2=A034=C2=A0 =C2=A01.74 MBytes<br> [=C2=A0 5]=C2=A0 =C2=A03.00-4.00=C2=A0 =C2=A0sec=C2=A0 2.08 GBytes=C2=A0 17= .9 Gbits/sec=C2=A0 =C2=A085=C2=A0 =C2=A01.75 MBytes<br> [=C2=A0 5]=C2=A0 =C2=A04.00-5.00=C2=A0 =C2=A0sec=C2=A0 2.11 GBytes=C2=A0 18= .2 Gbits/sec=C2=A0 =C2=A037=C2=A0 =C2=A01.75 MBytes<br> [=C2=A0 5]=C2=A0 =C2=A05.00-6.00=C2=A0 =C2=A0sec=C2=A0 2.09 GBytes=C2=A0 18= .0 Gbits/sec=C2=A0 =C2=A060=C2=A0 =C2=A01.75 MBytes<br> [=C2=A0 5]=C2=A0 =C2=A06.00-7.00=C2=A0 =C2=A0sec=C2=A0 2.11 GBytes=C2=A0 18= .2 Gbits/sec=C2=A0 =C2=A010=C2=A0 =C2=A01.50 MBytes<br> [=C2=A0 5]=C2=A0 =C2=A07.00-8.00=C2=A0 =C2=A0sec=C2=A0 1.51 GBytes=C2=A0 13= .0 Gbits/sec=C2=A0 =C2=A027=C2=A0 =C2=A01.75 MBytes<br> [=C2=A0 5]=C2=A0 =C2=A08.00-9.00=C2=A0 =C2=A0sec=C2=A0 1.48 GBytes=C2=A0 12= .7 Gbits/sec=C2=A0 =C2=A075=C2=A0 =C2=A01.50 MBytes<br> [=C2=A0 5]=C2=A0 =C2=A09.00-10.00=C2=A0 sec=C2=A0 2.09 GBytes=C2=A0 17.9 Gb= its/sec=C2=A0 =C2=A052=C2=A0 =C2=A01.58 MBytes<br> - - - - - - - - - - - - - - - - - - - - - - - - -<br> [ ID] Interval=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Transfer=C2=A0 =C2= =A0 =C2=A0Bitrate=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Retr<br> [=C2=A0 5]=C2=A0 =C2=A00.00-10.00=C2=A0 sec=C2=A0 18.7 GBytes=C2=A0 16.0 Gb= its/sec=C2=A0 481=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0sender<br> [=C2=A0 5]=C2=A0 =C2=A00.00-10.00=C2=A0 sec=C2=A0 18.7 GBytes=C2=A0 16.0 Gb= its/sec=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 recei= ver<br> ```<br> <br> <br> <br> Beno=C3=AEt<br> <br> <br> ------- Original Message -------<br> On Tuesday, May 23rd, 2023 at 23:15, Marko Zec <<a href=3D"mailto:zec@fe= r.hr" target=3D"_blank">zec@fer.hr</a>> wrote:<br> <br> <br> > On Tue, 23 May 2023 19:58:07 +0000<br> > Benoit Chesneau <a href=3D"mailto:benoitc@enki-multimedia.eu" target= =3D"_blank">benoitc@enki-multimedia.eu</a> wrote:<br> > <br> > > Hi all,<br> > > <br> > > I've created a jail using bastille and setup network. The mai= nin<br> > > terface is a 25Gbps nic and between hosts I get 24.6 Gbits/sec :<= br> > <br> > <br> > [...]<br> > <br> > > But between one host and the jail I only get 3.96 Gbits/sec<br> > <br> > <br> > [...]<br> > <br> > > Is there a way to increase the performance of the of the jail? Th= e<br> > > nice is a mellannox ConnectX-4 Lx, mce(4) .<br> > <br> > <br> > Modern NICs offload a lot of the protocol stack processing (checksum,<= br> > segmentation, and / or reassembly) from the CPU to dedicated silicon,<= br> > whereas inter-vnet traffic needs to be handled completely in software,= <br> > that's where the difference comes from.<br> > <br> > Perhaps we could gain some speed by abusing mbuf flags to skip RXCSUM<= br> > for epair traffic, maybe even skip and fake TXCSUM...<br> > <br> > Marko<br> <br> </blockquote></div></div> --0000000000009b3c3c05fc793d88--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGaXuiKxQTZae9dsF1Wm%2BdQPu6qZVHPosFUU4_xKvQv3=okJ9g>