Date: Wed, 24 May 2023 06:18:52 +0000 From: Benoit Chesneau <benoitc@enki-multimedia.eu> To: Marko Zec <zec@fer.hr> Cc: "freebsd-net@FreeBSD.org" <freebsd-net@freebsd.org> Subject: Re: how to increase the vnet speed? Message-ID: <N-GhfYpIpKX6Wn1AneSq4X8yM0WO1KPW-E0H0sOW-HznJLdzK3gFSQeVHh2lugmzBlVWUaRZg2ikg8uo0TIAf-r_rGdSCahSOpKJ5a8GMN0=@enki-multimedia.eu> In-Reply-To: <20230523231533.32c1d022@x25> References: <TbR2mwTT9IpAlR-ds87h3qzhRbj1mVmWUGVVsmxQU2qq6Z1Bv52fsaB5LqUxaZT9ooHdXo2VIhcYQu2zkT7eVMxwbbfxIv6q2JNHp7l-ydM=@enki-multimedia.eu> <20230523231533.32c1d022@x25>
next in thread | previous in thread | raw e-mail | index | archive | help
Sorry, I thought I posted it but it's a bridge: ``` vlan200: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metri= c 0 mtu 9000 =09options=3D1c280401<RXCSUM,LRO,LINKSTATE,RXCSUM_IPV6,NOMAP,TXTLS4,TXTLS6> =09ether 9c:dc:71:4c:84:f0 =09groups: vlan =09vlan: 200 vlanproto: 802.1q vlanpcp: 0 parent interface: mce0 =09media: Ethernet 25GBase-SR <full-duplex,rxpause,txpause> =09status: active =09nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> vlan200bridge: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric = 0 mtu 9000 =09ether 58:9c:fc:10:ff:95 =09id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 =09maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 =09root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 =09member: e0a_bastille0 flags=3D143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> =09 ifmaxaddr 0 port 10 priority 128 path cost 2000 =09member: vlan200 flags=3D143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> =09 ifmaxaddr 0 port 8 priority 128 path cost 800 =09groups: bridge =09nd6 options=3D9<PERFORMNUD,IFDISABLED> e0a_bastille0: flags=3D8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>= metric 0 mtu 9000 =09description: vnet host interface for Bastille jail testing =09options=3D8<VLAN_MTU> =09ether 02:20:98:4c:84:f0 =09hwaddr 02:68:8a:24:67:0a =09groups: epair =09media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) =09status: active =09nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> ``` After relaunching the machine and removed filtering: ``` net.link.bridge.pfil_bridge=3D0 net.link.bridge.pfil_onlyip=3D0 net.link.bridge.pfil_member=3D0 ``` I get better results. Still not dull speed but since it's in a bridge seems= normal. Unsure what was the issue... ``` [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.01 sec 1.75 GBytes 14.8 Gbits/sec 74 936 KBytes [ 5] 1.01-2.00 sec 1.31 GBytes 11.3 Gbits/sec 27 1.76 MBytes [ 5] 2.00-3.00 sec 2.12 GBytes 18.2 Gbits/sec 34 1.74 MBytes [ 5] 3.00-4.00 sec 2.08 GBytes 17.9 Gbits/sec 85 1.75 MBytes [ 5] 4.00-5.00 sec 2.11 GBytes 18.2 Gbits/sec 37 1.75 MBytes [ 5] 5.00-6.00 sec 2.09 GBytes 18.0 Gbits/sec 60 1.75 MBytes [ 5] 6.00-7.00 sec 2.11 GBytes 18.2 Gbits/sec 10 1.50 MBytes [ 5] 7.00-8.00 sec 1.51 GBytes 13.0 Gbits/sec 27 1.75 MBytes [ 5] 8.00-9.00 sec 1.48 GBytes 12.7 Gbits/sec 75 1.50 MBytes [ 5] 9.00-10.00 sec 2.09 GBytes 17.9 Gbits/sec 52 1.58 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 18.7 GBytes 16.0 Gbits/sec 481 sende= r [ 5] 0.00-10.00 sec 18.7 GBytes 16.0 Gbits/sec recei= ver ``` Beno=C3=AEt ------- Original Message ------- On Tuesday, May 23rd, 2023 at 23:15, Marko Zec <zec@fer.hr> wrote: > On Tue, 23 May 2023 19:58:07 +0000 > Benoit Chesneau benoitc@enki-multimedia.eu wrote: >=20 > > Hi all, > >=20 > > I've created a jail using bastille and setup network. The mainin > > terface is a 25Gbps nic and between hosts I get 24.6 Gbits/sec : >=20 >=20 > [...] >=20 > > But between one host and the jail I only get 3.96 Gbits/sec >=20 >=20 > [...] >=20 > > Is there a way to increase the performance of the of the jail? The > > nice is a mellannox ConnectX-4 Lx, mce(4) . >=20 >=20 > Modern NICs offload a lot of the protocol stack processing (checksum, > segmentation, and / or reassembly) from the CPU to dedicated silicon, > whereas inter-vnet traffic needs to be handled completely in software, > that's where the difference comes from. >=20 > Perhaps we could gain some speed by abusing mbuf flags to skip RXCSUM > for epair traffic, maybe even skip and fake TXCSUM... >=20 > Marko
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?N-GhfYpIpKX6Wn1AneSq4X8yM0WO1KPW-E0H0sOW-HznJLdzK3gFSQeVHh2lugmzBlVWUaRZg2ikg8uo0TIAf-r_rGdSCahSOpKJ5a8GMN0=>