From nobody Wed May 24 06:18:52 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QR1Gv05Fvz4CGK3 for ; Wed, 24 May 2023 06:19:03 +0000 (UTC) (envelope-from benoitc@enki-multimedia.eu) Received: from mail-4022.proton.ch (mail-4022.proton.ch [185.70.40.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QR1Gt4dJSz3nx7 for ; Wed, 24 May 2023 06:19:02 +0000 (UTC) (envelope-from benoitc@enki-multimedia.eu) Authentication-Results: mx1.freebsd.org; none Date: Wed, 24 May 2023 06:18:52 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enki-multimedia.eu; s=protonmail3; t=1684909138; x=1685168338; bh=8cxvXkB4a8KA0+vRA8hCVY+rxg16bv0imLiV3vMg7nQ=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=WHW9IIHopnAglq4NDyV4CBNWrN6HZH33u/nbYymoFTnMb2XeVL4AoJt4PSLKJ+L31 wgTKtOdkKsivZ5S4kNdggIgrRGT5CpfPsnykfAMJQh6m2vbGLGHJGKyGm3l5JG3NNS DOeIWU6gO1nOaK5Cp1AuC2IasaUk0T98GCmkgJbd3C0PEVsn94Y8Qw/IeKCUNpq1yJ L0MWI7fHRjQtvohMMOlnpVxrT/ozYkoXukok6v5FLtJz0VYpzNmfpmShls/YMMShyr y1Sz3fJzSCLlbZchouWklD+4XILWCTB/YiFdzvomT+AwJ83+xlJsHYAFUiCzF7Pc4s BubCNAaPXN6GQ== To: Marko Zec From: Benoit Chesneau Cc: "freebsd-net@FreeBSD.org" Subject: Re: how to increase the vnet speed? Message-ID: In-Reply-To: <20230523231533.32c1d022@x25> References: <20230523231533.32c1d022@x25> Feedback-ID: 9066678:user:proton List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4QR1Gt4dJSz3nx7 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:62371, ipnet:185.70.40.0/24, country:CH] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N Sorry, I thought I posted it but it's a bridge: ``` vlan200: flags=3D8943 metri= c 0 mtu 9000 =09options=3D1c280401 =09ether 9c:dc:71:4c:84:f0 =09groups: vlan =09vlan: 200 vlanproto: 802.1q vlanpcp: 0 parent interface: mce0 =09media: Ethernet 25GBase-SR =09status: active =09nd6 options=3D29 vlan200bridge: flags=3D8843 metric = 0 mtu 9000 =09ether 58:9c:fc:10:ff:95 =09id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 =09maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 =09root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 =09member: e0a_bastille0 flags=3D143 =09 ifmaxaddr 0 port 10 priority 128 path cost 2000 =09member: vlan200 flags=3D143 =09 ifmaxaddr 0 port 8 priority 128 path cost 800 =09groups: bridge =09nd6 options=3D9 e0a_bastille0: flags=3D8963= metric 0 mtu 9000 =09description: vnet host interface for Bastille jail testing =09options=3D8 =09ether 02:20:98:4c:84:f0 =09hwaddr 02:68:8a:24:67:0a =09groups: epair =09media: Ethernet 10Gbase-T (10Gbase-T ) =09status: active =09nd6 options=3D29 ``` After relaunching the machine and removed filtering: ``` net.link.bridge.pfil_bridge=3D0 net.link.bridge.pfil_onlyip=3D0 net.link.bridge.pfil_member=3D0 ``` I get better results. Still not dull speed but since it's in a bridge seems= normal. Unsure what was the issue... ``` [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.01 sec 1.75 GBytes 14.8 Gbits/sec 74 936 KBytes [ 5] 1.01-2.00 sec 1.31 GBytes 11.3 Gbits/sec 27 1.76 MBytes [ 5] 2.00-3.00 sec 2.12 GBytes 18.2 Gbits/sec 34 1.74 MBytes [ 5] 3.00-4.00 sec 2.08 GBytes 17.9 Gbits/sec 85 1.75 MBytes [ 5] 4.00-5.00 sec 2.11 GBytes 18.2 Gbits/sec 37 1.75 MBytes [ 5] 5.00-6.00 sec 2.09 GBytes 18.0 Gbits/sec 60 1.75 MBytes [ 5] 6.00-7.00 sec 2.11 GBytes 18.2 Gbits/sec 10 1.50 MBytes [ 5] 7.00-8.00 sec 1.51 GBytes 13.0 Gbits/sec 27 1.75 MBytes [ 5] 8.00-9.00 sec 1.48 GBytes 12.7 Gbits/sec 75 1.50 MBytes [ 5] 9.00-10.00 sec 2.09 GBytes 17.9 Gbits/sec 52 1.58 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 18.7 GBytes 16.0 Gbits/sec 481 sende= r [ 5] 0.00-10.00 sec 18.7 GBytes 16.0 Gbits/sec recei= ver ``` Beno=C3=AEt ------- Original Message ------- On Tuesday, May 23rd, 2023 at 23:15, Marko Zec wrote: > On Tue, 23 May 2023 19:58:07 +0000 > Benoit Chesneau benoitc@enki-multimedia.eu wrote: >=20 > > Hi all, > >=20 > > I've created a jail using bastille and setup network. The mainin > > terface is a 25Gbps nic and between hosts I get 24.6 Gbits/sec : >=20 >=20 > [...] >=20 > > But between one host and the jail I only get 3.96 Gbits/sec >=20 >=20 > [...] >=20 > > Is there a way to increase the performance of the of the jail? The > > nice is a mellannox ConnectX-4 Lx, mce(4) . >=20 >=20 > Modern NICs offload a lot of the protocol stack processing (checksum, > segmentation, and / or reassembly) from the CPU to dedicated silicon, > whereas inter-vnet traffic needs to be handled completely in software, > that's where the difference comes from. >=20 > Perhaps we could gain some speed by abusing mbuf flags to skip RXCSUM > for epair traffic, maybe even skip and fake TXCSUM... >=20 > Marko