From nobody Mon Sep 16 09:08:52 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X6fH56hB8z5Wrs3 for ; Mon, 16 Sep 2024 09:09:05 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from mail-yw1-x1134.google.com (mail-yw1-x1134.google.com [IPv6:2607:f8b0:4864:20::1134]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X6fH51Nnlz4dDr for ; Mon, 16 Sep 2024 09:09:05 +0000 (UTC) (envelope-from dfr@rabson.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x1134.google.com with SMTP id 00721157ae682-6d5893cd721so21733917b3.0 for ; Mon, 16 Sep 2024 02:09:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rabson-org.20230601.gappssmtp.com; s=20230601; t=1726477744; x=1727082544; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=sveA9snzm0D96MwGpoG2ayvFpwBD/0X+XQIVa4p4Fso=; b=ooCdFd5T0jq9DaPFXlJO2PANJFPXkhxXvfEgh0iYNySnYWoTqUQgeuAh1ONgSLsLHP RUaWmxkqP3u6+d3KpG6Jg/ZiaNFebdsY7W5KjKNn2Kcndk+LIrRutXgSruK49roO6CzT bAVQ66ipHzVz7Lze2vBePjaGlydq0B7zGbJ+cN8UZYVBZss1L27ZrHdA+gVxN123BDWa NdONzO8ElL3nGABbCtFAfqhJfBQBFvVzUW72ZxsA1me2aO91gEdDQMcI4CJ42g7m9nJ4 9klDHBvZLfg5Q8mawpNNKfQs1/qcBjmkGZK2RNM1CpPVnTVlWe3zoVc+v2M4ZDBjL4q3 yt5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726477744; x=1727082544; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=sveA9snzm0D96MwGpoG2ayvFpwBD/0X+XQIVa4p4Fso=; b=a64Q3K5gGzJDlOBYQ1QcNMZmmh6lX3ovc+MJZQx1wUVCXQg7EQvlLHWxNrBJ0wlakm BpUGA/cwYeaASc+fPu7VAS2cXhZFXucL3AnFzRsEmNcFNsAPm7V3OEy3xPw4r5aQMiWD BuEk+TZ9H18CrgGd1Nz8+PniVyYntxTm0vx5/lnrNinVjhMnA90YcM0sukY7aeZm0YRn /GzY171yAj9INsyXQbJCjHW2KvLj67E5vrZfXvfd+I2TJFYJF5OyUYH2DefSlOgydhLk hPUlaBEibqvT8HPhyoQYdSCcVc4kPyCnz8+FMjvKqLP8clx35ayGahdhivLWycynpn64 CCNw== X-Forwarded-Encrypted: i=1; AJvYcCXwCy7aNz+GJu0JMkfN/tWleLxJDw4TIAtjClIAFA41g7LM5cU8fHATojI8KiSkRn9vL6CeClCiaUn7dw==@freebsd.org X-Gm-Message-State: AOJu0YwFzD3TKuLpwHNh0SILAj1bEALRiNsTWI5uI0bkmPS4M50/T6S0 OHeu0mrFEWDcIZbupkyYBwP3B78FGyUVYnAgXoXifTJKQkLoNeTuMNaNLjVGtsRawRe1ia9OxfY yZr2JKqItfZosaTUqfBdWPNQw/2bbDiRJ4SCoKw== X-Google-Smtp-Source: AGHT+IHwf2Qdttbk3jypJbPmFf4RRRWXUefY/mbqhiRN585d2T9AUU9rQ9h6rwXST3zqX5BzEgDDusSkcKBHmDsLjRI= X-Received: by 2002:a05:690c:2d86:b0:6b5:5042:2c9d with SMTP id 00721157ae682-6dbcc4c08fdmr65753477b3.24.1726477743946; Mon, 16 Sep 2024 02:09:03 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 References: <20240913100938.3eac55c9fbd976fa72d58bb5@gmail.com> <39B2C95D-1E4F-4133-8923-AD305DFA9435@longcount.org> <20240913155439.1e171a88bd01ce9b97558a90@gmail.com> <20240914112516.cfb31bae68ab90b83ca7ad4b@gmail.com> <20240915185654.b51cfec5aa2520e5b801cc87@gmail.com> In-Reply-To: <20240915185654.b51cfec5aa2520e5b801cc87@gmail.com> From: Doug Rabson Date: Mon, 16 Sep 2024 10:08:52 +0100 Message-ID: Subject: Re: Performance issues with vnet jails + epair + bridge To: Sad Clouds Cc: Zhenlei Huang , Mark Saad , FreeBSD Net Content-Type: multipart/alternative; boundary="000000000000a41d4d062238ec76" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4X6fH51Nnlz4dDr --000000000000a41d4d062238ec76 Content-Type: text/plain; charset="UTF-8" On Sun, 15 Sept 2024 at 18:56, Sad Clouds wrote: > On Sun, 15 Sep 2024 18:01:07 +0100 > Doug Rabson wrote: > > > I just did a throughput test with iperf3 client on a FreeBSD 14.1 host > with > > an intel 10GB nic connecting to an iperf3 server running in a vnet jail > on > > a truenas host (13.something) also with an intel 10GB nic and I get full > > 10GB throughput in this setup. In the past, I had to disable LRO on the > > truenas host for this to work properly. > > > > Doug. > > Hello Doug, can you please confirm that you are NOT using if_epair(4)? I > imagine you dedicate one of the Intel 10Gb ports to a jail. This is not > an option for some of us, so a virtual NIC of some sort is the only > option with vnet jails. Other people also mentioned that vnet by itself > is not an issue and your test confirms this, however I'm observing poor > scalability specifically with the epair virtual NIC. > > I will be trying netgraph when I have some more time. If there are > other alternatives to if_epair then I would be interested to learn > about them. > I am using epair on the server side of that test. On the truenas server, I have an if_bridge instance which has one vlan of the physical intel nic as member along with one side of an epair for each of the several jails running on the host. As I mentioned, disabling LRO on the physical nic was helpful in reaching this throughput. Doug. --000000000000a41d4d062238ec76 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sun, 15 Sept 2024 at 18:56, Sad Cl= ouds <cryintothebluesky@g= mail.com> wrote:
On Sun, 15 Sep 2024 18:01= :07 +0100
Doug Rabson <dfr@rab= son.org> wrote:

> I just did a throughput test with iperf3 client on a FreeBSD 14.1 host= with
> an intel 10GB nic connecting to an iperf3 server running in a vnet jai= l on
> a truenas host (13.something) also with an intel 10GB nic and I get fu= ll
> 10GB throughput in this setup. In the past, I had to disable LRO on th= e
> truenas host for this to work properly.
>
> Doug.

Hello Doug, can you please confirm that you are NOT using if_epair(4)? I imagine you dedicate one of the Intel 10Gb ports to a jail. This is not
an option for some of us, so a virtual NIC of some sort is the only
option with vnet jails. Other people also mentioned that vnet by itself
is not an issue and your test confirms this, however I'm observing poor=
scalability specifically with the epair virtual NIC.

I will be trying netgraph when I have some more time. If there are
other alternatives to if_epair then I would be interested to learn
about them.

I am using epair=C2=A0on th= e server side of that test. On the truenas server, I have an if_bridge inst= ance which has one vlan of the physical intel nic as member along with one = side of an epair=C2=A0for each of the several jails running on the host. As= I mentioned, disabling LRO on the physical nic was helpful in reaching thi= s throughput.

Doug.=C2=A0
--000000000000a41d4d062238ec76--