Date: Mon, 16 Sep 2024 10:08:52 +0100 From: Doug Rabson <dfr@rabson.org> To: Sad Clouds <cryintothebluesky@gmail.com> Cc: Zhenlei Huang <zlei@freebsd.org>, Mark Saad <nonesuch@longcount.org>, FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Performance issues with vnet jails + epair + bridge Message-ID: <CACA0VUikg6LPdpHZsnzDH3t_C6-k_svpfhteL6CpUyqSqm31=w@mail.gmail.com> In-Reply-To: <20240915185654.b51cfec5aa2520e5b801cc87@gmail.com> References: <20240913100938.3eac55c9fbd976fa72d58bb5@gmail.com> <39B2C95D-1E4F-4133-8923-AD305DFA9435@longcount.org> <20240913155439.1e171a88bd01ce9b97558a90@gmail.com> <A95066A8-F5FC-451B-85CE-C463952ABADE@FreeBSD.org> <20240914112516.cfb31bae68ab90b83ca7ad4b@gmail.com> <CACA0VUjE43FUTaqAtXTur-4akQybKytv-oc1rHxwoUUXM3VQ=Q@mail.gmail.com> <20240915185654.b51cfec5aa2520e5b801cc87@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000a41d4d062238ec76 Content-Type: text/plain; charset="UTF-8" On Sun, 15 Sept 2024 at 18:56, Sad Clouds <cryintothebluesky@gmail.com> wrote: > On Sun, 15 Sep 2024 18:01:07 +0100 > Doug Rabson <dfr@rabson.org> wrote: > > > I just did a throughput test with iperf3 client on a FreeBSD 14.1 host > with > > an intel 10GB nic connecting to an iperf3 server running in a vnet jail > on > > a truenas host (13.something) also with an intel 10GB nic and I get full > > 10GB throughput in this setup. In the past, I had to disable LRO on the > > truenas host for this to work properly. > > > > Doug. > > Hello Doug, can you please confirm that you are NOT using if_epair(4)? I > imagine you dedicate one of the Intel 10Gb ports to a jail. This is not > an option for some of us, so a virtual NIC of some sort is the only > option with vnet jails. Other people also mentioned that vnet by itself > is not an issue and your test confirms this, however I'm observing poor > scalability specifically with the epair virtual NIC. > > I will be trying netgraph when I have some more time. If there are > other alternatives to if_epair then I would be interested to learn > about them. > I am using epair on the server side of that test. On the truenas server, I have an if_bridge instance which has one vlan of the physical intel nic as member along with one side of an epair for each of the several jails running on the host. As I mentioned, disabling LRO on the physical nic was helpful in reaching this throughput. Doug. --000000000000a41d4d062238ec76 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Sun, 15 Sept 2024 at 18:56, Sad Cl= ouds <<a href=3D"mailto:cryintothebluesky@gmail.com">cryintothebluesky@g= mail.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D= "margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;bor= der-left-color:rgb(204,204,204);padding-left:1ex">On Sun, 15 Sep 2024 18:01= :07 +0100<br> Doug Rabson <<a href=3D"mailto:dfr@rabson.org" target=3D"_blank">dfr@rab= son.org</a>> wrote:<br> <br> > I just did a throughput test with iperf3 client on a FreeBSD 14.1 host= with<br> > an intel 10GB nic connecting to an iperf3 server running in a vnet jai= l on<br> > a truenas host (13.something) also with an intel 10GB nic and I get fu= ll<br> > 10GB throughput in this setup. In the past, I had to disable LRO on th= e<br> > truenas host for this to work properly.<br> > <br> > Doug.<br> <br> Hello Doug, can you please confirm that you are NOT using if_epair(4)? I<br= > imagine you dedicate one of the Intel 10Gb ports to a jail. This is not<br> an option for some of us, so a virtual NIC of some sort is the only<br> option with vnet jails. Other people also mentioned that vnet by itself<br> is not an issue and your test confirms this, however I'm observing poor= <br> scalability specifically with the epair virtual NIC.<br> <br> I will be trying netgraph when I have some more time. If there are<br> other alternatives to if_epair then I would be interested to learn<br> about them.<br></blockquote><div><br></div><div>I am using epair=C2=A0on th= e server side of that test. On the truenas server, I have an if_bridge inst= ance which has one vlan of the physical intel nic as member along with one = side of an epair=C2=A0for each of the several jails running on the host. As= I mentioned, disabling LRO on the physical nic was helpful in reaching thi= s throughput.</div><div><br></div><div>Doug.=C2=A0</div></div></div> --000000000000a41d4d062238ec76--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACA0VUikg6LPdpHZsnzDH3t_C6-k_svpfhteL6CpUyqSqm31=w>