Date: Sun, 15 Sep 2024 18:01:07 +0100 From: Doug Rabson <dfr@rabson.org> To: Sad Clouds <cryintothebluesky@gmail.com> Cc: Zhenlei Huang <zlei@freebsd.org>, Mark Saad <nonesuch@longcount.org>, FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Performance issues with vnet jails + epair + bridge Message-ID: <CACA0VUjE43FUTaqAtXTur-4akQybKytv-oc1rHxwoUUXM3VQ=Q@mail.gmail.com> In-Reply-To: <20240914112516.cfb31bae68ab90b83ca7ad4b@gmail.com> References: <20240913100938.3eac55c9fbd976fa72d58bb5@gmail.com> <39B2C95D-1E4F-4133-8923-AD305DFA9435@longcount.org> <20240913155439.1e171a88bd01ce9b97558a90@gmail.com> <A95066A8-F5FC-451B-85CE-C463952ABADE@FreeBSD.org> <20240914112516.cfb31bae68ab90b83ca7ad4b@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] I just did a throughput test with iperf3 client on a FreeBSD 14.1 host with an intel 10GB nic connecting to an iperf3 server running in a vnet jail on a truenas host (13.something) also with an intel 10GB nic and I get full 10GB throughput in this setup. In the past, I had to disable LRO on the truenas host for this to work properly. Doug. On Sat, 14 Sept 2024 at 11:25, Sad Clouds <cryintothebluesky@gmail.com> wrote: > On Sat, 14 Sep 2024 10:45:03 +0800 > Zhenlei Huang <zlei@FreeBSD.org> wrote: > > > The overhead of vnet jail should be neglectable, compared to legacy jail > > or no-jail. Bare in mind when VIMAGE option is enabled, there is a > default > > vnet 0. It is not visible via jls and can not be destroyed. So when you > see > > bottlenecks, for example this case, it is mostly caused by other > components > > such as if_epair, but not the vnet jail itself. > > Perhaps this needs a correction - the vnet itself may be OK, but due to > a single physical NIC on this appliance, I cannot use vnet jails > without virtualised devices like if_epair(4) and if_bridge(4). I think > there may be other scalability bottlenecks. > > I have a similar setup on Solaris > > Here devel is a Solaris zone with exclusive IP configuration, which I > think may be similar to FreeBSD vnet. It has a virtual NIC devel/net0 > which operates over the physical NIC also called net0 in the global > zone: > > $ dladm > LINK CLASS MTU STATE OVER > net0 phys 1500 up -- > net1 phys 1500 up -- > net2 phys 1500 up -- > net3 phys 1500 up -- > pkgsrc/net0 vnic 1500 up net0 > devel/net0 vnic 1500 up net0 > > If I run TCP bulk data benchmark with 64 concurrent threads, 32 > threads with server process in the global zone and 32 threads with > client process in the devel zone, then the system evenly spreads the > load across all CPU cores and none of them are sitting idle: > > $ mpstat -A core 1 > COR minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys st > idl sze > 0 0 0 2262 2561 4 4744 2085 209 7271 0 747842 272 528 > 0 0 8 > 1 0 0 3187 4209 2 9102 3768 514 10605 0 597012 221 579 > 0 0 8 > 2 0 0 2091 3251 7 6768 2884 307 9557 0 658124 244 556 > 0 0 8 > 3 0 0 1745 1786 16 3494 1520 176 8847 0 746373 273 527 > 0 0 8 > 4 0 0 2797 2767 3 5908 2414 371 7849 0 692873 253 547 > 0 0 8 > 5 0 0 2782 2359 5 4857 2012 324 9431 0 684840 251 549 > 0 0 8 > 6 0 0 4324 4133 0 9138 3592 538 12525 0 516342 191 609 > 0 0 8 > 7 0 0 2180 3249 0 6960 2926 321 8825 0 697861 257 543 > 0 0 8 > > With FreeBSD I tried "options RSS" and increasing "net.isr.maxthreads" > however this resulted in some really flaky kernel behavior. So I'm > thinking that if_epair(4) may be OK for some low-bandwidth use cases, > i.e. testing firewall rules, etc, but not suitable for things like > file/object storage servers, etc. > > [-- Attachment #2 --] <div dir="ltr">I just did a throughput test with iperf3 client on a FreeBSD 14.1 host with an intel 10GB nic connecting to an iperf3 server running in a vnet jail on a truenas host (13.something) also with an intel 10GB nic and I get full 10GB throughput in this setup. In the past, I had to disable LRO on the truenas host for this to work properly.<div><br></div><div>Doug.</div><div><br><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, 14 Sept 2024 at 11:25, Sad Clouds <<a href="mailto:cryintothebluesky@gmail.com">cryintothebluesky@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">On Sat, 14 Sep 2024 10:45:03 +0800<br> Zhenlei Huang <zlei@FreeBSD.org> wrote:<br> <br> > The overhead of vnet jail should be neglectable, compared to legacy jail<br> > or no-jail. Bare in mind when VIMAGE option is enabled, there is a default<br> > vnet 0. It is not visible via jls and can not be destroyed. So when you see<br> > bottlenecks, for example this case, it is mostly caused by other components<br> > such as if_epair, but not the vnet jail itself.<br> <br> Perhaps this needs a correction - the vnet itself may be OK, but due to<br> a single physical NIC on this appliance, I cannot use vnet jails<br> without virtualised devices like if_epair(4) and if_bridge(4). I think<br> there may be other scalability bottlenecks.<br> <br> I have a similar setup on Solaris<br> <br> Here devel is a Solaris zone with exclusive IP configuration, which I<br> think may be similar to FreeBSD vnet. It has a virtual NIC devel/net0<br> which operates over the physical NIC also called net0 in the global<br> zone:<br> <br> $ dladm<br> LINK CLASS MTU STATE OVER<br> net0 phys 1500 up --<br> net1 phys 1500 up --<br> net2 phys 1500 up --<br> net3 phys 1500 up --<br> pkgsrc/net0 vnic 1500 up net0<br> devel/net0 vnic 1500 up net0<br> <br> If I run TCP bulk data benchmark with 64 concurrent threads, 32<br> threads with server process in the global zone and 32 threads with<br> client process in the devel zone, then the system evenly spreads the<br> load across all CPU cores and none of them are sitting idle:<br> <br> $ mpstat -A core 1<br> COR minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys st idl sze<br> 0 0 0 2262 2561 4 4744 2085 209 7271 0 747842 272 528 0 0 8<br> 1 0 0 3187 4209 2 9102 3768 514 10605 0 597012 221 579 0 0 8<br> 2 0 0 2091 3251 7 6768 2884 307 9557 0 658124 244 556 0 0 8<br> 3 0 0 1745 1786 16 3494 1520 176 8847 0 746373 273 527 0 0 8<br> 4 0 0 2797 2767 3 5908 2414 371 7849 0 692873 253 547 0 0 8<br> 5 0 0 2782 2359 5 4857 2012 324 9431 0 684840 251 549 0 0 8<br> 6 0 0 4324 4133 0 9138 3592 538 12525 0 516342 191 609 0 0 8<br> 7 0 0 2180 3249 0 6960 2926 321 8825 0 697861 257 543 0 0 8<br> <br> With FreeBSD I tried "options RSS" and increasing "net.isr.maxthreads"<br> however this resulted in some really flaky kernel behavior. So I'm<br> thinking that if_epair(4) may be OK for some low-bandwidth use cases,<br> i.e. testing firewall rules, etc, but not suitable for things like<br> file/object storage servers, etc.<br> <br> </blockquote></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACA0VUjE43FUTaqAtXTur-4akQybKytv-oc1rHxwoUUXM3VQ=Q>