Date: Thu, 12 Sep 2024 13:25:32 -0400 From: Paul Procacci <pprocacci@gmail.com> To: Sad Clouds <cryintothebluesky@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: Performance issues with vnet jails + epair + bridge Message-ID: <CAFbbPujAEer3aO7VcZ1CtgUUCHsG9eXfn_4s6SJok83GFW4JPA@mail.gmail.com> In-Reply-To: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> References: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000071abcf0621ef652e
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
On Thu, Sep 12, 2024 at 1:16=E2=80=AFPM Sad Clouds <cryintothebluesky@gmail=
.com>
wrote:
> Hi, I'm using FreeBSD-14.1 and on this particular system I only have a
> single physical network interface, so I followed instructions for
> networking vnet jails via epair and bridge, e.g.
>
> devel
> {
> vnet;
> vnet.interface =3D "e0b_devel";
> exec.prestart +=3D "/jails/jib addm devel genet0";
> exec.poststop +=3D "/jails/jib destroy devel";
> }
>
> The issue is bulk TCP performance throughput between this jail and the
> host is quite poor, with one CPU spinning 100% in kernel and others
> sitting mostly idle.
>
> It seems there is some lock contention somewhere, but I'm not sure if
> this is around vnet, epair or bridge subsystems. Are there
> other alternatives for vnet jails? Can anyone recommend specific
> deployment scenarios? I've seen references to netgraph which could be
> used with jails. Does it have better performance and scalability and
> could replace epair and bridge combination?
>
> Thanks.
>
>
You need to define `poor'.
You need to show `top -SH` while the `problem' occurs.
My guess is packets are getting shuttled between a global taskqueue thread.
This is the default, or at least I'm not aware of this default being
changed.
You can try enabling `options RSS' in your kernel as this would introduce a
taskqueue worker thread per cpu.
~Paul
--=20
__________________
:(){ :|:& };:
--00000000000071abcf0621ef652e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div><div><div dir=3D"ltr"><br></div><br><div class=3D"gma=
il_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Thu, Sep 12, 2024 at 1:1=
6=E2=80=AFPM Sad Clouds <<a href=3D"mailto:cryintothebluesky@gmail.com">=
cryintothebluesky@gmail.com</a>> wrote:<br></div><blockquote class=3D"gm=
ail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,=
204,204);padding-left:1ex">Hi, I'm using FreeBSD-14.1 and on this parti=
cular system I only have a<br>
single physical network interface, so I followed instructions for<br>
networking vnet jails via epair and bridge, e.g.<br>
<br>
devel<br>
{<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 vnet;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 vnet.interface =3D "e0b_devel";<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 exec.prestart +=3D "/jails/jib addm devel =
genet0";<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 exec.poststop +=3D "/jails/jib destroy dev=
el";<br>
}<br>
<br>
The issue is bulk TCP performance throughput between this jail and the<br>
host is quite poor, with one CPU spinning 100% in kernel and others<br>
sitting mostly idle.<br>
<br>
It seems there is some lock contention somewhere, but I'm not sure if<b=
r>
this is around vnet, epair or bridge subsystems. Are there<br>
other alternatives for vnet jails? Can anyone recommend specific<br>
deployment scenarios? I've seen references to netgraph which could be<b=
r>
used with jails. Does it have better performance and scalability and<br>
could replace epair and bridge combination?<br>
<br>
Thanks.<br>
<br>
</blockquote></div>
<div><br></div><div><br></div><div>You need to define `poor'.<br></div>=
You need to show `top -SH` while the `problem' occurs.
<br><br></div>My guess is packets are getting shuttled between a global tas=
kqueue thread.<br></div>This is the default, or at least I'm not aware =
of this default being changed.<br><div><div>You can try enabling `options R=
SS' in your kernel as this would introduce a taskqueue worker thread pe=
r cpu.<br><br></div><div>~Paul<br></div><div>
<br><span class=3D"gmail_signature_prefix">-- </span><br><div dir=3D"ltr" c=
lass=3D"gmail_signature">__________________<br><br>:(){ :|:& };:</div><=
/div></div></div>
--00000000000071abcf0621ef652e--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFbbPujAEer3aO7VcZ1CtgUUCHsG9eXfn_4s6SJok83GFW4JPA>