Date: Fri, 30 Apr 2021 23:40:43 +0300 From: =?UTF-8?B?w5Z6a2FuIEtJUklL?= <ozkan.kirik@gmail.com> To: Mark Johnston <markj@freebsd.org> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: IPsec performace - netisr hits %100 Message-ID: <CAAcX-AHSk92gXQ3HXw4KYpXQ-jTVCjX0svStu5z49ykH-tk2QQ@mail.gmail.com> In-Reply-To: <YIxpdL9b6v8%2BN%2BLg@nuc> References: <CAAcX-AF=0s5tueCuanFKkoALNkRnWJ-8QrzfCqSu=ReoWvqMug@mail.gmail.com> <YIxpdL9b6v8%2BN%2BLg@nuc>
next in thread | previous in thread | raw e-mail | index | archive | help
Thank you Mark,
I'm going to recompile kernel with this patch and share results.
After a few days, I can install stable/13 on same hardware and then I'll
repeat same tests.
Thanks again,
Cheers
On Fri, Apr 30, 2021 at 11:32 PM Mark Johnston <markj@freebsd.org> wrote:
> On Fri, Apr 30, 2021 at 11:11:48PM +0300, =C3=96zkan KIRIK wrote:
> > Hello,
> >
> > I'm using FreeBSD stable/12 built world on 12 April 2021.
> > my setup is:
> > [freebsd host cc0] <--------> [cc1 - same freebsd, but jail]
> >
> > without IPsec, I can achieve easily to 20Gbps. (test was run with
> different
> > source IPs using multiple iperf to scale across multiple queues)
> > My hardware is Xeon D-2146NT (8 core + SoC Qat), cc0 and cc1 is Chelsio
> > T62100-LP-CR.
> >
> > But with IPsec, throughput is limited to 2Gbps (with ccr) and only one
> > netisr thread hits %100 cpu.
> > with aesni throughput is 1,4 Gbps
> > with QAT throughput is 1,6 Gbps (qat0 C62x, qat1 C62x)
> > with CCR throughput is 2,0 Gbps (t6nex0)
> > But always bottleneck is netisr.
> >
> > Is there any way to workaround this netisr bottleneck ?
> > I tried to switch net.isr.dispatch to deferred and hybrid, but
> performance
> > drops a bit.
>
> I can suggest a couple of things to try. First, we configure only one
> netisr thread by default. You can create more by setting the
> net.isr.maxthreads loader tunable. I believe netipsec selects a thread
> using the SPI so adding more threads might not help much depending on
> your configuration, but testing with e.g., maxthreads =3D ncpu could be
> illuminating.
>
> Second, netipsec unconditionally hands rx processing off to netisr
> threads for some reason, that's why changing the dispatch policy doesn't
> help. Maybe it's to help avoid running out of kernel stack space or to
> somehow avoid packet reordering in some case that is not clear to me. I
> tried a patch (see below) which eliminates this and it helped somewhat.
> If anyone can provide an explanation for the current behaviour I'd
> appreciate it.
>
> Could you try both approaches and report back? It would also be
> interesting to know how your results compare with 13.0, if possible.
>
> commit 618ab87449d412a74bfee4932d84a6fc17afce6c
> Author: Mark Johnston <markj@FreeBSD.org>
> Date: Thu Jan 7 11:29:14 2021 -0500
>
> netipsec: Avoid deferred dispatch on the input path
>
> diff --git a/sys/netipsec/ipsec_input.c b/sys/netipsec/ipsec_input.c
> index 48acba68a1fe..98d0954c4c53 100644
> --- a/sys/netipsec/ipsec_input.c
> +++ b/sys/netipsec/ipsec_input.c
> @@ -425,7 +425,7 @@ ipsec4_common_input_cb(struct mbuf *m, struct secasva=
r
> *sav, int skip,
> error =3D ipsec_if_input(m, sav, af);
> if (error =3D=3D 0) {
> NET_EPOCH_ENTER(et);
> - error =3D netisr_queue_src(isr_prot, (uintptr_t)sav->spi,=
m);
> + error =3D netisr_dispatch_src(isr_prot, (uintptr_t)sav->s=
pi,
> m);
> NET_EPOCH_EXIT(et);
> if (error) {
> IPSEC_ISTAT(sproto, qfull);
> @@ -624,7 +624,7 @@ ipsec6_common_input_cb(struct mbuf *m, struct secasva=
r
> *sav, int skip,
> error =3D ipsec_if_input(m, sav, af);
> if (error =3D=3D 0) {
> NET_EPOCH_ENTER(et);
> - error =3D netisr_queue_src(isr_prot,
> + error =3D netisr_dispatch_src(isr_prot,
> (uintptr_t)sav->spi, m);
> NET_EPOCH_EXIT(et);
> if (error) {
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAcX-AHSk92gXQ3HXw4KYpXQ-jTVCjX0svStu5z49ykH-tk2QQ>