Date: Sat, 1 May 2021 01:26:40 +0300 From: =?UTF-8?B?w5Z6a2FuIEtJUklL?= <ozkan.kirik@gmail.com> To: Mark Johnston <markj@freebsd.org> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: IPsec performace - netisr hits %100 Message-ID: <CAAcX-AGHNzU%2BvWD0Dvr_BQYcb25V=RHqyLeT7n_XkQiVXSwN0g@mail.gmail.com> In-Reply-To: <YIx6eHEH53B4g1iB@nuc> References: <CAAcX-AF=0s5tueCuanFKkoALNkRnWJ-8QrzfCqSu=ReoWvqMug@mail.gmail.com> <YIxpdL9b6v8%2BN%2BLg@nuc> <CAAcX-AHSk92gXQ3HXw4KYpXQ-jTVCjX0svStu5z49ykH-tk2QQ@mail.gmail.com> <CAAcX-AG2KyN-7yMm%2BMpKbCRDKivFQjq6BVR0r50t4P3HpDRx=Q@mail.gmail.com> <YIx6eHEH53B4g1iB@nuc>
next in thread | previous in thread | raw e-mail | index | archive | help
Here is fresh results;
a bit performance gain achieved. But strangely QAT is better now.
- with CCR - 2.14 Gbps
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
15 root -16 - 0B 16K CPU7 7 0:11 100.00% [crypto
returns 9]
11 root -92 - 0B 1120K CPU4 4 0:10 98.74%
[intr{irq295: t6nex0:0a0}]
14 root -16 - 0B 16K CPU11 11 0:09 95.32% [crypto
returns 8]
11 root -92 - 0B 1120K CPU8 8 0:07 63.49%
[intr{irq297: t6nex0:0a2}]
11 root -92 - 0B 1120K WAIT 12 0:05 42.73%
[intr{irq307: t6nex0:1a2}]
16 root -16 - 0B 16K CPU15 15 0:04 33.82% [crypto
returns 10]
13 root -16 - 0B 16K RUN 3 0:02 21.78% [crypto
returns 7]
4317 root 22 0 23M 4348K sbwait 14 0:01 7.24% iperf -B
172.16.70.6 -c 172.16.68.1 -P 2 -t 20{iperf}
4329 root 23 0 23M 4348K sbwait 14 0:01 7.18% iperf -B
172.16.70.10 -c 172.16.68.1 -P 2 -t 20{iperf}
...
11 root -92 - 0B 1120K WAIT 3 0:00 0.31%
[intr{irq294: t6nex0:evt}]
0 root -92 - 0B 2144K - 14 0:00 0.30%
[kernel{t6nex0 tq0}]
...
- with QAT - 2.63 Gbps
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
8 root -16 - 0B 16K CPU7 7 0:31 92.05% [crypto
returns 5]
7 root -16 - 0B 16K CPU0 0 0:29 86.03% [crypto
returns 4]
9 root -16 - 0B 16K RUN 15 0:15 43.32% [crypto
returns 6]
11 root -92 - 0B 1664K RUN 4 0:15 43.03%
[intr{irq297: t6nex0:0a2}]
6 root -16 - 0B 16K crypto 4 0:09 26.01% [crypto
returns 3]
11 root -92 - 0B 1664K WAIT 12 0:07 19.88%
[intr{irq307: t6nex0:1a2}]
11 root -92 - 0B 1664K WAIT 8 0:02 7.22%
[intr{irq368: qat1}]
4313 root 22 0 23M 4348K sbwait 2 0:01 6.51% iperf -B
172.16.70.8 -c 172.16.68.1 -P 2 -t 20{iperf}
4316 root 22 0 23M 4348K sbwait 3 0:01 6.18% iperf -B
172.16.70.9 -c 172.16.68.1 -P 2 -t 20{iperf}
...
11 root -92 - 0B 1664K WAIT 12 0:02 4.52%
[intr{irq355: qat0}]
...
11 root -92 - 0B 1664K WAIT 12 0:01 1.89%
[intr{irq372: qat1}]
11 root -92 - 0B 1664K WAIT 0 0:00 1.12%
[intr{irq360: qat1}]
11 root -92 - 0B 1664K WAIT 2 0:00 1.11%
[intr{irq362: qat1}]
11 root -92 - 0B 1664K WAIT 14 0:00 1.10%
[intr{irq374: qat1}]
11 root -92 - 0B 1664K WAIT 4 0:00 1.10%
[intr{irq364: qat1}]
11 root -92 - 0B 1664K WAIT 10 0:00 1.10%
[intr{irq370: qat1}]
11 root -92 - 0B 1664K WAIT 7 0:00 1.09%
[intr{irq367: qat1}]
11 root -92 - 0B 1664K WAIT 11 0:00 1.07%
[intr{irq371: qat1}]
11 root -92 - 0B 1664K WAIT 5 0:00 1.04%
[intr{irq365: qat1}]
11 root -92 - 0B 1664K WAIT 15 0:00 1.04%
[intr{irq375: qat1}]
11 root -92 - 0B 1664K WAIT 1 0:00 1.04%
[intr{irq361: qat1}]
11 root -92 - 0B 1664K WAIT 6 0:00 1.03%
[intr{irq366: qat1}]
On Sat, May 1, 2021 at 12:45 AM Mark Johnston <markj@freebsd.org> wrote:
> On Sat, May 01, 2021 at 12:31:57AM +0300, =C3=96zkan KIRIK wrote:
> > Hello again,
> >
> > patch is applied, now netisr is not eating CPU. but performance drops
> > around 0.2Gbps according to previous kernel.
> >
> > I tried also both net.isr.maxthreads=3D1 and net.isr.maxthreads=3D4 . r=
esults
> > are same
> >
> > Results are:
> >
> > - with CCR - 1.8Gbps
> > top:
> > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU
> COMMAND
> > 14 root -16 - 0B 16K CPU5 5 1:38 100.00%
> [crypto
> > returns 8]
> > 3 root -16 - 0B 16K CPU1 1 0:58 77.83%
> [crypto
> > returns 0]
>
> Could you also try this patch? It won't help with aesni.
>
> diff --git a/sys/opencrypto/crypto.c b/sys/opencrypto/crypto.c
> index dfd22662e87f..bc23056e86ad 100644
> --- a/sys/opencrypto/crypto.c
> +++ b/sys/opencrypto/crypto.c
> @@ -94,6 +94,7 @@ struct crypto_session {
> void *softc;
> uint32_t hid;
> uint32_t capabilities;
> + uint64_t id;
> };
>
> SDT_PROVIDER_DEFINE(opencrypto);
> @@ -572,6 +573,7 @@ crypto_select_driver(const struct cryptoini *cri, int
> flags)
> int
> crypto_newsession(crypto_session_t *cses, struct cryptoini *cri, int cri=
d)
> {
> + static uint64_t sessid =3D 0;
> crypto_session_t res;
> void *softc_mem;
> struct cryptocap *cap;
> @@ -616,6 +618,7 @@ crypto_newsession(crypto_session_t *cses, struct
> cryptoini *cri, int crid)
> softc_mem =3D malloc(softc_size, M_CRYPTO_DATA, M_WAITOK | M_ZERO=
);
> res =3D uma_zalloc(cryptoses_zone, M_WAITOK | M_ZERO);
> res->softc =3D softc_mem;
> + res->id =3D atomic_fetchadd_64(&sessid, 1);
>
> CRYPTO_DRIVER_LOCK();
> cap =3D crypto_checkdriver(hid);
> @@ -1016,7 +1019,7 @@ crypto_dispatch(struct cryptop *crp)
> binuptime(&crp->crp_tstamp);
> #endif
>
> - crp->crp_retw_id =3D ((uintptr_t)crp->crp_session) %
> crypto_workers_num;
> + crp->crp_retw_id =3D crp->crp_session->id % crypto_workers_num;
>
> if (CRYPTOP_ASYNC(crp)) {
> if (crp->crp_flags & CRYPTO_F_ASYNC_KEEPORDER) {
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAcX-AGHNzU%2BvWD0Dvr_BQYcb25V=RHqyLeT7n_XkQiVXSwN0g>