Date: Sat, 1 May 2021 10:51:20 -0400 From: Mark Johnston <markj@freebsd.org> To: =?iso-8859-1?Q?=D6zkan?= KIRIK <ozkan.kirik@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Navdeep Parhar <np@freebsd.org>, John Baldwin <jhb@freebsd.org> Subject: Re: IPsec performace - netisr hits %100 Message-ID: <YI1q6ByrN2akPt79@nuc> In-Reply-To: <CAAcX-AH=vGn_NJQGLx8dvxXdvfD=ZB1CG-C5ztZO5kwYa55KAw@mail.gmail.com> References: <CAAcX-AF=0s5tueCuanFKkoALNkRnWJ-8QrzfCqSu=ReoWvqMug@mail.gmail.com> <YIxpdL9b6v8%2BN%2BLg@nuc> <CAAcX-AHSk92gXQ3HXw4KYpXQ-jTVCjX0svStu5z49ykH-tk2QQ@mail.gmail.com> <CAAcX-AG2KyN-7yMm%2BMpKbCRDKivFQjq6BVR0r50t4P3HpDRx=Q@mail.gmail.com> <YIx6eHEH53B4g1iB@nuc> <CAAcX-AGHNzU%2BvWD0Dvr_BQYcb25V=RHqyLeT7n_XkQiVXSwN0g@mail.gmail.com> <CAAcX-AE_jRirL64tbL4ikRa4XDuvkeQgDObLqphJN7HtXyqwLg@mail.gmail.com> <CAAcX-AH=vGn_NJQGLx8dvxXdvfD=ZB1CG-C5ztZO5kwYa55KAw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, May 01, 2021 at 04:30:59PM +0300, Özkan KIRIK wrote: > This bug is related to CCR. @Navdeep Parhar <np@freebsd.org> , @John Baldwin > <jhb@freebsd.org> if you are interested to fix this bug related with CCR, I > can test if you provide patches. Test environment is explained in my first > email on this thread. > > @Mark Johnston <markj@freebsd.org> Now again on stable/13, > - with aesni, without netipsec/ipsec_input.c patch - 1.44Gbps - single > netisr thread eats %100 cpu > - with qat, without netipsec/ipsec_input.c patch - 1.88Gbps - single netisr > thread eats %100 cpu > - with aesni, with netipsec/ipsec_input.c patch - 1.33Gbps > - with qat, with netipsec/ipsec_input.c patch - 2.85Gbps - > > stable/13 results are better then stable/12 but not enough fast. There is > something makes bottleneck for IPsec. So with these results it looks like we have 4 crypto threads running, which is what I'd expect for two pairs of IP addresses. There is still a single-threaded bottleneck. I would suggest generating a flame graph using DTrace and https://github.com/brendangregg/FlameGraph to see where we're spending CPU time. It would also be useful to know if we're getting errors or drops anywhere. The QAT (sysctl dev.qat.*.stats) and ESP/AH (netstat -s -p (esp|ah)) counters would be a useful start, in addition to counters from cxgbe.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YI1q6ByrN2akPt79>