Date: Fri, 30 Apr 2021 23:11:48 +0300 From: =?UTF-8?B?w5Z6a2FuIEtJUklL?= <ozkan.kirik@gmail.com> To: FreeBSD Net <freebsd-net@freebsd.org> Subject: IPsec performace - netisr hits %100 Message-ID: <CAAcX-AF=0s5tueCuanFKkoALNkRnWJ-8QrzfCqSu=ReoWvqMug@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello,
I'm using FreeBSD stable/12 built world on 12 April 2021.
my setup is:
[freebsd host cc0] <--------> [cc1 - same freebsd, but jail]
without IPsec, I can achieve easily to 20Gbps. (test was run with different
source IPs using multiple iperf to scale across multiple queues)
My hardware is Xeon D-2146NT (8 core + SoC Qat), cc0 and cc1 is Chelsio
T62100-LP-CR.
But with IPsec, throughput is limited to 2Gbps (with ccr) and only one
netisr thread hits %100 cpu.
with aesni throughput is 1,4 Gbps
with QAT throughput is 1,6 Gbps (qat0 C62x, qat1 C62x)
with CCR throughput is 2,0 Gbps (t6nex0)
But always bottleneck is netisr.
Is there any way to workaround this netisr bottleneck ?
I tried to switch net.isr.dispatch to deferred and hybrid, but performance
drops a bit.
my configuration is below:
net.isr.numthreads: 4
net.isr.maxprot: 16
net.isr.defaultqlimit: 8192
net.isr.maxqlimit: 1000000
net.isr.bindthreads: 1
net.isr.maxthreads: 4
net.isr.dispatch: direct
net.inet.ipsec.async_crypto: 1
FreeBSD Host:
kldload ccr # Chelsio Crypto Accelerator
ifconfig lo1 create 172.16.68.1/24 up
ifconfig cc0 192.168.1.3/24 up
ifconfig ipsec0 create reqid 100
ifconfig ipsec0 inet tunnel 192.168.1.3 192.168.1.5
ifconfig ipsec0 inet 172.16.0.3/16 172.16.0.5
setkey -c << EOF
add 192.168.1.3 192.168.1.5 esp 10000 -m tunnel -u 100 -E aes-gcm-16
"VerySecureKey!!10000";
add 192.168.1.5 192.168.1.3 esp 10001 -m tunnel -u 100 -E aes-gcm-16
"VerySecureKey!!20000";
EOF
route add 172.16.70.0/24 172.16.0.5
iperf -s
FreeBSD Jail:
jail -c name=3Dclient persist vnet vnet.interface=3Dcc1 host.hostname=3Dcli=
ent
jexec client sysctl net.inet.ipsec.async_crypto=3D1
jexec client ifconfig lo1 create 172.16.70.1/24 up
jexec client bash -c 'for i in $(seq 2 10); do ifconfig lo1 172.16.70.$i/32
alias; done'
jexec client ifconfig cc1 192.168.1.5/24 up
jexec client ifconfig ipsec0 create reqid 200
jexec client ifconfig ipsec0 inet tunnel 192.168.1.5 192.168.1.3
jexec client ifconfig ipsec0 inet 172.16.0.5/16 172.16.0.3
jexec client setkey -c
add 192.168.1.3 192.168.1.5 esp 10000 -m tunnel -u 200 -E aes-gcm-16
"VerySecureKey!!10000";
add 192.168.1.5 192.168.1.3 esp 10001 -m tunnel -u 200 -E aes-gcm-16
"VerySecureKey!!20000";
^D
jexec client route add 172.16.68.0/24 172.16.0.3
jexec client bash -c 'for i in $(seq 1 10); do (iperf -B 172.16.70.$i -c
172.16.68.1 -P 2 | grep SUM &); done'
---------------------------------
top -azSHj
PID JID USERNAME PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
11 0 root -72 - 0B 1120K CPU2 2 0:26 100.00%
[intr{swi1: netisr 2}]
11 0 root -92 - 0B 1120K CPU4 4 0:36 72.55%
[intr{irq295: t6nex0:0a0}]
3 0 root -16 - 0B 16K CPU8 8 0:13 51.11%
[crypto returns 0]
11 0 root -92 - 0B 1120K WAIT 11 0:18 43.40%
[intr{irq297: t6nex0:0a2}]
14 0 root -16 - 0B 16K crypto 14 0:09 33.43%
[crypto returns 8]
11 0 root -92 - 0B 1120K WAIT 12 0:11 21.17%
[intr{irq307: t6nex0:1a2}]
9049 1 root 32 0 23M 4356K CPU14 14 0:00 7.50%
iperf -B 172.16.70.9 -c 172.16.68.1 -P 2{iperf}
9040 1 root 30 0 23M 4356K sbwait 8 0:00 6.92%
iperf -B 172.16.70.6 -c 172.16.68.1 -P 2{iperf}
9043 1 root 33 0 23M 4356K sbwait 8 0:00 6.90%
iperf -B 172.16.70.7 -c 172.16.68.1 -P 2{iperf}
9046 1 root 32 0 23M 4356K sbwait 11 0:00 6.72%
iperf -B 172.16.70.8 -c 172.16.68.1 -P 2{iperf}
9031 1 root 24 0 23M 4356K sbwait 6 0:00 6.40%
iperf -B 172.16.70.3 -c 172.16.68.1 -P 2{iperf}
9037 1 root 29 0 23M 4356K sbwait 11 0:00 6.05%
iperf -B 172.16.70.5 -c 172.16.68.1 -P 2{iperf}
9037 1 root 29 0 23M 4356K sbwait 6 0:00 5.82%
iperf -B 172.16.70.5 -c 172.16.68.1 -P 2{iperf}
9025 1 root 22 0 23M 4356K sbwait 14 0:00 5.71%
iperf -B 172.16.70.1 -c 172.16.68.1 -P 2{iperf}
9052 1 root 32 0 23M 4356K sbwait 10 0:00 5.24%
iperf -B 172.16.70.10 -c 172.16.68.1 -P 2{iperf}
If you need more information, i can provide.
Regards,
=C3=96zkan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAcX-AF=0s5tueCuanFKkoALNkRnWJ-8QrzfCqSu=ReoWvqMug>