From owner-freebsd-net@freebsd.org Fri Apr 30 20:12:02 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 48F84627F56 for ; Fri, 30 Apr 2021 20:12:02 +0000 (UTC) (envelope-from ozkan.kirik@gmail.com) Received: from mail-vs1-xe2a.google.com (mail-vs1-xe2a.google.com [IPv6:2607:f8b0:4864:20::e2a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FX3S10zblz3LgW for ; Fri, 30 Apr 2021 20:12:00 +0000 (UTC) (envelope-from ozkan.kirik@gmail.com) Received: by mail-vs1-xe2a.google.com with SMTP id a24so1036406vso.4 for ; Fri, 30 Apr 2021 13:12:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=eh0yK+jCopyxWQvnt/FcK2X6qsULDOk+IfKpPZLFdFw=; b=TS4RBYd9LDjdcMYvnSyteEr+0CXEBqfhtncKFT3s/Me306BULl+1hUNqKO5CT9yat0 rTjFvXkbkd1axrtwSIgrHnVVzWoge8lN+l0bB+Prr4MvBXs04yfCkYm9O26ahcL1bgqG WqlXro0IXhEZEDZZ9sprOImcJ7w9sDzpE1glvrLB6XHhXZFe2hdaZUM5YPo6WVY01Zbb 7D1oNdIvxxJUZhmojNH4GSCxvf448JqYmz1wyTl7RGNO+UYYG7wdaEVwj6BU6X/0R4FZ zeUyBPIXFwp6IVmsAHK34SjfCRQUrM0cXxvh2BI0+cvCmeTwkxfU1FEnPyewEE5nUAGX 4FqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=eh0yK+jCopyxWQvnt/FcK2X6qsULDOk+IfKpPZLFdFw=; b=VnSiXjzhGWCbZp8WUeX9VvsfBxiGchxXmhpHLyCWo7srxXBDgo9bz7RmcGHLOlYvzw wSudDvTzMVEDFjX/IVou6iJGXUM+xXD4Gch499k1Jj/qw8YIsRL0GcLivXELkqXlbTYa +dDWOm3eqNF4fpsV0J/7r8PokUIui90QEskk31noActznVclu/xVOlRTJDys+NYoeQqn QkZx/d3hpwt3MPaZkz+vbRtRjESR87h6m4ssHm29IHYTszJyKubRaK/yF7VVy6ZKAFPL kKx1SXTaPfGoA3wfpcYkAUHSktFP2112HufzDPq/hJPwRqdXjzoWTwR/V4SezpYHfe2o 1wYA== X-Gm-Message-State: AOAM531ZSRG1QlqSygkCCy3mTNbF1EUOu2oU+08vCWi0WExY8WQAiGxy JrZRUy4MkL+V8eO0sVP3EZL1Cd/+Tr6+SCJK0hK/Vut11hCXCQ== X-Google-Smtp-Source: ABdhPJxnQ2S3ww/CxKLZTyD7Eh0DqLoN7RRsLzKSJOLdm5R90Kq24SRPMANgBNtABtlIATgk9R6Eht92q1P68p/wt28= X-Received: by 2002:a05:6102:389:: with SMTP id m9mr8666952vsq.33.1619813518676; Fri, 30 Apr 2021 13:11:58 -0700 (PDT) MIME-Version: 1.0 From: =?UTF-8?B?w5Z6a2FuIEtJUklL?= Date: Fri, 30 Apr 2021 23:11:48 +0300 Message-ID: Subject: IPsec performace - netisr hits %100 To: FreeBSD Net X-Rspamd-Queue-Id: 4FX3S10zblz3LgW X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=TS4RBYd9; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of ozkankirik@gmail.com designates 2607:f8b0:4864:20::e2a as permitted sender) smtp.mailfrom=ozkankirik@gmail.com X-Spamd-Result: default: False [-2.29 / 15.00]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_MIXED_CHARSET(0.71)[subject]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::e2a:from]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::e2a:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::e2a:from]; HTTP_TO_IP(1.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Apr 2021 20:12:02 -0000 Hello, I'm using FreeBSD stable/12 built world on 12 April 2021. my setup is: [freebsd host cc0] <--------> [cc1 - same freebsd, but jail] without IPsec, I can achieve easily to 20Gbps. (test was run with different source IPs using multiple iperf to scale across multiple queues) My hardware is Xeon D-2146NT (8 core + SoC Qat), cc0 and cc1 is Chelsio T62100-LP-CR. But with IPsec, throughput is limited to 2Gbps (with ccr) and only one netisr thread hits %100 cpu. with aesni throughput is 1,4 Gbps with QAT throughput is 1,6 Gbps (qat0 C62x, qat1 C62x) with CCR throughput is 2,0 Gbps (t6nex0) But always bottleneck is netisr. Is there any way to workaround this netisr bottleneck ? I tried to switch net.isr.dispatch to deferred and hybrid, but performance drops a bit. my configuration is below: net.isr.numthreads: 4 net.isr.maxprot: 16 net.isr.defaultqlimit: 8192 net.isr.maxqlimit: 1000000 net.isr.bindthreads: 1 net.isr.maxthreads: 4 net.isr.dispatch: direct net.inet.ipsec.async_crypto: 1 FreeBSD Host: kldload ccr # Chelsio Crypto Accelerator ifconfig lo1 create 172.16.68.1/24 up ifconfig cc0 192.168.1.3/24 up ifconfig ipsec0 create reqid 100 ifconfig ipsec0 inet tunnel 192.168.1.3 192.168.1.5 ifconfig ipsec0 inet 172.16.0.3/16 172.16.0.5 setkey -c << EOF add 192.168.1.3 192.168.1.5 esp 10000 -m tunnel -u 100 -E aes-gcm-16 "VerySecureKey!!10000"; add 192.168.1.5 192.168.1.3 esp 10001 -m tunnel -u 100 -E aes-gcm-16 "VerySecureKey!!20000"; EOF route add 172.16.70.0/24 172.16.0.5 iperf -s FreeBSD Jail: jail -c name=3Dclient persist vnet vnet.interface=3Dcc1 host.hostname=3Dcli= ent jexec client sysctl net.inet.ipsec.async_crypto=3D1 jexec client ifconfig lo1 create 172.16.70.1/24 up jexec client bash -c 'for i in $(seq 2 10); do ifconfig lo1 172.16.70.$i/32 alias; done' jexec client ifconfig cc1 192.168.1.5/24 up jexec client ifconfig ipsec0 create reqid 200 jexec client ifconfig ipsec0 inet tunnel 192.168.1.5 192.168.1.3 jexec client ifconfig ipsec0 inet 172.16.0.5/16 172.16.0.3 jexec client setkey -c add 192.168.1.3 192.168.1.5 esp 10000 -m tunnel -u 200 -E aes-gcm-16 "VerySecureKey!!10000"; add 192.168.1.5 192.168.1.3 esp 10001 -m tunnel -u 200 -E aes-gcm-16 "VerySecureKey!!20000"; ^D jexec client route add 172.16.68.0/24 172.16.0.3 jexec client bash -c 'for i in $(seq 1 10); do (iperf -B 172.16.70.$i -c 172.16.68.1 -P 2 | grep SUM &); done' --------------------------------- top -azSHj PID JID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 0 root -72 - 0B 1120K CPU2 2 0:26 100.00% [intr{swi1: netisr 2}] 11 0 root -92 - 0B 1120K CPU4 4 0:36 72.55% [intr{irq295: t6nex0:0a0}] 3 0 root -16 - 0B 16K CPU8 8 0:13 51.11% [crypto returns 0] 11 0 root -92 - 0B 1120K WAIT 11 0:18 43.40% [intr{irq297: t6nex0:0a2}] 14 0 root -16 - 0B 16K crypto 14 0:09 33.43% [crypto returns 8] 11 0 root -92 - 0B 1120K WAIT 12 0:11 21.17% [intr{irq307: t6nex0:1a2}] 9049 1 root 32 0 23M 4356K CPU14 14 0:00 7.50% iperf -B 172.16.70.9 -c 172.16.68.1 -P 2{iperf} 9040 1 root 30 0 23M 4356K sbwait 8 0:00 6.92% iperf -B 172.16.70.6 -c 172.16.68.1 -P 2{iperf} 9043 1 root 33 0 23M 4356K sbwait 8 0:00 6.90% iperf -B 172.16.70.7 -c 172.16.68.1 -P 2{iperf} 9046 1 root 32 0 23M 4356K sbwait 11 0:00 6.72% iperf -B 172.16.70.8 -c 172.16.68.1 -P 2{iperf} 9031 1 root 24 0 23M 4356K sbwait 6 0:00 6.40% iperf -B 172.16.70.3 -c 172.16.68.1 -P 2{iperf} 9037 1 root 29 0 23M 4356K sbwait 11 0:00 6.05% iperf -B 172.16.70.5 -c 172.16.68.1 -P 2{iperf} 9037 1 root 29 0 23M 4356K sbwait 6 0:00 5.82% iperf -B 172.16.70.5 -c 172.16.68.1 -P 2{iperf} 9025 1 root 22 0 23M 4356K sbwait 14 0:00 5.71% iperf -B 172.16.70.1 -c 172.16.68.1 -P 2{iperf} 9052 1 root 32 0 23M 4356K sbwait 10 0:00 5.24% iperf -B 172.16.70.10 -c 172.16.68.1 -P 2{iperf} If you need more information, i can provide. Regards, =C3=96zkan