From owner-freebsd-net@freebsd.org Fri Apr 30 20:40:56 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 771026291AE for ; Fri, 30 Apr 2021 20:40:56 +0000 (UTC) (envelope-from ozkan.kirik@gmail.com) Received: from mail-vs1-xe32.google.com (mail-vs1-xe32.google.com [IPv6:2607:f8b0:4864:20::e32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FX45M5342z3NQT; Fri, 30 Apr 2021 20:40:55 +0000 (UTC) (envelope-from ozkan.kirik@gmail.com) Received: by mail-vs1-xe32.google.com with SMTP id k124so36321823vsk.3; Fri, 30 Apr 2021 13:40:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QRl+ComvP9OiGS/DWtC+lBRljyKUB1YRVFOuYN1maSk=; b=acxU6to/A6Z4sXe7LfQk8iLQ1O3FakDTQ0zA1+gdW85FZvEl5IwaxEoM63T9XATXqX fiNcSlXCI3sWxPrfeAk0MFjJ4EPFpqPyDoAW2nB2p2mP8PaV7t53iDYtNquIrYGFhF02 Qhps1XZ+pTn0swWkp7IvE5fw/lSjX3M0KSaSxfTN4DFp+cf6dzt+16ksVhpeJBoXNahg dSljtUZZWMl5D2lHxzt6zCUDO5y72T6IiNHr7l5tKlmXcROW/L8T3Cwp1dgd0alFRYFF xXNf75OYbTYT2Bd30fuedrRpofyAWKFscXPaQZRE2uwe5sd9bv8r5pebKiFyR6/r5NRx /zgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QRl+ComvP9OiGS/DWtC+lBRljyKUB1YRVFOuYN1maSk=; b=XZ5bfXWWyJYhD5mz60IomTJyg3qaYOnwwekO+h3uxpjt9BGJBih8/MMraQgvDa7+Ri dr4jE1XzIkAfv8QDJmSej6sUUiCfKJABCPPK1uLmXAk0XKhiOOSYi+j1SorFthM1aXV5 dFwR/MKsgS6nyM3DDvvW5NstKSyYCGWQ5QbhwEF5ds9FKF5c1Nqye7U10Y+muVdSKbOz 9Jy4i63ns8y0ZqoWavNVvlHQAKqd+SpLH1lii26vo5olAwy0+QE4Og1WunhgD+QHPSTC ZQ7evMCJLL6XWtLFvNeGf5+iyZabQzxZTmis6z0MdMGee4Fmew5+UYF6a+xwIVFH/1F2 PL2w== X-Gm-Message-State: AOAM532m3x0uezRXyU8IF9PqLQJhRhjY7s29b0LqMIEqWUYwJb3KeoTA zk3iEfwDXbm2MM0+kiGacenl3cv1JLMUa3ZilM8puTjSixs= X-Google-Smtp-Source: ABdhPJz9wbKYW1fCwrqF4E0Qe1eRdIfrrv+vWDqkHVyu7ZlW/R857QbHu6Iu0gy1wcToAvIipvVOS5JKvTl4eHLTchY= X-Received: by 2002:a67:f693:: with SMTP id n19mr8496674vso.55.1619815254147; Fri, 30 Apr 2021 13:40:54 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?B?w5Z6a2FuIEtJUklL?= Date: Fri, 30 Apr 2021 23:40:43 +0300 Message-ID: Subject: Re: IPsec performace - netisr hits %100 To: Mark Johnston Cc: FreeBSD Net X-Rspamd-Queue-Id: 4FX45M5342z3NQT X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=acxU6to/; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of ozkankirik@gmail.com designates 2607:f8b0:4864:20::e32 as permitted sender) smtp.mailfrom=ozkankirik@gmail.com X-Spamd-Result: default: False [-3.38 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_MIXED_CHARSET(0.62)[subject]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::e32:from]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::e32:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::e32:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Apr 2021 20:40:56 -0000 Thank you Mark, I'm going to recompile kernel with this patch and share results. After a few days, I can install stable/13 on same hardware and then I'll repeat same tests. Thanks again, Cheers On Fri, Apr 30, 2021 at 11:32 PM Mark Johnston wrote: > On Fri, Apr 30, 2021 at 11:11:48PM +0300, =C3=96zkan KIRIK wrote: > > Hello, > > > > I'm using FreeBSD stable/12 built world on 12 April 2021. > > my setup is: > > [freebsd host cc0] <--------> [cc1 - same freebsd, but jail] > > > > without IPsec, I can achieve easily to 20Gbps. (test was run with > different > > source IPs using multiple iperf to scale across multiple queues) > > My hardware is Xeon D-2146NT (8 core + SoC Qat), cc0 and cc1 is Chelsio > > T62100-LP-CR. > > > > But with IPsec, throughput is limited to 2Gbps (with ccr) and only one > > netisr thread hits %100 cpu. > > with aesni throughput is 1,4 Gbps > > with QAT throughput is 1,6 Gbps (qat0 C62x, qat1 C62x) > > with CCR throughput is 2,0 Gbps (t6nex0) > > But always bottleneck is netisr. > > > > Is there any way to workaround this netisr bottleneck ? > > I tried to switch net.isr.dispatch to deferred and hybrid, but > performance > > drops a bit. > > I can suggest a couple of things to try. First, we configure only one > netisr thread by default. You can create more by setting the > net.isr.maxthreads loader tunable. I believe netipsec selects a thread > using the SPI so adding more threads might not help much depending on > your configuration, but testing with e.g., maxthreads =3D ncpu could be > illuminating. > > Second, netipsec unconditionally hands rx processing off to netisr > threads for some reason, that's why changing the dispatch policy doesn't > help. Maybe it's to help avoid running out of kernel stack space or to > somehow avoid packet reordering in some case that is not clear to me. I > tried a patch (see below) which eliminates this and it helped somewhat. > If anyone can provide an explanation for the current behaviour I'd > appreciate it. > > Could you try both approaches and report back? It would also be > interesting to know how your results compare with 13.0, if possible. > > commit 618ab87449d412a74bfee4932d84a6fc17afce6c > Author: Mark Johnston > Date: Thu Jan 7 11:29:14 2021 -0500 > > netipsec: Avoid deferred dispatch on the input path > > diff --git a/sys/netipsec/ipsec_input.c b/sys/netipsec/ipsec_input.c > index 48acba68a1fe..98d0954c4c53 100644 > --- a/sys/netipsec/ipsec_input.c > +++ b/sys/netipsec/ipsec_input.c > @@ -425,7 +425,7 @@ ipsec4_common_input_cb(struct mbuf *m, struct secasva= r > *sav, int skip, > error =3D ipsec_if_input(m, sav, af); > if (error =3D=3D 0) { > NET_EPOCH_ENTER(et); > - error =3D netisr_queue_src(isr_prot, (uintptr_t)sav->spi,= m); > + error =3D netisr_dispatch_src(isr_prot, (uintptr_t)sav->s= pi, > m); > NET_EPOCH_EXIT(et); > if (error) { > IPSEC_ISTAT(sproto, qfull); > @@ -624,7 +624,7 @@ ipsec6_common_input_cb(struct mbuf *m, struct secasva= r > *sav, int skip, > error =3D ipsec_if_input(m, sav, af); > if (error =3D=3D 0) { > NET_EPOCH_ENTER(et); > - error =3D netisr_queue_src(isr_prot, > + error =3D netisr_dispatch_src(isr_prot, > (uintptr_t)sav->spi, m); > NET_EPOCH_EXIT(et); > if (error) { >