From owner-freebsd-net@freebsd.org Fri Apr 30 20:32:54 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9FA44628CCF for ; Fri, 30 Apr 2021 20:32:54 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x730.google.com (mail-qk1-x730.google.com [IPv6:2607:f8b0:4864:20::730]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FX3w53hdzz3Mll for ; Fri, 30 Apr 2021 20:32:53 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qk1-x730.google.com with SMTP id q136so51115003qka.7 for ; Fri, 30 Apr 2021 13:32:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=HraNHC/MLd5+1hxz3m+65lN6QwntfA3z5a9lRBe1Rls=; b=MifqGZC6/U1jGKrGFyEv1ZaZWS5S+tK0G0K4UIx1UlFyPTiBug7JE9h2zJYHMvd3jN q2Zj1kYGnp5epcOiIv0z10iS3HZ9t1kwAI807UOonfcdvOlaEmbK3VOdArW+Y19E9PYa L0PuFuOzAlVN+cuL44BdwuyyzaKVkhKpsFFFKh6hM+60Gf69lxEfrNvyHf+DhMtw5Qk7 RY/5kKTVT22pLqmBrCL4dqUaZLLjhfTKiUDlfjppmnpKS1FOm4m1FZV+EU7VE9QkpFlb AfBNr9WyyAUloTpYM6/brulk+xa08V9/SXh5mCimsNnid/s/ta8JkiP9IYP6Vsf1ydf4 25gQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=HraNHC/MLd5+1hxz3m+65lN6QwntfA3z5a9lRBe1Rls=; b=thQRAJFFmfaevm5/KfHGbUs4w42Hik1I6JJyEpuQzXGJILg1k+PZwgnfoq33pW+FVW VesxQCLYmGoZdHKx/5VGDp7EqxKjHasFS5TmnOsqqAUXSzzFUQLRv4ENJKC1Lm4lOYR6 pcMHGJhJlNwTXw0Uv+2SmdK81/j8EgwqmFL1S5dO38MtaYWbgcOu0BBBdf42ZQbX54VA WUC4C+L2w+XfuLEFZZJ7tLDV8CUBUOmSNzIZwGfXWHpJoU1ugu1ouY9/vw7Fftq/XwTs pg7FYRfYL97dw7ulOcy3EeocGrek47bxlhk01DFP2l/NMrHSRcB3Gzf/DaDtnslODhO5 GZzA== X-Gm-Message-State: AOAM532QqVO9QH23eKvtBFADIwoeAoGa88GKljMdumQCObvGf445A5Df Dd0TWC6OGz75BoQUXEeLHXI= X-Google-Smtp-Source: ABdhPJzNebc85nV/n4HKwe40fhOT9wujNNu8//l3zSulr79WaMnxeZ7p/81HjzdnU845PvLJlJDw9A== X-Received: by 2002:a05:620a:ece:: with SMTP id x14mr7657408qkm.98.1619814772167; Fri, 30 Apr 2021 13:32:52 -0700 (PDT) Received: from nuc ([142.126.164.150]) by smtp.gmail.com with ESMTPSA id 191sm2348875qkk.31.2021.04.30.13.32.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Apr 2021 13:32:51 -0700 (PDT) Sender: Mark Johnston Date: Fri, 30 Apr 2021 16:32:52 -0400 From: Mark Johnston To: =?iso-8859-1?Q?=D6zkan?= KIRIK Cc: FreeBSD Net Subject: Re: IPsec performace - netisr hits %100 Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 4FX3w53hdzz3Mll X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=MifqGZC6; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::730 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-2.70 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::730:from]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::730:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::730:from]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Apr 2021 20:32:54 -0000 On Fri, Apr 30, 2021 at 11:11:48PM +0300, Özkan KIRIK wrote: > Hello, > > I'm using FreeBSD stable/12 built world on 12 April 2021. > my setup is: > [freebsd host cc0] <--------> [cc1 - same freebsd, but jail] > > without IPsec, I can achieve easily to 20Gbps. (test was run with different > source IPs using multiple iperf to scale across multiple queues) > My hardware is Xeon D-2146NT (8 core + SoC Qat), cc0 and cc1 is Chelsio > T62100-LP-CR. > > But with IPsec, throughput is limited to 2Gbps (with ccr) and only one > netisr thread hits %100 cpu. > with aesni throughput is 1,4 Gbps > with QAT throughput is 1,6 Gbps (qat0 C62x, qat1 C62x) > with CCR throughput is 2,0 Gbps (t6nex0) > But always bottleneck is netisr. > > Is there any way to workaround this netisr bottleneck ? > I tried to switch net.isr.dispatch to deferred and hybrid, but performance > drops a bit. I can suggest a couple of things to try. First, we configure only one netisr thread by default. You can create more by setting the net.isr.maxthreads loader tunable. I believe netipsec selects a thread using the SPI so adding more threads might not help much depending on your configuration, but testing with e.g., maxthreads = ncpu could be illuminating. Second, netipsec unconditionally hands rx processing off to netisr threads for some reason, that's why changing the dispatch policy doesn't help. Maybe it's to help avoid running out of kernel stack space or to somehow avoid packet reordering in some case that is not clear to me. I tried a patch (see below) which eliminates this and it helped somewhat. If anyone can provide an explanation for the current behaviour I'd appreciate it. Could you try both approaches and report back? It would also be interesting to know how your results compare with 13.0, if possible. commit 618ab87449d412a74bfee4932d84a6fc17afce6c Author: Mark Johnston Date: Thu Jan 7 11:29:14 2021 -0500 netipsec: Avoid deferred dispatch on the input path diff --git a/sys/netipsec/ipsec_input.c b/sys/netipsec/ipsec_input.c index 48acba68a1fe..98d0954c4c53 100644 --- a/sys/netipsec/ipsec_input.c +++ b/sys/netipsec/ipsec_input.c @@ -425,7 +425,7 @@ ipsec4_common_input_cb(struct mbuf *m, struct secasvar *sav, int skip, error = ipsec_if_input(m, sav, af); if (error == 0) { NET_EPOCH_ENTER(et); - error = netisr_queue_src(isr_prot, (uintptr_t)sav->spi, m); + error = netisr_dispatch_src(isr_prot, (uintptr_t)sav->spi, m); NET_EPOCH_EXIT(et); if (error) { IPSEC_ISTAT(sproto, qfull); @@ -624,7 +624,7 @@ ipsec6_common_input_cb(struct mbuf *m, struct secasvar *sav, int skip, error = ipsec_if_input(m, sav, af); if (error == 0) { NET_EPOCH_ENTER(et); - error = netisr_queue_src(isr_prot, + error = netisr_dispatch_src(isr_prot, (uintptr_t)sav->spi, m); NET_EPOCH_EXIT(et); if (error) {