From owner-freebsd-net@freebsd.org Sat May 1 14:51:22 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 19F9662E387 for ; Sat, 1 May 2021 14:51:22 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-io1-xd2b.google.com (mail-io1-xd2b.google.com [IPv6:2607:f8b0:4864:20::d2b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FXXHY4Xpnz4cbJ; Sat, 1 May 2021 14:51:21 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-io1-xd2b.google.com with SMTP id s4so677006ioc.3; Sat, 01 May 2021 07:51:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=3yrjNTRPnxOaVsIcqd0u1EDEAK5BUTAgqnhfDXf7gaE=; b=GT8O0+ZO9wr6xiswq8j49V6A54bmvqIoBQeLBbtMFSUMMPLFMFgX7tUvgOl2LkA58+ xXITRasQQcHzxpcyesn7EcuNm8m0bCOH9F1FcG8HSmVTgiVDX+9AvYX2qudbt1demLP2 WhUrlYFl1cfQXBu4ubEx8ekCnKPPTJbIW/D0GBOJZL1UzA6E6ttRZDMmogRa/N48sDgJ GIIEes0imb4aAjNNjWdrF8/N7HtfOhIh349xWuYvX4mm9kwUw4moxwl8ngvv6J8b5kWV Lk5a5+xIVArqaslzR4NcbQI288zwvW9IcQnad0jgue5G4K1SblbqYwhU5i7KLRmaCLYw 3X1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=3yrjNTRPnxOaVsIcqd0u1EDEAK5BUTAgqnhfDXf7gaE=; b=F5PMBf7PeDotoy+T20KYlDOFB+gVowW7DwB8o6spOTecFcR7r9w/AwlsPXxiYO/5Mu 8u1QHgYpgFXanGLn9wx9iw5Ag6YNcrBNk3GJVTj/JcHeaGFTqC9iuN2DW5y2B8WEDueX O7Ln8EKYVCvfbSstpfsWuE49LwRt5Ji3Ozg6WyF1ftkq32gV9Dc14RGB/6zJBQWEEQsC qmc8oYxNYNtb6wx5qujiaURt1ox1Rrs2KNsgOigN7rHVCI8LOncUmNYeEyf9kVOWll5z dtozZgdjpMC7HNau2uV79a5+MONJrymK67bahl0VrOTDgRVmelqMK5KXR5W+XT1RYiQQ eO7Q== X-Gm-Message-State: AOAM531jBxIbe4sojCiGz1aDJ3B/mnf7HkSBZ9dwHdR8GHBsvViqQRhn 9uZzgCnaqMc/frr4MTtUPDY= X-Google-Smtp-Source: ABdhPJy4j78747Lvw+LJJ1DlsLjlU0pXze4/9MqMwyjkHpgSd05lYH/LDuJuB+X6JqoVU0XRRJoXwA== X-Received: by 2002:a5d:8f88:: with SMTP id l8mr8018590iol.94.1619880680473; Sat, 01 May 2021 07:51:20 -0700 (PDT) Received: from nuc ([142.126.164.150]) by smtp.gmail.com with ESMTPSA id h4sm2506309ili.52.2021.05.01.07.51.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 01 May 2021 07:51:19 -0700 (PDT) Sender: Mark Johnston Date: Sat, 1 May 2021 10:51:20 -0400 From: Mark Johnston To: =?iso-8859-1?Q?=D6zkan?= KIRIK Cc: FreeBSD Net , Navdeep Parhar , John Baldwin Subject: Re: IPsec performace - netisr hits %100 Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 4FXXHY4Xpnz4cbJ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=GT8O0+ZO; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::d2b as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-2.67 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::d2b:from]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.975]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::d2b:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::d2b:from]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 May 2021 14:51:22 -0000 On Sat, May 01, 2021 at 04:30:59PM +0300, Özkan KIRIK wrote: > This bug is related to CCR. @Navdeep Parhar , @John Baldwin > if you are interested to fix this bug related with CCR, I > can test if you provide patches. Test environment is explained in my first > email on this thread. > > @Mark Johnston Now again on stable/13, > - with aesni, without netipsec/ipsec_input.c patch - 1.44Gbps - single > netisr thread eats %100 cpu > - with qat, without netipsec/ipsec_input.c patch - 1.88Gbps - single netisr > thread eats %100 cpu > - with aesni, with netipsec/ipsec_input.c patch - 1.33Gbps > - with qat, with netipsec/ipsec_input.c patch - 2.85Gbps - > > stable/13 results are better then stable/12 but not enough fast. There is > something makes bottleneck for IPsec. So with these results it looks like we have 4 crypto threads running, which is what I'd expect for two pairs of IP addresses. There is still a single-threaded bottleneck. I would suggest generating a flame graph using DTrace and https://github.com/brendangregg/FlameGraph to see where we're spending CPU time. It would also be useful to know if we're getting errors or drops anywhere. The QAT (sysctl dev.qat.*.stats) and ESP/AH (netstat -s -p (esp|ah)) counters would be a useful start, in addition to counters from cxgbe.