From owner-freebsd-net@freebsd.org Fri Apr 30 21:45:30 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0C1DD62B737 for ; Fri, 30 Apr 2021 21:45:30 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-io1-xd34.google.com (mail-io1-xd34.google.com [IPv6:2607:f8b0:4864:20::d34]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FX5Ws0ZvGz3hmB for ; Fri, 30 Apr 2021 21:45:28 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-io1-xd34.google.com with SMTP id t21so18288297iob.2 for ; Fri, 30 Apr 2021 14:45:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=oXSEeLZmYaDYuTRBYxLZU5E3QFGkS3nRnG4iskWXRUw=; b=DueCkMhZkPJkbd1LcMFU5j0odNmfRcvhpbkg+wNOei83O1+2oW/Zv77NAw3g/csADX K+CTh+AUGpmtZa3h6bQKXZaoyI/H0OThpAJQC4VMwg1LGhVXg2hrq8z6jIobOCEPy2ho obK4VIQEQDgzgwjz9MTP2RbsSpFRUkV3dCjO27l+mPLCbY1vFVhqwnDe2ZqD2EEdGnqb vDWe0BhGH44GDFY6pLbNqkHaQrIyBMBXMwTybV0/iTUjh7p6cNTgxLBzrfzZduDYSMtN azCaoXEd+LqfvPpFCLi2bHlFqggVZEr2QVUd2Dc1S4olZtfEe3hhPXSz49HH6bmOZlSL y8gQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=oXSEeLZmYaDYuTRBYxLZU5E3QFGkS3nRnG4iskWXRUw=; b=PYOkDfsZvOi+MZDRfe9OrSK17RjJVv9csQb3FsYyTSgf2DEBs5ytPeBPUEh2Qf8AKD KzMCaXu3eBRb4B36beESYRuGD6QTAZmiySoXzKbC4+hqxpnquqPHi06JtOJ901xScGjm QmZxw875jlab+elFFizhmNrA+zWjtdOXV3kn9k+7uq1kikeIV98hIjZAsAcGl/aI83sl M70q4ECcR2RdiQalcytTIif6ULs2MGAUKXnt2BHFPu2XKML/BEr+LS5QJ2ZS8J59KQLa w5W9BPlXsY11IS1+ft0MHPKtlQHEC0JfbiablzUc3ZoLjkRVtpluGDYd2yh+EiYV7FEO Zfdg== X-Gm-Message-State: AOAM532rfhrHm4lS7tJnWdN9uZVPl6FfjS/cjEjQBRGnH/2qwVhTcE4j UMZ4hHh7UPpS+URlz4mnG8s= X-Google-Smtp-Source: ABdhPJwn8rnX0mYRZb7/itMEE02WdtNqff/H+Pdj/GfhY+Vz5GQ/2Wkqhf/tKxj1Ox3Mfbqq7lnDvA== X-Received: by 2002:a02:9048:: with SMTP id y8mr6830488jaf.66.1619819127860; Fri, 30 Apr 2021 14:45:27 -0700 (PDT) Received: from nuc ([142.126.164.150]) by smtp.gmail.com with ESMTPSA id m8sm1398685ilc.13.2021.04.30.14.45.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Apr 2021 14:45:27 -0700 (PDT) Sender: Mark Johnston Date: Fri, 30 Apr 2021 17:45:28 -0400 From: Mark Johnston To: =?iso-8859-1?Q?=D6zkan?= KIRIK Cc: FreeBSD Net Subject: Re: IPsec performace - netisr hits %100 Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 4FX5Ws0ZvGz3hmB X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=DueCkMhZ; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::d34 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-2.70 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::d34:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::d34:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::d34:from]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Apr 2021 21:45:30 -0000 On Sat, May 01, 2021 at 12:31:57AM +0300, Özkan KIRIK wrote: > Hello again, > > patch is applied, now netisr is not eating CPU. but performance drops > around 0.2Gbps according to previous kernel. > > I tried also both net.isr.maxthreads=1 and net.isr.maxthreads=4 . results > are same > > Results are: > > - with CCR - 1.8Gbps > top: > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 14 root -16 - 0B 16K CPU5 5 1:38 100.00% [crypto > returns 8] > 3 root -16 - 0B 16K CPU1 1 0:58 77.83% [crypto > returns 0] Could you also try this patch? It won't help with aesni. diff --git a/sys/opencrypto/crypto.c b/sys/opencrypto/crypto.c index dfd22662e87f..bc23056e86ad 100644 --- a/sys/opencrypto/crypto.c +++ b/sys/opencrypto/crypto.c @@ -94,6 +94,7 @@ struct crypto_session { void *softc; uint32_t hid; uint32_t capabilities; + uint64_t id; }; SDT_PROVIDER_DEFINE(opencrypto); @@ -572,6 +573,7 @@ crypto_select_driver(const struct cryptoini *cri, int flags) int crypto_newsession(crypto_session_t *cses, struct cryptoini *cri, int crid) { + static uint64_t sessid = 0; crypto_session_t res; void *softc_mem; struct cryptocap *cap; @@ -616,6 +618,7 @@ crypto_newsession(crypto_session_t *cses, struct cryptoini *cri, int crid) softc_mem = malloc(softc_size, M_CRYPTO_DATA, M_WAITOK | M_ZERO); res = uma_zalloc(cryptoses_zone, M_WAITOK | M_ZERO); res->softc = softc_mem; + res->id = atomic_fetchadd_64(&sessid, 1); CRYPTO_DRIVER_LOCK(); cap = crypto_checkdriver(hid); @@ -1016,7 +1019,7 @@ crypto_dispatch(struct cryptop *crp) binuptime(&crp->crp_tstamp); #endif - crp->crp_retw_id = ((uintptr_t)crp->crp_session) % crypto_workers_num; + crp->crp_retw_id = crp->crp_session->id % crypto_workers_num; if (CRYPTOP_ASYNC(crp)) { if (crp->crp_flags & CRYPTO_F_ASYNC_KEEPORDER) {