From owner-dev-commits-src-branches@freebsd.org Wed Feb 3 14:37:25 2021 Return-Path: Delivered-To: dev-commits-src-branches@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C191F53871F; Wed, 3 Feb 2021 14:37:25 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DW45d56Fzz4g0v; Wed, 3 Feb 2021 14:37:25 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A23A922ABC; Wed, 3 Feb 2021 14:37:25 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 113EbPbS001734; Wed, 3 Feb 2021 14:37:25 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 113EbPve001733; Wed, 3 Feb 2021 14:37:25 GMT (envelope-from git) Date: Wed, 3 Feb 2021 14:37:25 GMT Message-Id: <202102031437.113EbPve001733@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mark Johnston Subject: git: 10ed8ab4ab9b - stable/12 - opencrypto: Fix assignment of crypto completions to worker threads MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/stable/12 X-Git-Reftype: branch X-Git-Commit: 10ed8ab4ab9bd0239f2913ac2f35af9b9f76221d Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-branches@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commits to the stable branches of the FreeBSD src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Feb 2021 14:37:25 -0000 The branch stable/12 has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=10ed8ab4ab9bd0239f2913ac2f35af9b9f76221d commit 10ed8ab4ab9bd0239f2913ac2f35af9b9f76221d Author: Mark Johnston AuthorDate: 2021-01-20 01:34:35 +0000 Commit: Mark Johnston CommitDate: 2021-02-03 14:37:17 +0000 opencrypto: Fix assignment of crypto completions to worker threads Since r336439 we simply take the session pointer value mod the number of worker threads (ncpu by default). On small systems this ends up funneling all completion work through a single thread, which becomes a bottleneck when processing IPSec traffic using hardware crypto drivers. (Software drivers such as aesni(4) are unaffected since they invoke completion handlers synchonously.) Instead, maintain an incrementing counter with a unique value per session, and use that to distribute work to completion threads. Reviewed by: cem, jhb Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D28159 (cherry picked from commit 98d788c867b9e1d7a7e290254443b87ea77d8ab1) --- sys/opencrypto/crypto.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sys/opencrypto/crypto.c b/sys/opencrypto/crypto.c index dfd22662e87f..bc23056e86ad 100644 --- a/sys/opencrypto/crypto.c +++ b/sys/opencrypto/crypto.c @@ -94,6 +94,7 @@ struct crypto_session { void *softc; uint32_t hid; uint32_t capabilities; + uint64_t id; }; SDT_PROVIDER_DEFINE(opencrypto); @@ -572,6 +573,7 @@ again: int crypto_newsession(crypto_session_t *cses, struct cryptoini *cri, int crid) { + static uint64_t sessid = 0; crypto_session_t res; void *softc_mem; struct cryptocap *cap; @@ -616,6 +618,7 @@ restart: softc_mem = malloc(softc_size, M_CRYPTO_DATA, M_WAITOK | M_ZERO); res = uma_zalloc(cryptoses_zone, M_WAITOK | M_ZERO); res->softc = softc_mem; + res->id = atomic_fetchadd_64(&sessid, 1); CRYPTO_DRIVER_LOCK(); cap = crypto_checkdriver(hid); @@ -1016,7 +1019,7 @@ crypto_dispatch(struct cryptop *crp) binuptime(&crp->crp_tstamp); #endif - crp->crp_retw_id = ((uintptr_t)crp->crp_session) % crypto_workers_num; + crp->crp_retw_id = crp->crp_session->id % crypto_workers_num; if (CRYPTOP_ASYNC(crp)) { if (crp->crp_flags & CRYPTO_F_ASYNC_KEEPORDER) {