From nobody Tue Jun 24 15:58:59 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4bRV4N1SKzz5yy4B; Tue, 24 Jun 2025 15:59:00 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4bRV4N0hc9z3jD4; Tue, 24 Jun 2025 15:59:00 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1750780740; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=DFqzOWUcleZ5GkOKoT0tgl1vHiWWFwOYAiddxGu6BvU=; b=ZUmMU16P8UaIv98o8scKcPCOVjoSg9ueT+AJxMoVNQvztg5f5juInUaiKfKdJKolRvXwHy i6xs4GtIJ8/nxaRtrfYJEOCjwLeP2HDPQ2yQtOplwJkBHt5lHqzH6wXpRBXLxMLr+kdurL yk43PdaQjdoYZC6LyxkzSl+gRzK+YFx6JpdADZgpaBbn/onIqf01pRliLvdrGqdY3uKuNA n/k45HdCbbtK0GXyMyLDWqFEYcEYoGQbiyHqxWUh3Z6RALAd/2bEipuiOh724g8tJFo+bh 6lNImki7tZP7eX2ZuQn0KASgSV4JguFy0VOCzBrpANazi6CSTxmg/dQwEX8rJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1750780740; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=DFqzOWUcleZ5GkOKoT0tgl1vHiWWFwOYAiddxGu6BvU=; b=wnFbbwqi65qfIAlWgT/jakmOgdUbCrsOM6h4dAUlTPyJ4fneLq9PE/UczT0W4TlWHjbVg3 FmWHgsVC167Et4dNp6H4nSEcl8vAlRO5yfBUeepPkUs3dNXPcmcCv0QOYzObxLppwLhG13 PYsh+B5rj01JRUtY8QSI4Wu3Ggg9v77gpY5qQEnZkEiIE16wujC8bt+a4X9hsF2LxtzXcr Q3IlG3w17rvGLSgfQcQGDDjx72DrFwo0UyKG6yUTgrgXXBVOCT9ntsVb6PBEZSrQml/ojR 0RieDh4hNz5DZXPOzMHaaFXIq30eKWAWdEaMcD6e7ecU2gXBbDzI+FKe/l0Vrg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1750780740; a=rsa-sha256; cv=none; b=Re98yJX7BwgexBXuMdaSu5X4QgB7GQCGWddl2jKTHlFNc3rF/HGAybWDT/cSS/6beLJ/V7 2MrxTD/AroqbieXAwnzFSQ+BwvwrbcsYpG6ujGCFMILL5+5wmdG8uY06YxtCpi8lVPKTvM LHSoUGwiYaNcV/9KPyCynm/GeCQAng6uEjc4HBtYRAeQWoRKdN5g9LX8bhkUT7+gR2nd/b Iu/ewFstp5t1Xg3EoL8ucXBjcLuoED/MLVrjmByNHVMVPu9fOdZXSz/IH+9kh7tKJUVBWi 0M+u+50dzS2Uc+lqCctDya2oLowYc7t6sYeE0FFGf4QEMLdYii52M+OPFAOVig== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4bRV4M6yZMztYq; Tue, 24 Jun 2025 15:58:59 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 55OFwxIk004761; Tue, 24 Jun 2025 15:58:59 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 55OFwx9X004758; Tue, 24 Jun 2025 15:58:59 GMT (envelope-from git) Date: Tue, 24 Jun 2025 15:58:59 GMT Message-Id: <202506241558.55OFwx9X004758@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Konstantin Belousov Subject: git: 614087c65e99 - main - sysctl net.inet.tcp.ktlslist: do not rely on global generation for ktls sessions List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kib X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 614087c65e997fcdedcd60f368a035a4b09d106d Auto-Submitted: auto-generated The branch main has been updated by kib: URL: https://cgit.FreeBSD.org/src/commit/?id=614087c65e997fcdedcd60f368a035a4b09d106d commit 614087c65e997fcdedcd60f368a035a4b09d106d Author: Konstantin Belousov AuthorDate: 2025-06-24 02:51:52 +0000 Commit: Konstantin Belousov CommitDate: 2025-06-24 15:58:53 +0000 sysctl net.inet.tcp.ktlslist: do not rely on global generation for ktls sessions Disallow parallel executions for the sysctl, which makes it possible to have the generation count for the sysctl requests itself instead of for the sessions. When the first pass over inpcbs is done, assign them the request' gen count. On the second pass, only externalize the inpcbs with ktls sessions which gen count is equal to the current request. This way, we can be sure that the second pass does not copy out more inpcbs than was counted for in the first pass, while eliminating the global atomic op during ktls session creation. Requested by: gallatin Reviewed by: gallatin, markj Sponsored by: NVidia networking Differential revision: https://reviews.freebsd.org/D51000 --- sys/kern/uipc_ktls.c | 10 ++-------- sys/netinet/tcp_subr.c | 41 +++++++++++++++++++++++++++++------------ sys/sys/ktls.h | 7 ------- 3 files changed, 31 insertions(+), 27 deletions(-) diff --git a/sys/kern/uipc_ktls.c b/sys/kern/uipc_ktls.c index 1cbaa7db2e84..ce09042abdac 100644 --- a/sys/kern/uipc_ktls.c +++ b/sys/kern/uipc_ktls.c @@ -595,8 +595,6 @@ start: return (error); } -uint64_t ktls_glob_gen = 1; - static int ktls_create_session(struct socket *so, struct tls_enable *en, struct ktls_session **tlsp, int direction) @@ -821,8 +819,7 @@ ktls_create_session(struct socket *so, struct tls_enable *en, arc4rand(tls->params.iv + 8, sizeof(uint64_t), 0); } - atomic_thread_fence_rel(); - tls->gen = atomic_fetchadd_64(&ktls_glob_gen, 1); + tls->gen = 0; *tlsp = tls; return (0); } @@ -865,8 +862,7 @@ ktls_clone_session(struct ktls_session *tls, int direction) memcpy(tls_new->params.cipher_key, tls->params.cipher_key, tls->params.cipher_key_len); - atomic_thread_fence_rel(); - tls_new->gen = atomic_fetchadd_64(&ktls_glob_gen, 1); + tls_new->gen = 0; return (tls_new); } @@ -1946,8 +1942,6 @@ ktls_destroy(struct ktls_session *tls) MPASS(tls->refcount == 0); - atomic_add_acq_64(&ktls_glob_gen, 1); - inp = tls->inp; if (tls->tx) { wlocked = INP_WLOCKED(inp); diff --git a/sys/netinet/tcp_subr.c b/sys/netinet/tcp_subr.c index 7cd2168d262b..4542c5ccb8bb 100644 --- a/sys/netinet/tcp_subr.c +++ b/sys/netinet/tcp_subr.c @@ -2669,8 +2669,13 @@ SYSCTL_PROC(_net_inet_tcp, TCPCTL_PCBLIST, pcblist, #define SND_TAG_STATUS_MAXLEN 128 #ifdef KERN_TLS + +static struct sx ktlslist_lock; +SX_SYSINIT(ktlslistlock, &ktlslist_lock, "ktlslist"); +static uint64_t ktls_glob_gen = 1; + static int -tcp_ktlslist(SYSCTL_HANDLER_ARGS, bool export_keys) +tcp_ktlslist_locked(SYSCTL_HANDLER_ARGS, bool export_keys) { struct xinpgen xig; struct inpcb *inp; @@ -2684,6 +2689,7 @@ tcp_ktlslist(SYSCTL_HANDLER_ARGS, bool export_keys) int error; bool ek, p; + sx_assert(&ktlslist_lock, SA_XLOCKED); if (req->newptr != NULL) return (EPERM); @@ -2692,7 +2698,7 @@ tcp_ktlslist(SYSCTL_HANDLER_ARGS, bool export_keys) ipi_gencnt = V_tcbinfo.ipi_gencnt; bzero(&xig, sizeof(xig)); xig.xig_len = sizeof(xig); - xig.xig_gen = atomic_load_acq_64(&ktls_glob_gen); + xig.xig_gen = ktls_glob_gen++; xig.xig_sogen = so_gencnt; struct inpcb_iterator inpi = INP_ALL_ITERATOR(&V_tcbinfo, @@ -2708,7 +2714,8 @@ tcp_ktlslist(SYSCTL_HANDLER_ARGS, bool export_keys) ek = export_keys && cr_canexport_ktlskeys( req->td, inp); ksr = so->so_rcv.sb_tls_info; - if (ktls_session_genvis(ksr, xig.xig_gen)) { + if (ksr != NULL) { + ksr->gen = xig.xig_gen; p = true; if (ek) { sz = SIZE_T_MAX; @@ -2726,7 +2733,8 @@ tcp_ktlslist(SYSCTL_HANDLER_ARGS, bool export_keys) } } kss = so->so_snd.sb_tls_info; - if (ktls_session_genvis(kss, xig.xig_gen)) { + if (kss != NULL) { + kss->gen = xig.xig_gen; p = true; if (ek) { sz = SIZE_T_MAX; @@ -2783,11 +2791,11 @@ tcp_ktlslist(SYSCTL_HANDLER_ARGS, bool export_keys) ksr = so->so_rcv.sb_tls_info; kss = so->so_snd.sb_tls_info; xktls = (struct xktls_session *)buf; - if (ktls_session_genvis(ksr, xig.xig_gen)) { + if (ksr != NULL && ksr->gen == xig.xig_gen) { p = true; ktls_session_to_xktls_onedir(ksr, ek, &xktls->rcv); } - if (ktls_session_genvis(kss, xig.xig_gen)) { + if (kss != NULL && kss->gen == xig.xig_gen) { p = true; ktls_session_to_xktls_onedir(kss, ek, &xktls->snd); } @@ -2798,7 +2806,7 @@ tcp_ktlslist(SYSCTL_HANDLER_ARGS, bool export_keys) xktls->so_pcb = (kvaddr_t)inp; memcpy(&xktls->coninf, &inp->inp_inc, sizeof(xktls->coninf)); len = sizeof(*xktls); - if (ktls_session_genvis(ksr, xig.xig_gen)) { + if (ksr != NULL && ksr->gen == xig.xig_gen) { if (ek) { sz = buflen - len; ktls_session_copy_keys(ksr, buf + len, &sz); @@ -2815,7 +2823,7 @@ tcp_ktlslist(SYSCTL_HANDLER_ARGS, bool export_keys) len += sz; } } - if (ktls_session_genvis(kss, xig.xig_gen)) { + if (kss != NULL && kss->gen == xig.xig_gen) { if (ek) { sz = buflen - len; ktls_session_copy_keys(kss, buf + len, &sz); @@ -2845,8 +2853,6 @@ tcp_ktlslist(SYSCTL_HANDLER_ARGS, bool export_keys) } if (error == 0) { - atomic_thread_fence_rel(); - xig.xig_gen = atomic_load_64(&ktls_glob_gen); xig.xig_sogen = so_gencnt; xig.xig_count = cnt; error = SYSCTL_OUT(req, &xig, sizeof(xig)); @@ -2856,16 +2862,27 @@ tcp_ktlslist(SYSCTL_HANDLER_ARGS, bool export_keys) return (error); } +static int +tcp_ktlslist1(SYSCTL_HANDLER_ARGS, bool export_keys) +{ + int res; + + sx_xlock(&ktlslist_lock); + res = tcp_ktlslist_locked(oidp, arg1, arg2, req, export_keys); + sx_xunlock(&ktlslist_lock); + return (res); +} + static int tcp_ktlslist_nokeys(SYSCTL_HANDLER_ARGS) { - return (tcp_ktlslist(oidp, arg1, arg2, req, false)); + return (tcp_ktlslist1(oidp, arg1, arg2, req, false)); } static int tcp_ktlslist_wkeys(SYSCTL_HANDLER_ARGS) { - return (tcp_ktlslist(oidp, arg1, arg2, req, true)); + return (tcp_ktlslist1(oidp, arg1, arg2, req, true)); } SYSCTL_PROC(_net_inet_tcp, TCPCTL_KTLSLIST, ktlslist, diff --git a/sys/sys/ktls.h b/sys/sys/ktls.h index 0f9e5c5ed87b..a940bcfaba25 100644 --- a/sys/sys/ktls.h +++ b/sys/sys/ktls.h @@ -233,7 +233,6 @@ struct ktls_session { } __aligned(CACHE_LINE_SIZE); extern unsigned int ktls_ifnet_max_rexmit_pct; -extern uint64_t ktls_glob_gen; typedef enum { KTLS_MBUF_CRYPTO_ST_MIXED = 0, @@ -283,12 +282,6 @@ ktls_free(struct ktls_session *tls) ktls_destroy(tls); } -static inline bool -ktls_session_genvis(const struct ktls_session *ks, uint64_t gen) -{ - return (ks != NULL && ks->gen <= gen); -} - void ktls_session_to_xktls_onedir(const struct ktls_session *ks, bool export_keys, struct xktls_session_onedir *xktls_od); void ktls_session_copy_keys(const struct ktls_session *ktls,