From nobody Sun Nov 19 21:03:26 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SYNQk6s0Tz51Kqg; Sun, 19 Nov 2023 21:03:26 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SYNQk58Qjz3MGx; Sun, 19 Nov 2023 21:03:26 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700427806; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=XZ7MJQoCF/wGVLfRWh380HJ6W8HgfQSWNSRCYOVSP9w=; b=g8QdgyRH7Xq8EmJf5NySN1T9tCzK6EPGwBGFIBstcjff2nix2wrsuF13mE7hmirD78ud2J g1FO7iJuOFC/ohkQQIh8Pki3c+4/ZPZ8AI/nrZfdkQUV3OiQjksiMm+y26Nxm3IkYcJzqX RKD9S23LJoEZHePDnH4/wzXFeBamv3aTqRJkXgtmBs+tG0drEFNuzhtmFuzLY7JBGjMGa/ wc5XyL2KHi1uTvgl1irYgIH2NWnhglB9wsk6MWQWdrs62IVnHz3RxjxoJIiYAQZ1VA2Rjq nAK3Wbkrqn8zrMlGl6aWmkJXOpcMcqXpvSz37I05SO7J3CXt2QNuB60FKFrMNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700427806; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=XZ7MJQoCF/wGVLfRWh380HJ6W8HgfQSWNSRCYOVSP9w=; b=c0oeMictE3ZuwDC3phZpRZCJNGZU7NFOHYgnqcKveX9ota2Mfo/HhdmMWpTIJZ1kmYqZJc 9t4FHAeCJXf1bObYpnM6CmDgnrfL6sv2Ue5VP95tltUZH1Qi2wVsXEVG+AajRA8oH/XCGT aPE0gF34Pde5my25y5nvxpS8niMbkV1G1JaL4TR6t2mAgJ28xQYx/reOPZEQDJEMTro0Ci 56FqWaWg+nZE5fQ19z1jXLDTWmKuqjIPGXVl++Pq2j2NdQ9dhfj2crD0XySCTdpveDd5c2 YCmNa1OHhkZNB8s3DppblHEZLwGbS+N2VkFfp9ijmRWHtu8w5uMAM+kefSyJfw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1700427806; a=rsa-sha256; cv=none; b=c16VKexa5rat3aaj1L7SUfjBRYY/h6gULPiYglJjQUd1403LezL+JfzHHRJxw9JBqjYn2a kMuhFFR5zlg0ef01Rt6+X0duZ6taf50yzYeQU6FKejTSDnj+a+xkojUuoSF6mPEAgM1dQ9 A21mwshIVm4rPDqpZ0gsvR8t4ecJJ3Bl84/6bvc+9IUBR3U2/uD6H0sKNctlStrgDSrRxV YrJtfAT5gECz7eW7qa8WyJESORNt4NKkQ4da2+FCCNQazjj4AKZlbTcvAIfKYFNNqlwTQH 6/WYbpLAWNTlI20GYD2vuaA98Lqz7hr1B2tNthHHGWcYP9u1LgJgyunleuMh6w== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SYNQk4GQRz9KT; Sun, 19 Nov 2023 21:03:26 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 3AJL3Qg1085326; Sun, 19 Nov 2023 21:03:26 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 3AJL3Q7N085323; Sun, 19 Nov 2023 21:03:26 GMT (envelope-from git) Date: Sun, 19 Nov 2023 21:03:26 GMT Message-Id: <202311192103.3AJL3Q7N085323@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: "Alfredo Dal'Ava Junior" Subject: git: 91e53779b4fc - stable/14 - powerpc: Implement fpu_kern_enter/fpu_kern_leave List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: alfredo X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 91e53779b4fc39e755a03190f785ce0cf3f83eb7 Auto-Submitted: auto-generated The branch stable/14 has been updated by alfredo: URL: https://cgit.FreeBSD.org/src/commit/?id=91e53779b4fc39e755a03190f785ce0cf3f83eb7 commit 91e53779b4fc39e755a03190f785ce0cf3f83eb7 Author: Shawn Anastasio AuthorDate: 2023-09-17 14:40:48 +0000 Commit: Alfredo Dal'Ava Junior CommitDate: 2023-11-19 20:55:24 +0000 powerpc: Implement fpu_kern_enter/fpu_kern_leave Summary: Provide an implementation of fpu_kern_enter/fpu_kern_leave for PPC to enable FPU, VSX, and Altivec usage in-kernel. The functions currently only support FPU_KERN_NOCTX, but this is sufficient for ossl(1) and many other users of the API. This patchset has been tested on powerpc64le using a modified version of the in-tree tools/tools/crypto/cryptocheck.c tool to check for FPU/Vec register clobbering along with a follow-up patch to enable ossl(4) on powerpc64*. Reviewed by: jhibbits MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D41540 Relnotes: yes (cherry picked from commit a6662c37b6ffee46e18be5f7570149edc64c1d0b) --- share/man/man9/fpu_kern.9 | 5 +- sys/powerpc/include/altivec.h | 2 + sys/powerpc/include/fpu.h | 20 ++++++ sys/powerpc/include/pcb.h | 7 ++- sys/powerpc/powerpc/altivec.c | 34 +++++++++- sys/powerpc/powerpc/exec_machdep.c | 21 +++---- sys/powerpc/powerpc/fpu.c | 125 +++++++++++++++++++++++++++++++++++++ sys/sys/param.h | 2 +- 8 files changed, 199 insertions(+), 17 deletions(-) diff --git a/share/man/man9/fpu_kern.9 b/share/man/man9/fpu_kern.9 index c9dd58e96adf..92dc0eaa7e17 100644 --- a/share/man/man9/fpu_kern.9 +++ b/share/man/man9/fpu_kern.9 @@ -185,7 +185,8 @@ and false otherwise. .Sh NOTES The .Nm -is currently implemented only for the i386, amd64, and arm64 architectures. +is currently implemented only for the i386, amd64, arm64, and powerpc +architectures. .Pp There is no way to handle floating point exceptions raised from kernel mode. @@ -205,6 +206,8 @@ facitily and this manual page were written by .An Konstantin Belousov Aq Mt kib@FreeBSD.org . The arm64 support was added by .An Andrew Turner Aq Mt andrew@FreeBSD.org . +The powerpc support was added by +.An Shawn Anastasio Aq Mt sanastasio@raptorengineering.com . .Sh BUGS .Fn fpu_kern_leave should probably have type diff --git a/sys/powerpc/include/altivec.h b/sys/powerpc/include/altivec.h index 581a568b7034..e5151529f698 100644 --- a/sys/powerpc/include/altivec.h +++ b/sys/powerpc/include/altivec.h @@ -35,5 +35,7 @@ void enable_vec(struct thread *); void save_vec(struct thread *); void save_vec_nodrop(struct thread *); +void enable_vec_kern(void); +void disable_vec(struct thread *td); #endif /* _MACHINE_ALTIVEC_H_ */ diff --git a/sys/powerpc/include/fpu.h b/sys/powerpc/include/fpu.h index 30df3a470b09..aa5640ea31fb 100644 --- a/sys/powerpc/include/fpu.h +++ b/sys/powerpc/include/fpu.h @@ -76,6 +76,26 @@ void save_fpu(struct thread *); void save_fpu_nodrop(struct thread *); void cleanup_fpscr(void); u_int get_fpu_exception(struct thread *); +void enable_fpu_kern(void); +void disable_fpu(struct thread *td); + +/* + * Flags for fpu_kern_alloc_ctx(), fpu_kern_enter() and fpu_kern_thread(). + */ +#define FPU_KERN_NORMAL 0x0000 +#define FPU_KERN_NOWAIT 0x0001 +#define FPU_KERN_KTHR 0x0002 +#define FPU_KERN_NOCTX 0x0004 + +struct fpu_kern_ctx; + +struct fpu_kern_ctx *fpu_kern_alloc_ctx(u_int flags); +void fpu_kern_free_ctx(struct fpu_kern_ctx *ctx); +void fpu_kern_enter(struct thread *td, struct fpu_kern_ctx *ctx, + u_int flags); +int fpu_kern_leave(struct thread *td, struct fpu_kern_ctx *ctx); +int fpu_kern_thread(u_int flags); +int is_fpu_kern_thread(u_int flags); #endif /* _KERNEL */ diff --git a/sys/powerpc/include/pcb.h b/sys/powerpc/include/pcb.h index e5e6e3223406..050ada6b0f64 100644 --- a/sys/powerpc/include/pcb.h +++ b/sys/powerpc/include/pcb.h @@ -48,7 +48,7 @@ struct pcb { register_t pcb_toc; /* toc pointer */ register_t pcb_lr; /* link register */ register_t pcb_dscr; /* dscr value */ - register_t pcb_fscr; + register_t pcb_fscr; register_t pcb_tar; struct pmap *pcb_pm; /* pmap of our vmspace */ jmp_buf *pcb_onfault; /* For use during @@ -56,11 +56,14 @@ struct pcb { int pcb_flags; #define PCB_FPU 0x1 /* Process uses FPU */ #define PCB_FPREGS 0x2 /* Process had FPU registers initialized */ -#define PCB_VEC 0x4 /* Process had Altivec initialized */ +#define PCB_VEC 0x4 /* Process uses Altivec */ #define PCB_VSX 0x8 /* Process had VSX initialized */ #define PCB_CDSCR 0x10 /* Process had Custom DSCR initialized */ #define PCB_HTM 0x20 /* Process had HTM initialized */ #define PCB_CFSCR 0x40 /* Process had FSCR updated */ +#define PCB_KERN_FPU 0x80 /* Kernel is using FPU/Vector unit */ +#define PCB_KERN_FPU_NOSAVE 0x100 /* FPU/Vec state not saved for kernel use */ +#define PCB_VECREGS 0x200 /* Process had Altivec registers initialized */ struct fpu { union { #if _BYTE_ORDER == _BIG_ENDIAN diff --git a/sys/powerpc/powerpc/altivec.c b/sys/powerpc/powerpc/altivec.c index 16e4477703d2..5072cf3dc6f6 100644 --- a/sys/powerpc/powerpc/altivec.c +++ b/sys/powerpc/powerpc/altivec.c @@ -105,10 +105,11 @@ enable_vec(struct thread *td) * the thread, initialise the vector registers and VSCR to 0, and * set the flag to indicate that the vector unit is in use. */ + pcb->pcb_flags |= PCB_VEC; tf->srr1 |= PSL_VEC; - if (!(pcb->pcb_flags & PCB_VEC)) { + if (!(pcb->pcb_flags & PCB_VECREGS)) { memset(&pcb->pcb_vec, 0, sizeof pcb->pcb_vec); - pcb->pcb_flags |= PCB_VEC; + pcb->pcb_flags |= PCB_VECREGS; } /* @@ -170,3 +171,32 @@ save_vec_nodrop(struct thread *td) if (td == PCPU_GET(vecthread)) save_vec_int(td); } + +void +enable_vec_kern(void) +{ + mtmsr(mfmsr() | PSL_VEC); +} + +void +disable_vec(struct thread *td) +{ + register_t msr; + struct pcb *pcb; + struct trapframe *tf; + + pcb = td->td_pcb; + tf = trapframe(td); + + /* Disable PSL_VEC in kernel (if enabled) */ + msr = mfmsr() & ~PSL_VEC; + isync(); + mtmsr(msr); + + /* + * Disable PSL_VEC in userspace. It will be re-enabled when + * an Altivec instruction is executed. + */ + tf->srr1 &= ~PSL_VEC; + pcb->pcb_flags &= ~PCB_VEC; +} diff --git a/sys/powerpc/powerpc/exec_machdep.c b/sys/powerpc/powerpc/exec_machdep.c index b42978ff94a8..05d3a3cf79ba 100644 --- a/sys/powerpc/powerpc/exec_machdep.c +++ b/sys/powerpc/powerpc/exec_machdep.c @@ -441,12 +441,14 @@ grab_mcontext(struct thread *td, mcontext_t *mcp, int flags) * Repeat for Altivec context */ - if (pcb->pcb_flags & PCB_VEC) { - KASSERT(td == curthread, - ("get_mcontext: fp save not curthread")); - critical_enter(); - save_vec(td); - critical_exit(); + if (pcb->pcb_flags & PCB_VECREGS) { + if (pcb->pcb_flags & PCB_VEC) { + KASSERT(td == curthread, + ("get_mcontext: altivec save not curthread")); + critical_enter(); + save_vec(td); + critical_exit(); + } mcp->mc_flags |= _MC_AV_VALID; mcp->mc_vscr = pcb->pcb_vec.vscr; mcp->mc_vrsave = pcb->pcb_vec.vrsave; @@ -543,11 +545,8 @@ set_mcontext(struct thread *td, mcontext_t *mcp) } if (mcp->mc_flags & _MC_AV_VALID) { - if ((pcb->pcb_flags & PCB_VEC) != PCB_VEC) { - critical_enter(); - enable_vec(td); - critical_exit(); - } + /* enable_vec() will happen lazily on a fault */ + pcb->pcb_flags |= PCB_VECREGS; pcb->pcb_vec.vscr = mcp->mc_vscr; pcb->pcb_vec.vrsave = mcp->mc_vrsave; memcpy(pcb->pcb_vec.vr, mcp->mc_avec, sizeof(mcp->mc_avec)); diff --git a/sys/powerpc/powerpc/fpu.c b/sys/powerpc/powerpc/fpu.c index cc1381046b4b..8f5df2f7d576 100644 --- a/sys/powerpc/powerpc/fpu.c +++ b/sys/powerpc/powerpc/fpu.c @@ -42,6 +42,7 @@ #include #include #include +#include static void save_fpu_int(struct thread *td) @@ -259,3 +260,127 @@ get_fpu_exception(struct thread *td) return ucode; } +void +enable_fpu_kern(void) +{ + register_t msr; + + msr = mfmsr() | PSL_FP; + + if (cpu_features & PPC_FEATURE_HAS_VSX) + msr |= PSL_VSX; + + mtmsr(msr); +} + +void +disable_fpu(struct thread *td) +{ + register_t msr; + struct pcb *pcb; + struct trapframe *tf; + + pcb = td->td_pcb; + tf = trapframe(td); + + /* Disable FPU in kernel (if enabled) */ + msr = mfmsr() & ~(PSL_FP | PSL_VSX); + isync(); + mtmsr(msr); + + /* + * Disable FPU in userspace. It will be re-enabled when + * an FP or VSX instruction is executed. + */ + tf->srr1 &= ~(PSL_FP | PSL_VSX); + pcb->pcb_flags &= ~(PCB_FPU | PCB_VSX); +} + +#ifndef __SPE__ +/* + * XXX: Implement fpu_kern_alloc_ctx/fpu_kern_free_ctx once fpu_kern_enter and + * fpu_kern_leave can handle !FPU_KERN_NOCTX. + */ +struct fpu_kern_ctx { +#define FPU_KERN_CTX_DUMMY 0x01 /* avoided save for the kern thread */ +#define FPU_KERN_CTX_INUSE 0x02 + uint32_t flags; +}; + +void +fpu_kern_enter(struct thread *td, struct fpu_kern_ctx *ctx, u_int flags) +{ + struct pcb *pcb; + + pcb = td->td_pcb; + + KASSERT((flags & FPU_KERN_NOCTX) != 0 || ctx != NULL, + ("ctx is required when !FPU_KERN_NOCTX")); + KASSERT(ctx == NULL || (ctx->flags & FPU_KERN_CTX_INUSE) == 0, + ("using inuse ctx")); + KASSERT((pcb->pcb_flags & PCB_KERN_FPU_NOSAVE) == 0, + ("recursive fpu_kern_enter while in PCB_KERN_FPU_NOSAVE state")); + + if ((flags & FPU_KERN_NOCTX) != 0) { + critical_enter(); + + if (pcb->pcb_flags & PCB_FPU) { + save_fpu(td); + pcb->pcb_flags |= PCB_FPREGS; + } + enable_fpu_kern(); + + if (pcb->pcb_flags & PCB_VEC) { + save_vec(td); + pcb->pcb_flags |= PCB_VECREGS; + } + enable_vec_kern(); + + pcb->pcb_flags |= PCB_KERN_FPU | PCB_KERN_FPU_NOSAVE; + return; + } + + KASSERT(0, ("fpu_kern_enter with !FPU_KERN_NOCTX not implemented!")); +} + +int +fpu_kern_leave(struct thread *td, struct fpu_kern_ctx *ctx) +{ + struct pcb *pcb; + + pcb = td->td_pcb; + + if ((pcb->pcb_flags & PCB_KERN_FPU_NOSAVE) != 0) { + KASSERT(ctx == NULL, ("non-null ctx after FPU_KERN_NOCTX")); + KASSERT(PCPU_GET(fpcurthread) == NULL, + ("non-NULL fpcurthread for PCB_FP_NOSAVE")); + CRITICAL_ASSERT(td); + + /* Disable FPU, VMX, and VSX */ + disable_fpu(td); + disable_vec(td); + + pcb->pcb_flags &= ~PCB_KERN_FPU_NOSAVE; + + critical_exit(); + } else { + KASSERT(0, ("fpu_kern_leave with !FPU_KERN_NOCTX not implemented!")); + } + + pcb->pcb_flags &= ~PCB_KERN_FPU; + + return 0; +} + +int +is_fpu_kern_thread(u_int flags __unused) +{ + struct pcb *curpcb; + + if ((curthread->td_pflags & TDP_KTHREAD) == 0) + return (0); + curpcb = curthread->td_pcb; + return ((curpcb->pcb_flags & PCB_KERN_FPU) != 0); +} + +#endif /* !__SPE__ */ diff --git a/sys/sys/param.h b/sys/sys/param.h index e47588c7b879..438b4808817b 100644 --- a/sys/sys/param.h +++ b/sys/sys/param.h @@ -75,7 +75,7 @@ * cannot include sys/param.h and should only be updated here. */ #undef __FreeBSD_version -#define __FreeBSD_version 1400500 +#define __FreeBSD_version 1400501 /* * __FreeBSD_kernel__ indicates that this system uses the kernel of FreeBSD,