From nobody Thu Dec 8 20:16:09 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NSlls2S20z4jLwp; Thu, 8 Dec 2022 20:16:09 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NSlls20NVz3J0H; Thu, 8 Dec 2022 20:16:09 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1670530569; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=YZ6TBCBMmQLtqIGPQHpcpg6+LPVsf51HRr/obZbMC+c=; b=AkxoKwS9MhEazDHBe3Awi/6pOhh56l5/CUOvq7CpE1cjddzQ6ERjBPR4JLNiDI1tZnWrjJ c6c9fber8j0hAYSFNZr2mC1igGDwaqbrmkCFWmjMHvLimJuzBSpXSCKIQqIf8RAHKu3ahO RaceyEJlpohBy+uf++4VM+ANwPCgs6Ej5TI5Bkv1FOIxNlxf/1JHAYcZIAsQ6SQWsq5nqp aUQgA+XeNja6hy9OIM2nYF/TxsQoXmwy9ALZbm6lXmP4B2YQcKiadsT8uyia99HRD59P7Y UAUu5mbIZMi2uz/MduJGfd1cInW99vatY+N6QdjI/sq8gmMhN7Mif+lPBfbYnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1670530569; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=YZ6TBCBMmQLtqIGPQHpcpg6+LPVsf51HRr/obZbMC+c=; b=bjD6ROCQtZ1DiLgx2ykNoynnBOe2tD5IN698IL5NDO49BzqGC3Aj5FS2T5tMEfq8sSI5wh 1pm+VURq8qN3DLLWcOxVNazj2yl+GYadWDheJUEgSuZk5ko2TYT72a9Y718LABV2vzN5ot YJyld49DniabgRg0Y5nukMB8qzllCiBuL61Bn9An5/2lD+TxKaza6LZoxT6tqhDDqLdjny 7jAVKSQ/UWRFUN6aopPIR6PxdFPoKB8YmFDyxC+Rt0n0uz4oZhlG9bMB70a2LDLM28GAun 6+WmaOJD/jAQU6cYYjtdG4K6v8WmD7d+at6Lt+RdAlOapAzTlmlGURnMw+4pug== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1670530569; a=rsa-sha256; cv=none; b=dgwXS27uCQ8Zzc3kXYUz3DEfZsuMlKLkDh2ViZcpZPwdJp2nyO37Vzio87Nxhxr2CYrVTf Q2pW2Jpc4xYxN3MBVAZCCVsIsVHGgxpiWujz3qS4nowVBK9/Z/1YlXxhOoB0p4EPxWPv1p Ut/ZCPVupL0fsWMUZFctYD22B/ZHLvL8kUkWzg3LGzUlmXrk7+b7PNA9AplJHcQsawA8jt saRiAYf9RKSvqOdZCKhI6IuWyt/v6TaQlnIHPpuTp//r6eAbpI9I0rbcrkMECm/cXzSOHu Xazf1IQdyrbsMoG23htBkhyfvkgEBPm5Eu19ctvWbPf6aG6PaW+lrpGmHJ3lQw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4NSlls13zfz17j6; Thu, 8 Dec 2022 20:16:09 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 2B8KG9HD014879; Thu, 8 Dec 2022 20:16:09 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 2B8KG9bA014878; Thu, 8 Dec 2022 20:16:09 GMT (envelope-from git) Date: Thu, 8 Dec 2022 20:16:09 GMT Message-Id: <202212082016.2B8KG9bA014878@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mark Johnston Subject: git: 84d7fe4a6f64 - main - kinst: Add per-CPU interrupt trampolines List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 84d7fe4a6f647faa2c91cb254b155e88e68c798c Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=84d7fe4a6f647faa2c91cb254b155e88e68c798c commit 84d7fe4a6f647faa2c91cb254b155e88e68c798c Author: Mark Johnston AuthorDate: 2022-12-08 20:03:51 +0000 Commit: Mark Johnston CommitDate: 2022-12-08 20:03:51 +0000 kinst: Add per-CPU interrupt trampolines In the common case, kinst emulates a traced instruction by copying it to a trampoline, where it is followed by a jump back to the original code, and pointing the interrupted thread's %rip at the trampoline. In particular, the trampoline is executed with the same CPU context as the original instruction, so if interrupts are enabled at the point where the probe fires, they will be enabled when the trampoline is subsequently executed. It can happen that an interrupt is raised while a thread is executing a kinst trampoline. In that case, it is possible that the interrupt handler will trigger a kinst probe, so we must ensure that the thread does not recurse and overwrite its trampoline before it is finished executing the original contents, otherwise an attempt to trace code called from interrupt handlers can crash the kernel. To that end, add a per-CPU trampoline, used when the probe fired with interrupts disabled. Note that this is not quite complete since it does not handle the possibility of kinst probes firing while executing an NMI handler. Also ensure that we do not trace instructions which set IF, since in that case it is not clear which trampoline (the per-thread trampoline or the per-CPU trampoline) we should use, and since such instructions are rare. Reported and tested by: Domagoj Stolfa Reviewed by: christos Fixes: f0bc4ed144fc ("kinst: Initial revision") Differential Revision: https://reviews.freebsd.org/D37619 --- sys/cddl/dev/kinst/amd64/kinst_isa.c | 73 +++++++++++++++++++++++++++++++++--- sys/cddl/dev/kinst/kinst.c | 7 ++++ sys/cddl/dev/kinst/kinst.h | 3 ++ 3 files changed, 78 insertions(+), 5 deletions(-) diff --git a/sys/cddl/dev/kinst/amd64/kinst_isa.c b/sys/cddl/dev/kinst/amd64/kinst_isa.c index e47cfbbf4d4e..f5fdeabd69e4 100644 --- a/sys/cddl/dev/kinst/amd64/kinst_isa.c +++ b/sys/cddl/dev/kinst/amd64/kinst_isa.c @@ -6,6 +6,7 @@ */ #include +#include #include #include @@ -39,6 +40,14 @@ #define KINST_F_JMP 0x0008 /* instruction is a %rip-relative jmp */ #define KINST_F_MOD_DIRECT 0x0010 /* operand is not a memory address */ +/* + * Per-CPU trampolines used when the interrupted thread is executing with + * interrupts disabled. If an interrupt is raised while executing a trampoline, + * the interrupt thread cannot safely overwrite its trampoline if it hits a + * kinst probe while executing the interrupt handler. + */ +DPCPU_DEFINE_STATIC(uint8_t *, intr_tramp); + /* * Map ModR/M register bits to a trapframe offset. */ @@ -185,7 +194,10 @@ kinst_invop(uintptr_t addr, struct trapframe *frame, uintptr_t scratch) } return (DTRACE_INVOP_CALL); } else { - tramp = curthread->t_kinst; + if ((frame->tf_rflags & PSL_I) == 0) + tramp = DPCPU_GET(intr_tramp); + else + tramp = curthread->t_kinst; if (tramp == NULL) { /* * A trampoline allocation failed, so this probe is @@ -495,7 +507,7 @@ kinst_make_probe(linker_file_t lf, int symindx, linker_symval_t *symval, struct kinst_probe *kp; dtrace_kinst_probedesc_t *pd; const char *func; - int error, n, off; + int error, instrsize, n, off; uint8_t *instr, *limit; pd = opaque; @@ -510,17 +522,37 @@ kinst_make_probe(linker_file_t lf, int symindx, linker_symval_t *symval, /* * Ignore functions not beginning with the usual function prologue. - * These might correspond to assembly routines with which we should not - * meddle. + * These might correspond to exception handlers with which we should not + * meddle. This does however exclude functions which can be safely + * traced, such as cpu_switch(). */ if (*instr != KINST_PUSHL_RBP) return (0); n = 0; while (instr < limit) { + instrsize = dtrace_instr_size(instr); off = (int)(instr - (uint8_t *)symval->value); if (pd->kpd_off != -1 && off != pd->kpd_off) { - instr += dtrace_instr_size(instr); + instr += instrsize; + continue; + } + + /* + * Check for instructions which may enable interrupts. Such + * instructions are tricky to trace since it is unclear whether + * to use the per-thread or per-CPU trampolines. Since they are + * rare, we don't bother to implement special handling for them. + * + * If the caller specified an offset, return an error, otherwise + * silently ignore the instruction so that it remains possible + * to enable all instructions in a function. + */ + if (instrsize == 1 && + (instr[0] == KINST_POPF || instr[0] == KINST_STI)) { + if (pd->kpd_off != -1) + return (EINVAL); + instr += instrsize; continue; } @@ -554,3 +586,34 @@ kinst_make_probe(linker_file_t lf, int symindx, linker_symval_t *symval, return (0); } + +int +kinst_md_init(void) +{ + uint8_t *tramp; + int cpu; + + CPU_FOREACH(cpu) { + tramp = kinst_trampoline_alloc(M_WAITOK); + if (tramp == NULL) + return (ENOMEM); + DPCPU_ID_SET(cpu, intr_tramp, tramp); + } + + return (0); +} + +void +kinst_md_deinit(void) +{ + uint8_t *tramp; + int cpu; + + CPU_FOREACH(cpu) { + tramp = DPCPU_ID_GET(cpu, intr_tramp); + if (tramp != NULL) { + kinst_trampoline_dealloc(DPCPU_ID_GET(cpu, intr_tramp)); + DPCPU_ID_SET(cpu, intr_tramp, NULL); + } + } +} diff --git a/sys/cddl/dev/kinst/kinst.c b/sys/cddl/dev/kinst/kinst.c index 88d59e0cfec4..8c9872ba86c8 100644 --- a/sys/cddl/dev/kinst/kinst.c +++ b/sys/cddl/dev/kinst/kinst.c @@ -180,10 +180,16 @@ kinst_load(void *dummy) error = kinst_trampoline_init(); if (error != 0) return (error); + error = kinst_md_init(); + if (error != 0) { + kinst_trampoline_deinit(); + return (error); + } error = dtrace_register("kinst", &kinst_attr, DTRACE_PRIV_USER, NULL, &kinst_pops, NULL, &kinst_id); if (error != 0) { + kinst_md_deinit(); kinst_trampoline_deinit(); return (error); } @@ -201,6 +207,7 @@ static int kinst_unload(void *dummy) { free(kinst_probetab, M_KINST); + kinst_md_deinit(); kinst_trampoline_deinit(); dtrace_invop_remove(kinst_invop); destroy_dev(kinst_cdev); diff --git a/sys/cddl/dev/kinst/kinst.h b/sys/cddl/dev/kinst/kinst.h index ea1a5b50004f..b7abfd1e61e0 100644 --- a/sys/cddl/dev/kinst/kinst.h +++ b/sys/cddl/dev/kinst/kinst.h @@ -57,6 +57,9 @@ int kinst_trampoline_deinit(void); uint8_t *kinst_trampoline_alloc(int); void kinst_trampoline_dealloc(uint8_t *); +int kinst_md_init(void); +void kinst_md_deinit(void); + #ifdef MALLOC_DECLARE MALLOC_DECLARE(M_KINST); #endif /* MALLOC_DECLARE */