From nobody Wed Jul 19 14:59:51 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R5fB02PSjz4nTkF; Wed, 19 Jul 2023 14:59:52 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R5fB01Dfgz3mN1; Wed, 19 Jul 2023 14:59:52 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1689778792; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=EhWSKzhZ5cgUmfAgm+55T1ACqrZgbmXcwp6anuwaaiI=; b=aBnZIGPawlhbNiKTUaG+dqR37Aq8bVXtSlbXuRrMrM3seSDuKAAccjZiZ8qbuwWwQq9hxC LpJqVK5enbCT+qsoHiqPAwpMrdqbtt1oChAxGJTQ0pLmID3aQoyfAys/WZt4YxAe0cI1gt 4on2+0UE5feopvHEwu62Q5ZGvH9+Gjpvv+iMvkXBcWh8PSApl02zJbaJXCFmON1yPwee+Q huMuZUId/gvkCUR2jbIAtHBWcLuBQ+Mnay03mEnS5JNF4BjrvG9qmFgdQF/5TC7o/bic6M 8xLfX+1Kn6yoet0+bSR3mRwf/6SaKmRFNsP4pSHgA6zydQkxSDZHJbX2d6TfUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1689778792; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=EhWSKzhZ5cgUmfAgm+55T1ACqrZgbmXcwp6anuwaaiI=; b=pJXgni6z/Tf7mvd3NCSj4Rol4tj69E7RxTfEs/uj3aa2Mx1nUCwxdY5aI9GqXRr2EIeaer W3Z2pBfCmUaUaXLudh1YytSP5I2o174SnhfKUeiyyN65bkC+PKWWlypc/kdyYAGPoubLxw byQ8pCZdGEU3UNjS/Smjq4wNxCI+toAv9lh6+rMmHIYg/EFQUIJLrAlrEtAj05pGPxYV2G XGJXVrdR7Xr7cym9ELGrVIZZqLXgAd2ZEfg/p2QQabF51dzqDFXxvfwqo62GjxZ1uLTQDF jFe1ZfizBtw6Zdh6fSIXmSxWqO1vzkdQ870VXE1W2BxcpLx0yfZm5JBoSLzxXg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1689778792; a=rsa-sha256; cv=none; b=rXMSU0phZllOQ+D7CLhy7toDt9t1GCe30p707VKXFsjmNofFD3/UTYVhZa0yD0diTvymJJ xfImvYjU060IphpRcQQq1Z+leHhWQx+Ybby399th10RnNZJNmOK46G+yG1CXI0XnZYgLBO /n8CxevJ+kENYXroQtBS/1z/FMhToiBWsRfyg59RLxFkqERQQ5EsXGjzEkniYJNTcOuM8K uY146qmF47lQLwGHdlK7iTpdaf6R/cqNB+Rv2ljgpuVrHrtmazz16mT0bEkwVIyCFbaJMD PuIeiRs2k55gYmJQFk7rNvAn3rDaqUlI4zY6v/yEF/fipepBnuYN39Ihajco5g== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4R5fB00B9Szs6s; Wed, 19 Jul 2023 14:59:52 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 36JExpeC067586; Wed, 19 Jul 2023 14:59:51 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 36JExpHT067585; Wed, 19 Jul 2023 14:59:51 GMT (envelope-from git) Date: Wed, 19 Jul 2023 14:59:51 GMT Message-Id: <202307191459.36JExpHT067585@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Christos Margiolis Subject: git: 5b701ed19c2e - main - kinst: start moving towards per-probe trampolines List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: christos X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 5b701ed19c2ed439457b0177681472aac21fde94 Auto-Submitted: auto-generated The branch main has been updated by christos: URL: https://cgit.FreeBSD.org/src/commit/?id=5b701ed19c2ed439457b0177681472aac21fde94 commit 5b701ed19c2ed439457b0177681472aac21fde94 Author: Christos Margiolis AuthorDate: 2023-07-19 14:57:21 +0000 Commit: Christos Margiolis CommitDate: 2023-07-19 14:57:21 +0000 kinst: start moving towards per-probe trampolines Using per-CPU and per-thread trampolines is expensive and error-prone, since we're rewriting the same memory blocks constantly. Per-probe trampolines solve this problem by giving each probe its own block of executable memory, which more or less remains the same after the initial write. What this patch does, is get rid of the initialization code which allocates a trampoline for each thread, and instead let each port of kinst allocate a trampoline for each new probe created. It also sets up the infrastructure needed to support the new trampoline scheme. This change is not currently supported on amd64, as the amd64 port needs further changes to work, so this is a temporary/gradual patch to fix the riscv and arm64 ports. Reviewed by: markj Approved by: markj (mentor) Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D40962 --- sys/cddl/dev/kinst/kinst.c | 3 +++ sys/cddl/dev/kinst/kinst.h | 28 ++++++++++++++++++++++++++++ sys/cddl/dev/kinst/trampoline.c | 35 ++++++++++++++++++++++++++++++----- 3 files changed, 61 insertions(+), 5 deletions(-) diff --git a/sys/cddl/dev/kinst/kinst.c b/sys/cddl/dev/kinst/kinst.c index 4bd3047f49db..60400a452b95 100644 --- a/sys/cddl/dev/kinst/kinst.c +++ b/sys/cddl/dev/kinst/kinst.c @@ -228,6 +228,9 @@ kinst_destroy(void *arg, dtrace_id_t id, void *parg) struct kinst_probe *kp = parg; LIST_REMOVE(kp, kp_hashnext); +#ifndef __amd64__ + kinst_trampoline_dealloc(kp->kp_tramp); +#endif free(kp, M_KINST); } diff --git a/sys/cddl/dev/kinst/kinst.h b/sys/cddl/dev/kinst/kinst.h index 0a47eb4f3583..390a2d1c13bf 100644 --- a/sys/cddl/dev/kinst/kinst.h +++ b/sys/cddl/dev/kinst/kinst.h @@ -35,10 +35,38 @@ struct kinst_probe { kinst_patchval_t kp_patchval; kinst_patchval_t kp_savedval; kinst_patchval_t *kp_patchpoint; + uint8_t *kp_tramp; struct kinst_probe_md kp_md; }; +struct kinst_cpu_state { + /* + * kinst uses a breakpoint to return from the trampoline and resume + * execution. To do this safely, kinst implements a per-CPU state + * machine; the state is set to KINST_PROBE_FIRED for the duration of + * the trampoline execution (i.e from the time we transfer execution to + * it, until we return). Upon return, the state is set to + * KINST_PROBE_ARMED to indicate that a probe is not currently firing. + * All CPUs have their state initialized to KINST_PROBE_ARMED when + * kinst is loaded. + */ + enum { + KINST_PROBE_ARMED, + KINST_PROBE_FIRED, + } state; + /* + * Points to the probe whose trampoline we're currently executing. + */ + struct kinst_probe *kp; + /* + * Because we execute trampolines with interrupts disabled, we have to + * cache the CPU's status in order to restore it when we return from + * the trampoline. + */ + uint64_t status; +}; + LIST_HEAD(kinst_probe_list, kinst_probe); extern struct kinst_probe_list *kinst_probetab; diff --git a/sys/cddl/dev/kinst/trampoline.c b/sys/cddl/dev/kinst/trampoline.c index 87c01e39745b..adc4eaa7fceb 100644 --- a/sys/cddl/dev/kinst/trampoline.c +++ b/sys/cddl/dev/kinst/trampoline.c @@ -49,8 +49,10 @@ static TAILQ_HEAD(, trampchunk) kinst_trampchunks = TAILQ_HEAD_INITIALIZER(kinst_trampchunks); static struct sx kinst_tramp_sx; SX_SYSINIT(kinst_tramp_sx, &kinst_tramp_sx, "kinst tramp"); +#ifdef __amd64__ static eventhandler_tag kinst_thread_ctor_handler; static eventhandler_tag kinst_thread_dtor_handler; +#endif /* * Fill the trampolines with KINST_TRAMP_FILL_PATTERN so that the kernel will @@ -150,12 +152,14 @@ kinst_trampoline_alloc_locked(int how) if ((how & M_NOWAIT) != 0) return (NULL); - /* - * We didn't find any free trampoline in the current list, - * allocate a new one. If that fails the provider will no - * longer be reliable, so try to warn the user. - */ if ((chunk = kinst_trampchunk_alloc()) == NULL) { +#ifdef __amd64__ + /* + * We didn't find any free trampoline in the current + * list, allocate a new one. If that fails the + * provider will no longer be reliable, so try to warn + * the user. + */ static bool once = true; if (once) { @@ -164,6 +168,7 @@ kinst_trampoline_alloc_locked(int how) "kinst: failed to allocate trampoline, " "probes may not fire"); } +#endif return (NULL); } off = 0; @@ -220,6 +225,7 @@ kinst_trampoline_dealloc(uint8_t *tramp) sx_xunlock(&kinst_tramp_sx); } +#ifdef __amd64__ static void kinst_thread_ctor(void *arg __unused, struct thread *td) { @@ -240,10 +246,12 @@ kinst_thread_dtor(void *arg __unused, struct thread *td) */ kinst_trampoline_dealloc(tramp); } +#endif int kinst_trampoline_init(void) { +#ifdef __amd64__ struct proc *p; struct thread *td; void *tramp; @@ -296,12 +304,21 @@ retry: out: sx_xunlock(&kinst_tramp_sx); sx_sunlock(&allproc_lock); +#else + int error = 0; + + sx_xlock(&kinst_tramp_sx); + TAILQ_INIT(&kinst_trampchunks); + sx_xunlock(&kinst_tramp_sx); +#endif + return (error); } int kinst_trampoline_deinit(void) { +#ifdef __amd64__ struct trampchunk *chunk, *tmp; struct proc *p; struct thread *td; @@ -324,6 +341,14 @@ kinst_trampoline_deinit(void) TAILQ_FOREACH_SAFE(chunk, &kinst_trampchunks, next, tmp) kinst_trampchunk_free(chunk); sx_xunlock(&kinst_tramp_sx); +#else + struct trampchunk *chunk, *tmp; + + sx_xlock(&kinst_tramp_sx); + TAILQ_FOREACH_SAFE(chunk, &kinst_trampchunks, next, tmp) + kinst_trampchunk_free(chunk); + sx_xunlock(&kinst_tramp_sx); +#endif return (0); }