From nobody Mon Aug 1 14:19:13 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4LxKxY6Ylhz4XtW5; Mon, 1 Aug 2022 14:19:13 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LxKxY5pHvz49F3; Mon, 1 Aug 2022 14:19:13 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1659363553; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=48XIz6B7wc64ARdOzcxdWfX/d7wD7Q3eMoRi12AEjHk=; b=rP6XCC2VyN7ovC+QYlR/3XnQnRaMSmzUFp/7h5wGCFZBq4vgp6tsqPp/bQIdVNBEHL5xe4 lGOBhjQw04WDOrzzfsv81PZZYeMcsHxlVko7oQnp8UPqlIZr7Lqy6F+QBl9FXUWYs5YSS5 g5nV8SekJiBJVo5t9d+VXhbweJzRaEY5aFiKrHNKdnKAba6+rgs1q/w2zV7Iz96+OSgM6Z pCMK8WnbnQSCuWkbJQzVrcFkPZVeK8OBxMW0CsuET7Cz2PdlB4LNHVtNsN6TUDaUjgR/7J v+4CxxBPAF4sieRYGrU8EjSpij0qq9oIOkOKU+A5QqD/w+8XQnsFa1Gf99Dnpg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4LxKxY4Y31zc8G; Mon, 1 Aug 2022 14:19:13 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 271EJDei025212; Mon, 1 Aug 2022 14:19:13 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 271EJDB6025211; Mon, 1 Aug 2022 14:19:13 GMT (envelope-from git) Date: Mon, 1 Aug 2022 14:19:13 GMT Message-Id: <202208011419.271EJDB6025211@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mark Johnston Subject: git: 84a0d34d10ac - stable/13 - x86: Add a required store-load barrier in cpu_idle() List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 84a0d34d10aca36b1e7f9d00d0c4883f3355883b Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1659363553; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=48XIz6B7wc64ARdOzcxdWfX/d7wD7Q3eMoRi12AEjHk=; b=Tl9PYcGT540fMJMwOhOFVwZoT7SEJAfs7zI2PGzfJ2GcLpcJeGVctbvBh4xsetntqn5kpH l1eAJx4GkejNs1k9EHJLVYeuypRi5CT2TZQoxJM0Uw8rVbCZtfPkbyBTNvRGEnsSg/xwov iRMCyN4c1ph87mMMx0VIO70T4d2T4O69t7K52MbLO4HO4cMAkfy6sAWB0PBZ8j27QTUIsw QIWC4XQIVWy6CUXekG6Rp/aZazPuXIgN9ig6I7kSyPpCVi2J0UTpCK8xC3iPyGxNAbi7Mx E3LCy3RugvF5gteXLX2bqFOkwsDzQaxqkuRgYf1qaCJp3f317TG0VEjT3qXKCQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1659363553; a=rsa-sha256; cv=none; b=iMikg9sZ5ETvXCEo30c/JedAOZHzsC6geOmZAwxlgPgYN5PpXr9SVRfa/7k6//i6bOCUxu oM1/yC7RVrETdi6YvYItSMrD2EkmA6UVfk7oLohNR4W4Z6LN4MoecgqrqjOPPbBXL8/I7Y RAJpeMT6QZk81jjNI3duFK3kEXIkBqJj+sjLv2o7/Sy0TbyJG+/DuS0+k5j/dCuODeYFxF 6sJk/Kw28/pSCT0//6CeaM+tI3JERwNAvw4QBIX6qDiwd+zW4Ns85eZbqyKnm/XadQiriC Yitz4GzM8IBVfJ5XOczLwtTmqpnj36KqfoolNTsXun11SHjW6r7sxqoKkdbNog== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=84a0d34d10aca36b1e7f9d00d0c4883f3355883b commit 84a0d34d10aca36b1e7f9d00d0c4883f3355883b Author: Mark Johnston AuthorDate: 2022-07-14 14:24:25 +0000 Commit: Mark Johnston CommitDate: 2022-08-01 14:12:49 +0000 x86: Add a required store-load barrier in cpu_idle() ULE's tdq_notify() tries to avoid delivering IPIs to the idle thread. In particular, it tries to detect whether the idle thread is running. There are two mechanisms for this: - tdq_cpu_idle, an MI flag which is set prior to calling cpu_idle(). If tdq_cpu_idle == 0, then no IPI is needed; - idle_state, an x86-specific state flag which is updated after cpu_idleclock() is called. The implementation of the second mechanism is racy; the race can cause a CPU to go to sleep with pending work. Specifically, cpu_idle_*() set idle_state = STATE_SLEEPING, then check for pending work by loading the tdq_load field of the CPU's runqueue. These operations can be reordered so that the idle thread observes tdq_load == 0, and tdq_notify() observes idle_state == STATE_RUNNING. Some counters indicate that the idle_state check in tdq_notify() frequently elides an IPI. So, fix the problem by inserting a fence after the store to idle_state, immediately before idling the CPU. PR: 264867 Reviewed by: mav, kib, jhb Sponsored by: The FreeBSD Foundation (cherry picked from commit 03f868b163ad46d6f7cb03dc46fb83ca01fb8f69) --- sys/x86/x86/cpu_machdep.c | 103 ++++++++++++++++++++++++++++------------------ 1 file changed, 62 insertions(+), 41 deletions(-) diff --git a/sys/x86/x86/cpu_machdep.c b/sys/x86/x86/cpu_machdep.c index 53b32672132a..1ec3fb669d01 100644 --- a/sys/x86/x86/cpu_machdep.c +++ b/sys/x86/x86/cpu_machdep.c @@ -52,6 +52,7 @@ __FBSDID("$FreeBSD$"); #include "opt_maxmem.h" #include "opt_mp_watchdog.h" #include "opt_platform.h" +#include "opt_sched.h" #ifdef __i386__ #include "opt_apic.h" #endif @@ -528,32 +529,25 @@ static int idle_mwait = 1; /* Use MONITOR/MWAIT for short idle. */ SYSCTL_INT(_machdep, OID_AUTO, idle_mwait, CTLFLAG_RWTUN, &idle_mwait, 0, "Use MONITOR/MWAIT for short idle"); -static void -cpu_idle_acpi(sbintime_t sbt) +static bool +cpu_idle_enter(int *statep, int newstate) { - int *state; + KASSERT(atomic_load_int(statep) == STATE_RUNNING, + ("%s: state %d", __func__, atomic_load_int(statep))); - state = &PCPU_PTR(monitorbuf)->idle_state; - atomic_store_int(state, STATE_SLEEPING); - - /* See comments in cpu_idle_hlt(). */ - disable_intr(); - if (sched_runnable()) - enable_intr(); - else if (cpu_idle_hook) - cpu_idle_hook(sbt); - else - acpi_cpu_c1(); - atomic_store_int(state, STATE_RUNNING); -} - -static void -cpu_idle_hlt(sbintime_t sbt) -{ - int *state; - - state = &PCPU_PTR(monitorbuf)->idle_state; - atomic_store_int(state, STATE_SLEEPING); + /* + * A fence is needed to prevent reordering of the load in + * sched_runnable() with this store to the idle state word. Without it, + * cpu_idle_wakeup() can observe the state as STATE_RUNNING after having + * added load to the queue, and elide an IPI. Then, sched_runnable() + * can observe tdq_load == 0, so the CPU ends up idling with pending + * work. tdq_notify() similarly ensures that a prior update to tdq_load + * is visible before calling cpu_idle_wakeup(). + */ + atomic_store_int(statep, newstate); +#if defined(SCHED_ULE) && defined(SMP) + atomic_thread_fence_seq_cst(); +#endif /* * Since we may be in a critical section from cpu_idle(), if @@ -572,35 +566,62 @@ cpu_idle_hlt(sbintime_t sbt) * interrupt. */ disable_intr(); - if (sched_runnable()) + if (sched_runnable()) { enable_intr(); - else - acpi_cpu_c1(); - atomic_store_int(state, STATE_RUNNING); + atomic_store_int(statep, STATE_RUNNING); + return (false); + } else { + return (true); + } } static void -cpu_idle_mwait(sbintime_t sbt) +cpu_idle_exit(int *statep) +{ + atomic_store_int(statep, STATE_RUNNING); +} + +static void +cpu_idle_acpi(sbintime_t sbt) { int *state; state = &PCPU_PTR(monitorbuf)->idle_state; - atomic_store_int(state, STATE_MWAIT); + if (cpu_idle_enter(state, STATE_SLEEPING)) { + if (cpu_idle_hook) + cpu_idle_hook(sbt); + else + acpi_cpu_c1(); + cpu_idle_exit(state); + } +} - /* See comments in cpu_idle_hlt(). */ - disable_intr(); - if (sched_runnable()) { +static void +cpu_idle_hlt(sbintime_t sbt) +{ + int *state; + + state = &PCPU_PTR(monitorbuf)->idle_state; + if (cpu_idle_enter(state, STATE_SLEEPING)) { + acpi_cpu_c1(); atomic_store_int(state, STATE_RUNNING); - enable_intr(); - return; } +} - cpu_monitor(state, 0, 0); - if (atomic_load_int(state) == STATE_MWAIT) - __asm __volatile("sti; mwait" : : "a" (MWAIT_C1), "c" (0)); - else - enable_intr(); - atomic_store_int(state, STATE_RUNNING); +static void +cpu_idle_mwait(sbintime_t sbt) +{ + int *state; + + state = &PCPU_PTR(monitorbuf)->idle_state; + if (cpu_idle_enter(state, STATE_MWAIT)) { + cpu_monitor(state, 0, 0); + if (atomic_load_int(state) == STATE_MWAIT) + __asm __volatile("sti; mwait" : : "a" (MWAIT_C1), "c" (0)); + else + enable_intr(); + cpu_idle_exit(state); + } } static void