From nobody Sun Jan 29 18:56:14 2023
X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P4gWg0vpRz3bntH;
Sun, 29 Jan 2023 18:56:15 +0000 (UTC)
(envelope-from git@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (4096 bits) client-digest SHA256)
(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4P4gWf5FMfz4kss;
Sun, 29 Jan 2023 18:56:14 +0000 (UTC)
(envelope-from git@FreeBSD.org)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim;
t=1675018574;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:mime-version:mime-version:content-type:content-type:
content-transfer-encoding:content-transfer-encoding;
bh=3K3DOKencWLoxKQc2KMt0I9LF4msr1uLrc2POxFogeA=;
b=huGApnGRDAoATm6ygHupDE7bOmJEaq9rGgkqpxwCkr14R+QOiZdBp9+bfjdBtkMdKmptdH
einwsNw3K0eX0ZPicAvhTY2MxzX6l2GHHdil2cZ2aMC4PAb7LpVZfxl0+zoe9zbe68+B9F
+Sl/eHlFLJBP706poPrZcVzaD6xJ+Orx1pw3mjOn4WoOf2Xretkd/n10bszHb4OIPycq/Q
I5/JO1nlojaCqJJCxlCj4OAJl2LNrhrlxhY9vFFkoNi3BQ0nhOZSzuhm8DHQIz25VRmRZq
3+g69AE50cLyjlw69uWjFc1JoqN/eTOpeEZ0+YAvjytSfik7B4XdwXLOu8vo5A==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
s=dkim; t=1675018574;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:mime-version:mime-version:content-type:content-type:
content-transfer-encoding:content-transfer-encoding;
bh=3K3DOKencWLoxKQc2KMt0I9LF4msr1uLrc2POxFogeA=;
b=e3OFsZzswi0ESFkDRkq9u2oGE7McZxSz7Y3AdA7NLSKtSToyuwoxIcRNAiDnhwoabN9InD
fpblb2jus9pRA5dBJD7kmdBuda4DRTeyPyy9x/RxoOAWnyht7YzVv5llNkucwbDy64NY+G
G3+tXtrtwqhrW1cQ6km2ISMvRlGQOyNdbieFXbHCHBJRA2vlbD2IImbfe+8Tbe8jZGcnp+
eEsLVSSu+LIzmhz4ix4GyRkCf4D/1IkNGmG/gDEz7ktV8cHPbMDYknn5RwsoVUAHH0GpBo
g0CC9VOW1kajgQnawYyvlZTLigrmmV+DGeNN8lzWf96plZWmmAFqWU5xZN/rVA==
ARC-Authentication-Results: i=1;
mx1.freebsd.org;
none
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1675018574; a=rsa-sha256; cv=none;
b=IAmhjQRnrw8+MR2c7UcBeqspy48pjwORORVxkz4T3Lc4/IdX9sYESpMdc4FaBOqGas21GI
2Egl9wFgcN5hVfz5Bxki93jJZFO1pX77NLT5+m8stBzGirRQOkijnHBRN12hOEIdpkj+m2
iWPzEqeDRreIIOeCn+jqJySv/T7PY7v1LHH6viOGJ14qP6JIEiwNvl4qEqHAZlYb1TPvOI
mPgEJKEXQTOx89UdCMRp+BXBsJjosChh0ZArcj+GOgmBQh2Fx/4PuTDuKwtcRNRWv1xwaa
9C+2QoekocQXYTGS6R+K7fHUBQtkZOXxpD6Bv16B/a6J9p8uEnLeQhd5yVm2fQ==
Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4P4gWf4Jc5zfhN;
Sun, 29 Jan 2023 18:56:14 +0000 (UTC)
(envelope-from git@FreeBSD.org)
Received: from gitrepo.freebsd.org ([127.0.1.44])
by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 30TIuE8Y092862;
Sun, 29 Jan 2023 18:56:14 GMT
(envelope-from git@gitrepo.freebsd.org)
Received: (from git@localhost)
by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 30TIuEWu092861;
Sun, 29 Jan 2023 18:56:14 GMT
(envelope-from git)
Date: Sun, 29 Jan 2023 18:56:14 GMT
Message-Id: <202301291856.30TIuEWu092861@gitrepo.freebsd.org>
To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org,
dev-commits-src-branches@FreeBSD.org
From: Mark Johnston <markj@FreeBSD.org>
Subject: git: 870464f11f3c - stable/12 - kvmclock: Fix initialization when EARLY_AP_STARTUP is not defined
List-Id: Commits to the stable branches of the FreeBSD src repository <dev-commits-src-branches.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches
List-Help: <mailto:dev-commits-src-branches+help@freebsd.org>
List-Post: <mailto:dev-commits-src-branches@freebsd.org>
List-Subscribe: <mailto:dev-commits-src-branches+subscribe@freebsd.org>
List-Unsubscribe: <mailto:dev-commits-src-branches+unsubscribe@freebsd.org>
Sender: owner-dev-commits-src-branches@freebsd.org
X-BeenThere: dev-commits-src-branches@freebsd.org
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Git-Committer: markj
X-Git-Repository: src
X-Git-Refname: refs/heads/stable/12
X-Git-Reftype: branch
X-Git-Commit: 870464f11f3c53fcc7114ea406f95655b9821263
Auto-Submitted: auto-generated
X-ThisMailContainsUnwantedMimeParts: N
The branch stable/12 has been updated by markj:
URL: https://cgit.FreeBSD.org/src/commit/?id=870464f11f3c53fcc7114ea406f95655b9821263
commit 870464f11f3c53fcc7114ea406f95655b9821263
Author: Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2023-01-13 15:01:00 +0000
Commit: Mark Johnston <markj@FreeBSD.org>
CommitDate: 2023-01-29 18:56:06 +0000
kvmclock: Fix initialization when EARLY_AP_STARTUP is not defined
To attach to the hypervisor, kvmclock needs to write a per-CPU MSR.
When EARLY_AP_STARTUP is not defined, device attach happens too early:
APs are not yet spun up, so smp_rendezvous only runs the callback on the
local CPU. As a result, the timecounter only gets initialized on the
BSP, and then timekeeping is broken on SMP systems.
Implement handling for !EARLY_AP_STARTUP kernels: keep track of the CPU
on which device attach ran, and then use a SI_SUB_SMP SYSINIT to
register the rest of the CPUs with the hypervisor.
Reported by: Shrikanth R Kamath <kshrikanth@juniper.net>
Reviewed by: kib, jhb (earlier versions)
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D37705
(cherry picked from commit 568f552b0410f4f72db9ce710f7803acca23f79f)
---
sys/dev/kvm_clock/kvm_clock.c | 46 ++++++++++++++++++++++++++++++++++++++-----
1 file changed, 41 insertions(+), 5 deletions(-)
diff --git a/sys/dev/kvm_clock/kvm_clock.c b/sys/dev/kvm_clock/kvm_clock.c
index 1a76432e417d..982a345a74c7 100644
--- a/sys/dev/kvm_clock/kvm_clock.c
+++ b/sys/dev/kvm_clock/kvm_clock.c
@@ -70,12 +70,16 @@ struct kvm_clock_softc {
struct pvclock_vcpu_time_info *timeinfos;
u_int msr_tc;
u_int msr_wc;
+#ifndef EARLY_AP_STARTUP
+ int firstcpu;
+#endif
};
static devclass_t kvm_clock_devclass;
static struct pvclock_wall_clock *kvm_clock_get_wallclock(void *arg);
-static void kvm_clock_system_time_enable(struct kvm_clock_softc *sc);
+static void kvm_clock_system_time_enable(struct kvm_clock_softc *sc,
+ const cpuset_t *cpus);
static void kvm_clock_system_time_enable_pcpu(void *arg);
static struct pvclock_wall_clock *
@@ -88,9 +92,10 @@ kvm_clock_get_wallclock(void *arg)
}
static void
-kvm_clock_system_time_enable(struct kvm_clock_softc *sc)
+kvm_clock_system_time_enable(struct kvm_clock_softc *sc, const cpuset_t *cpus)
{
- smp_rendezvous(NULL, kvm_clock_system_time_enable_pcpu, NULL, sc);
+ smp_rendezvous_cpus(*cpus, NULL, kvm_clock_system_time_enable_pcpu,
+ NULL, sc);
}
static void
@@ -104,6 +109,32 @@ kvm_clock_system_time_enable_pcpu(void *arg)
wrmsr(sc->msr_tc, vtophys(&(sc->timeinfos)[curcpu]) | 1);
}
+#ifndef EARLY_AP_STARTUP
+static void
+kvm_clock_init_smp(void *arg __unused)
+{
+ devclass_t kvm_clock_devclass;
+ cpuset_t cpus;
+ struct kvm_clock_softc *sc;
+
+ kvm_clock_devclass = devclass_find(KVM_CLOCK_DEVNAME);
+ sc = devclass_get_softc(kvm_clock_devclass, 0);
+ if (sc == NULL || mp_ncpus == 1)
+ return;
+
+ /*
+ * Register with the hypervisor on all CPUs except the one that
+ * registered in kvm_clock_attach().
+ */
+ cpus = all_cpus;
+ KASSERT(CPU_ISSET(sc->firstcpu, &cpus),
+ ("%s: invalid first CPU %d", __func__, sc->firstcpu));
+ CPU_CLR(sc->firstcpu, &cpus);
+ kvm_clock_system_time_enable(sc, &cpus);
+}
+SYSINIT(kvm_clock, SI_SUB_SMP, SI_ORDER_ANY, kvm_clock_init_smp, NULL);
+#endif
+
static void
kvm_clock_identify(driver_t *driver, device_t parent)
{
@@ -150,7 +181,12 @@ kvm_clock_attach(device_t dev)
/* Set up 'struct pvclock_vcpu_time_info' page(s): */
sc->timeinfos = (struct pvclock_vcpu_time_info *)kmem_malloc(mp_ncpus *
sizeof(struct pvclock_vcpu_time_info), M_WAITOK | M_ZERO);
- kvm_clock_system_time_enable(sc);
+#ifdef EARLY_AP_STARTUP
+ kvm_clock_system_time_enable(sc, &all_cpus);
+#else
+ sc->firstcpu = curcpu;
+ kvm_clock_system_time_enable_pcpu(sc);
+#endif
/*
* Init pvclock; register KVM clock wall clock, register KVM clock
@@ -191,7 +227,7 @@ kvm_clock_resume(device_t dev)
* conservatively assume that the system time must be re-inited in
* suspend/resume scenarios.
*/
- kvm_clock_system_time_enable(device_get_softc(dev));
+ kvm_clock_system_time_enable(device_get_softc(dev), &all_cpus);
pvclock_resume();
inittodr(time_second);
return (0);