From nobody Sun Jan 29 18:55:51 2023 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P4gWD1HGZz3bnff; Sun, 29 Jan 2023 18:55:52 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P4gWD0kCZz4lQ3; Sun, 29 Jan 2023 18:55:52 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675018552; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WzFvi20sRkKvXzjCeLzTg362uLTQc3MnTgL1Hf73Htg=; b=fqyed3cpZaPOlRDW9kILyJiIbBpZnzHoaaPSXQ9zYspA57OPcSEYfWfD9rLoh0aDumQYfE Y2ljeq8ra43HTdNzgqJ5OCBEvJcgiGKbM6XXuSrIyMYgYfLRsDo82E/gK4dUHREhsjKMeG rwx8D19LcvPPI9XxrxTlDFc6vEnhPMHEq1DhF1mK3xoIwSNbBB0p2lO/Ts8uJhJhqlO1Qm rTD7b0YQCB07Qgt/JNOzvmv/nueCOlJlbGH5EVsKxS84Hy/WASZTts7AGfj0xvr5OWMs7E prde9eFeg0F1fngj4qQKWqeDW3gZKFAksAC/GP8yXpBnRBAv+BvyWTP3CHdsng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675018552; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WzFvi20sRkKvXzjCeLzTg362uLTQc3MnTgL1Hf73Htg=; b=lOfo6lpv7W43Hnv+YdwexRHWOlUbATSNoZ6jCp2mQM26VzJUF5Bo0snZxDenZDQLkj7GBu U547iKsB67qKMMilDjqx/AGjvme8d9THFVKYuG+AfSwBB3gEl0Z7CZ2ScbdNu04nkogqUo hdtkAN9++c2BN0diA6rREJ0xHK45DELMXSPo6086QQJBiJDBN0J//aSJucvE7hF9nmUQrm vAdkvLeChuPDBUruWAp9/Fxzk/aQZV8LEDeuvEhGZTmzT2JZ+JAN2EcrTCL2adcXJBY0KV STuILQLLdGCk4WsV6wb6E4ilaiyUzlbfptdo41FEryuK3ZxhVWc0bSuCvoA6qg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1675018552; a=rsa-sha256; cv=none; b=EfEKoBXhuNxdtt0fgzmGXk1rSJhYKzMC+u/nG3ss9JpB7+dE38xtKT7qM/3qSyxAqby5nI b4jToGVaEKo6+zy1cceV4FvGG2GLjxij77w+G6hUEwI7glF17eVipEZeST9dTJk0goFTx4 prJC5PUSGYz2TKJfKezR+I4a6FFlZicFuV7oFUAXQoODZcdSsiW+bekA/No5nJuDbUybij G7zFqiXEoRJTFQTQdA8cwmPBO6b6s1Ld78GhEVkHDVRmdHAcKScWtvSub0EIXNPKPyVVXv tTgPgyLuoLbQT6qWQrMVM06Dcv1vw1/HTs5Ax5u9K3OqhaMBFEm4bto5eiYcrw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4P4gWC6sMQzfk5; Sun, 29 Jan 2023 18:55:51 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 30TItpYc092696; Sun, 29 Jan 2023 18:55:51 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 30TItpNP092695; Sun, 29 Jan 2023 18:55:51 GMT (envelope-from git) Date: Sun, 29 Jan 2023 18:55:51 GMT Message-Id: <202301291855.30TItpNP092695@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mark Johnston Subject: git: 8497b431f19e - stable/13 - kvmclock: Fix initialization when EARLY_AP_STARTUP is not defined List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 8497b431f19efac28a1f1f64f78037f37d7c671e Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=8497b431f19efac28a1f1f64f78037f37d7c671e commit 8497b431f19efac28a1f1f64f78037f37d7c671e Author: Mark Johnston AuthorDate: 2023-01-13 15:01:00 +0000 Commit: Mark Johnston CommitDate: 2023-01-29 18:55:42 +0000 kvmclock: Fix initialization when EARLY_AP_STARTUP is not defined To attach to the hypervisor, kvmclock needs to write a per-CPU MSR. When EARLY_AP_STARTUP is not defined, device attach happens too early: APs are not yet spun up, so smp_rendezvous only runs the callback on the local CPU. As a result, the timecounter only gets initialized on the BSP, and then timekeeping is broken on SMP systems. Implement handling for !EARLY_AP_STARTUP kernels: keep track of the CPU on which device attach ran, and then use a SI_SUB_SMP SYSINIT to register the rest of the CPUs with the hypervisor. Reported by: Shrikanth R Kamath Reviewed by: kib, jhb (earlier versions) Sponsored by: Klara, Inc. Sponsored by: Juniper Networks, Inc. MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D37705 (cherry picked from commit 568f552b0410f4f72db9ce710f7803acca23f79f) --- sys/dev/kvm_clock/kvm_clock.c | 46 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 41 insertions(+), 5 deletions(-) diff --git a/sys/dev/kvm_clock/kvm_clock.c b/sys/dev/kvm_clock/kvm_clock.c index 1a76432e417d..982a345a74c7 100644 --- a/sys/dev/kvm_clock/kvm_clock.c +++ b/sys/dev/kvm_clock/kvm_clock.c @@ -70,12 +70,16 @@ struct kvm_clock_softc { struct pvclock_vcpu_time_info *timeinfos; u_int msr_tc; u_int msr_wc; +#ifndef EARLY_AP_STARTUP + int firstcpu; +#endif }; static devclass_t kvm_clock_devclass; static struct pvclock_wall_clock *kvm_clock_get_wallclock(void *arg); -static void kvm_clock_system_time_enable(struct kvm_clock_softc *sc); +static void kvm_clock_system_time_enable(struct kvm_clock_softc *sc, + const cpuset_t *cpus); static void kvm_clock_system_time_enable_pcpu(void *arg); static struct pvclock_wall_clock * @@ -88,9 +92,10 @@ kvm_clock_get_wallclock(void *arg) } static void -kvm_clock_system_time_enable(struct kvm_clock_softc *sc) +kvm_clock_system_time_enable(struct kvm_clock_softc *sc, const cpuset_t *cpus) { - smp_rendezvous(NULL, kvm_clock_system_time_enable_pcpu, NULL, sc); + smp_rendezvous_cpus(*cpus, NULL, kvm_clock_system_time_enable_pcpu, + NULL, sc); } static void @@ -104,6 +109,32 @@ kvm_clock_system_time_enable_pcpu(void *arg) wrmsr(sc->msr_tc, vtophys(&(sc->timeinfos)[curcpu]) | 1); } +#ifndef EARLY_AP_STARTUP +static void +kvm_clock_init_smp(void *arg __unused) +{ + devclass_t kvm_clock_devclass; + cpuset_t cpus; + struct kvm_clock_softc *sc; + + kvm_clock_devclass = devclass_find(KVM_CLOCK_DEVNAME); + sc = devclass_get_softc(kvm_clock_devclass, 0); + if (sc == NULL || mp_ncpus == 1) + return; + + /* + * Register with the hypervisor on all CPUs except the one that + * registered in kvm_clock_attach(). + */ + cpus = all_cpus; + KASSERT(CPU_ISSET(sc->firstcpu, &cpus), + ("%s: invalid first CPU %d", __func__, sc->firstcpu)); + CPU_CLR(sc->firstcpu, &cpus); + kvm_clock_system_time_enable(sc, &cpus); +} +SYSINIT(kvm_clock, SI_SUB_SMP, SI_ORDER_ANY, kvm_clock_init_smp, NULL); +#endif + static void kvm_clock_identify(driver_t *driver, device_t parent) { @@ -150,7 +181,12 @@ kvm_clock_attach(device_t dev) /* Set up 'struct pvclock_vcpu_time_info' page(s): */ sc->timeinfos = (struct pvclock_vcpu_time_info *)kmem_malloc(mp_ncpus * sizeof(struct pvclock_vcpu_time_info), M_WAITOK | M_ZERO); - kvm_clock_system_time_enable(sc); +#ifdef EARLY_AP_STARTUP + kvm_clock_system_time_enable(sc, &all_cpus); +#else + sc->firstcpu = curcpu; + kvm_clock_system_time_enable_pcpu(sc); +#endif /* * Init pvclock; register KVM clock wall clock, register KVM clock @@ -191,7 +227,7 @@ kvm_clock_resume(device_t dev) * conservatively assume that the system time must be re-inited in * suspend/resume scenarios. */ - kvm_clock_system_time_enable(device_get_softc(dev)); + kvm_clock_system_time_enable(device_get_softc(dev), &all_cpus); pvclock_resume(); inittodr(time_second); return (0);