Date: Mon, 21 Aug 2023 19:54:01 GMT From: Ed Maste <emaste@FreeBSD.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Subject: git: 4258eb5a0d97 - main - x86: handle domains with no CPUs usable for intr delivery Message-ID: <202308211954.37LJs153079853@gitrepo.freebsd.org>
next in thread | raw e-mail | index | archive | help
The branch main has been updated by emaste: URL: https://cgit.FreeBSD.org/src/commit/?id=4258eb5a0d971cf9b1ea5e8e98535e29ef3053f1 commit 4258eb5a0d971cf9b1ea5e8e98535e29ef3053f1 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2023-08-18 03:29:33 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-08-21 19:52:10 +0000 x86: handle domains with no CPUs usable for intr delivery We can end up with a domain having no CPUs capable of receiving I/O interrupts. This can occur, for example, when all APIC IDs in a given domain are 256 or greater, and we have no IOMMU. In this case disable per-domain interrupt support, effectively reverting to the behaviour before commit a48de40bcc09 ("Only use CPUs in the domain the device is attached to for default"). This has a performance impact but at least allows the system to be functional. It is a stop- gap until we can rely on the presence of an IOMMU on all x86 platforms. Thanks to AMD for providing the high-thread-count machine I used for testing this change, and to cperciva for testing on other hardware. Reviewed by: jhb Tested by: cperciva, emaste Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D41501 --- sys/x86/x86/intr_machdep.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/sys/x86/x86/intr_machdep.c b/sys/x86/x86/intr_machdep.c index b8dbe3611a42..b43fa790d264 100644 --- a/sys/x86/x86/intr_machdep.c +++ b/sys/x86/x86/intr_machdep.c @@ -578,10 +578,17 @@ DB_SHOW_COMMAND(irqs, db_show_irqs) /* * Support for balancing interrupt sources across CPUs. For now we just * allocate CPUs round-robin. + * + * XXX If the system has a domain with without any usable CPUs (e.g., where all + * APIC IDs are 256 or greater and we do not have an IOMMU) we use + * intr_no_domain to fall back to assigning interrupts without regard for + * domain. Once we can rely on the presence of an IOMMU on all x86 platforms + * we can revert this. */ cpuset_t intr_cpus = CPUSET_T_INITIALIZER(0x1); static int current_cpu[MAXMEMDOM]; +static bool intr_no_domain; static void intr_init_cpus(void) @@ -589,7 +596,15 @@ intr_init_cpus(void) int i; for (i = 0; i < vm_ndomains; i++) { + if (CPU_OVERLAP(&cpuset_domain[i], &intr_cpus) == 0) { + intr_no_domain = true; + printf("%s: unable to route interrupts to CPUs in domain %d\n", + __func__, i); + } + current_cpu[i] = 0; + if (intr_no_domain && i > 0) + continue; if (!CPU_ISSET(current_cpu[i], &intr_cpus) || !CPU_ISSET(current_cpu[i], &cpuset_domain[i])) intr_next_cpu(i); @@ -615,6 +630,8 @@ intr_next_cpu(int domain) return (PCPU_GET(apic_id)); #endif + if (intr_no_domain) + domain = 0; mtx_lock_spin(&icu_lock); apic_id = cpu_apic_ids[current_cpu[domain]]; do { @@ -622,7 +639,8 @@ intr_next_cpu(int domain) if (current_cpu[domain] > mp_maxid) current_cpu[domain] = 0; } while (!CPU_ISSET(current_cpu[domain], &intr_cpus) || - !CPU_ISSET(current_cpu[domain], &cpuset_domain[domain])); + (!CPU_ISSET(current_cpu[domain], &cpuset_domain[domain]) && + !intr_no_domain)); mtx_unlock_spin(&icu_lock); return (apic_id); }
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?202308211954.37LJs153079853>