From owner-dev-commits-src-all@freebsd.org Fri Apr 30 18:33:52 2021 Return-Path: Delivered-To: dev-commits-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 547356267A0; Fri, 30 Apr 2021 18:33:52 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FX1Gm1wzdz3HNq; Fri, 30 Apr 2021 18:33:52 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 34D10D47; Fri, 30 Apr 2021 18:33:52 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 13UIXq4o071987; Fri, 30 Apr 2021 18:33:52 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 13UIXqjY071986; Fri, 30 Apr 2021 18:33:52 GMT (envelope-from git) Date: Fri, 30 Apr 2021 18:33:52 GMT Message-Id: <202104301833.13UIXqjY071986@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Eric van Gyzen Subject: git: 2f32a971b7f9 - main - Wait longer for a previous IPI to be sent MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: vangyzen X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 2f32a971b7f936170b4d61318238e6cf89f198b5 Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Apr 2021 18:33:52 -0000 The branch main has been updated by vangyzen: URL: https://cgit.FreeBSD.org/src/commit/?id=2f32a971b7f936170b4d61318238e6cf89f198b5 commit 2f32a971b7f936170b4d61318238e6cf89f198b5 Author: Eric van Gyzen AuthorDate: 2021-04-26 15:01:17 +0000 Commit: Eric van Gyzen CommitDate: 2021-04-30 18:32:29 +0000 Wait longer for a previous IPI to be sent When sending an IPI, if a previous IPI is still pending delivery, native_lapic_ipi_vectored() waits for the previous IPI to be sent. We've seen a few inexplicable panics with the current timeout of 50 ms. Increase the timeout to 1 second and make it tunable. No hardware specification mentions a timeout in this case; I checked the Intel SDM, Intel MP spec, and Intel x2APIC spec. Linux and illumos wait forever. In Linux, see __default_send_IPI_shortcut() in arch/x86/kernel/apic/ipi.c. In illumos, see apic_send_ipi() in usr/src/uts/i86pc/io/pcplusmp/apic_common.c. However, misbehaving hardware could hang the system if we wait forever. Reviewed by: mav kib MFC after: 1 week Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D29942 --- sys/x86/x86/local_apic.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c index 85c3cfb69277..36010fe38102 100644 --- a/sys/x86/x86/local_apic.c +++ b/sys/x86/x86/local_apic.c @@ -202,6 +202,7 @@ static u_long lapic_timer_divisor, count_freq; static struct eventtimer lapic_et; #ifdef SMP static uint64_t lapic_ipi_wait_mult; +static int __read_mostly lapic_ds_idle_timeout = 1000000; #endif unsigned int max_apic_id; @@ -212,6 +213,11 @@ SYSCTL_INT(_hw_apic, OID_AUTO, eoi_suppression, CTLFLAG_RD, &lapic_eoi_suppression, 0, ""); SYSCTL_INT(_hw_apic, OID_AUTO, timer_tsc_deadline, CTLFLAG_RD, &lapic_timer_tsc_deadline, 0, ""); +#ifdef SMP +SYSCTL_INT(_hw_apic, OID_AUTO, ds_idle_timeout, CTLFLAG_RWTUN, + &lapic_ds_idle_timeout, 0, + "timeout (in us) for APIC Delivery Status to become Idle (xAPIC only)"); +#endif static void lapic_calibrate_initcount(struct lapic *la); static void lapic_calibrate_deadline(struct lapic *la); @@ -2035,7 +2041,6 @@ native_lapic_ipi_raw(register_t icrlo, u_int dest) } } -#define BEFORE_SPIN 50000 #ifdef DETECT_DEADLOCK #define AFTER_SPIN 50 #endif @@ -2082,7 +2087,7 @@ native_lapic_ipi_vectored(u_int vector, int dest) icrlo |= APIC_DESTMODE_PHY | APIC_TRIGMOD_EDGE | APIC_LEVEL_ASSERT; /* Wait for an earlier IPI to finish. */ - if (!lapic_ipi_wait(BEFORE_SPIN)) { + if (!lapic_ipi_wait(lapic_ds_idle_timeout)) { if (KERNEL_PANICKED()) return; else