From nobody Wed Mar 2 21:57:15 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D8E4D19E02EA; Wed, 2 Mar 2022 21:57:16 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K87JD0pWPz3KS7; Wed, 2 Mar 2022 21:57:16 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1646258236; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=DbDcjngiv5GikLiCj1TfVNEQjuKJM2lmCXhjK+x3IY8=; b=NC6Q7gMI7ldGaecdHgXGVY0TAnVQno7h8BAO35qplwz5SyNGJWU3YpH8tvNdVAWEzxPmyQ 6U4pBQQVpSwPJAJ4fohGZULB+nAr0heL78nuuCFe5kQpdPfpzy4/Nj+QvzlQ+8g8Mp+d3C ZUgan+Jf+WhDW94MgoJ8JtEwZq9MmdUG2TAITSgX2rzXesJ6ngzU2bhy/zEb/FrwCjewPO AUkzZC+D4dKfAtBJk04PZmekwJyS6tlsBN0Ya5I6UvRfWwsVpQOdMyHviX9aTdiddaX7gg DvMHZQoQzEAkqIO4t4x8tjazSzMfC5ydKaU/FrpkFvCUsIDoB6Xtq9XR3LkL4g== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D3C1E3840; Wed, 2 Mar 2022 21:57:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 222LvFrY064993; Wed, 2 Mar 2022 21:57:15 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 222LvF90064992; Wed, 2 Mar 2022 21:57:15 GMT (envelope-from git) Date: Wed, 2 Mar 2022 21:57:15 GMT Message-Id: <202203022157.222LvF90064992@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Eric van Gyzen Subject: git: 94df301ac27e - stable/13 - Wait longer for a previous IPI to be sent List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: vangyzen X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 94df301ac27e7df785d1d581a36dc14d456436c2 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1646258236; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=DbDcjngiv5GikLiCj1TfVNEQjuKJM2lmCXhjK+x3IY8=; b=fhUChqfZNTVy6HgyzNsx6//tgWTF0+41jpAq+ufzPjqUX9nLHHwiEKEhXcJvCVPdbxMJCO KMqPQjEYY/T5zAdQGbIEyO4tr2OMQoDFj/jjj9jDXY7q40CGAE3+In1lOMI6RAqPk8h/Lz KwMsvJCrT2KK3K1sWgcF75ZmTwEfmXsq8Gc1FjdV3ImvFSl/kz9mlV2/dIm0w7aQTPjoWy g62ZNbfiX86Pqc2t9N8ti8d16iuVuH/bFn5n58bhv6MXOOV9ZdMzluQ/mVEqlovyvSOF5J kGNEcCijzf6ZOOgBp8STJgHcvcN3x+FJfsSxTg8ZIX32Yw+E/l8Fh+pN8UoLSQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1646258236; a=rsa-sha256; cv=none; b=aw+G+DJ2Ed3GBULnUaZx2kWKsO/ZzfISO2Yq9Hv5+X1DQHZ75WmTKLpLEw2JXRWMjVVH+h rafg1nV2i/bT1069qMnaQ+RGnQfCE7/V1zDSjAo8H3GllAd/WA/m7PKd0WyUbqYuu80756 bbF+yNVIZeF7aIZ+jrykGOniZeAD64UkytZdH79Ux4XHBSU7Fwa2ZoulsUX6jDn84os8wu ZN4R7pUkN/F1RsXP/bJ4BVOf5Kw+sCJf34Dbpmy5akBnxaftwTYpjutT/n2M1i+kvuH+hE 0unuKPRmPBTG/QE2gwdW/5webRwoQT/Q2REWIV/aL+6OGC03FmMFYVAWnd2zQA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by vangyzen: URL: https://cgit.FreeBSD.org/src/commit/?id=94df301ac27e7df785d1d581a36dc14d456436c2 commit 94df301ac27e7df785d1d581a36dc14d456436c2 Author: Eric van Gyzen AuthorDate: 2021-04-26 15:01:17 +0000 Commit: Eric van Gyzen CommitDate: 2022-03-02 21:56:30 +0000 Wait longer for a previous IPI to be sent When sending an IPI, if a previous IPI is still pending delivery, native_lapic_ipi_vectored() waits for the previous IPI to be sent. We've seen a few inexplicable panics with the current timeout of 50 ms. Increase the timeout to 1 second and make it tunable. No hardware specification mentions a timeout in this case; I checked the Intel SDM, Intel MP spec, and Intel x2APIC spec. Linux and illumos wait forever. In Linux, see __default_send_IPI_shortcut() in arch/x86/kernel/apic/ipi.c. In illumos, see apic_send_ipi() in usr/src/uts/i86pc/io/pcplusmp/apic_common.c. However, misbehaving hardware could hang the system if we wait forever. Reviewed by: mav kib MFC after: 1 week Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D29942 (cherry picked from commit 2f32a971b7f936170b4d61318238e6cf89f198b5) --- sys/x86/x86/local_apic.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c index 59b5cb600725..bbe3a63157d4 100644 --- a/sys/x86/x86/local_apic.c +++ b/sys/x86/x86/local_apic.c @@ -204,6 +204,7 @@ static u_long lapic_timer_divisor, count_freq; static struct eventtimer lapic_et; #ifdef SMP static uint64_t lapic_ipi_wait_mult; +static int __read_mostly lapic_ds_idle_timeout = 1000000; #endif unsigned int max_apic_id; @@ -214,6 +215,11 @@ SYSCTL_INT(_hw_apic, OID_AUTO, eoi_suppression, CTLFLAG_RD, &lapic_eoi_suppression, 0, ""); SYSCTL_INT(_hw_apic, OID_AUTO, timer_tsc_deadline, CTLFLAG_RD, &lapic_timer_tsc_deadline, 0, ""); +#ifdef SMP +SYSCTL_INT(_hw_apic, OID_AUTO, ds_idle_timeout, CTLFLAG_RWTUN, + &lapic_ds_idle_timeout, 0, + "timeout (in us) for APIC Delivery Status to become Idle (xAPIC only)"); +#endif static void lapic_calibrate_initcount(struct lapic *la); @@ -2088,7 +2094,6 @@ native_lapic_ipi_raw(register_t icrlo, u_int dest) } } -#define BEFORE_SPIN 50000 #ifdef DETECT_DEADLOCK #define AFTER_SPIN 50 #endif @@ -2135,7 +2140,7 @@ native_lapic_ipi_vectored(u_int vector, int dest) icrlo |= APIC_DESTMODE_PHY | APIC_TRIGMOD_EDGE | APIC_LEVEL_ASSERT; /* Wait for an earlier IPI to finish. */ - if (!lapic_ipi_wait(BEFORE_SPIN)) { + if (!lapic_ipi_wait(lapic_ds_idle_timeout)) { if (KERNEL_PANICKED()) return; else