From owner-svn-src-stable-10@freebsd.org Wed Apr 27 19:12:50 2016 Return-Path: Delivered-To: svn-src-stable-10@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BCFDBB1F507; Wed, 27 Apr 2016 19:12:50 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7BCF71116; Wed, 27 Apr 2016 19:12:50 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id u3RJCnDY079965; Wed, 27 Apr 2016 19:12:49 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id u3RJCn1l079964; Wed, 27 Apr 2016 19:12:49 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201604271912.u3RJCn1l079964@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Wed, 27 Apr 2016 19:12:49 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r298715 - stable/10/sys/x86/x86 X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable-10@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: SVN commit messages for only the 10-stable src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 19:12:50 -0000 Author: jhb Date: Wed Apr 27 19:12:49 2016 New Revision: 298715 URL: https://svnweb.freebsd.org/changeset/base/298715 Log: MFC 297039,297374,297398,297484: Poll the IPI status while waiting constantly instead of delaying 5 microseconds between checks. This avoids inserting a minimum latency of 5 microseconds on each IPI. 297039: Check IPI status more frequently when waiting. An IPI cannot be sent via the local APIC if a previous IPI is still being delivered. Attempts to send an IPI will wait for a pending IPI to clear. Prior to r278325 these checks used a spin loop with a hardcoded maximum count which broke AP startup on some systems. However, r278325 also enforced a minimum latency of 5 microseconds if an IPI was still pending which resulted in a measurable performance hit. This change reduces that minimum latency to 1 microsecond. 297374: Calibrate the frequency of the of the native_lapic_ipi_wait() loop, and avoid a delay while waiting for IPI delivery acknowledgement in xAPIC mode. This makes the loop exit immediately after the delivery bit in APIC_ICR register is set, instead of waiting for some microseconds. We only need to ensure that some amount of time is allowed for the LAPIC to react to the command, and we need that the wait time is finite and reasonable. For that reasons, it is irrelevant if the CPU frequency or throttling decrease the speed and make the loop, calibrated for full CPU speed at boot time, execute somewhat slower. 297398: Fix several bugs in r297374: - fix UP build [1] - do not obliterate initial reading of rdtsc by the loop counter [2] - restore the meaning of the argument -1 to native_lapic_ipi_wait() as wait until LAPIC acknowledge without timeout - correct formula for calculating loop iteration count for 1us, it was inverted, and ensure that even on unlikely slow CPUs at least one check for ack is performed. 297484: Style(9), use tabs for the #define LOOPS line. Print unsigned values with %u. Make code slightly more compact by inlining loop limit. Modified: stable/10/sys/x86/x86/local_apic.c Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/x86/x86/local_apic.c ============================================================================== --- stable/10/sys/x86/x86/local_apic.c Wed Apr 27 19:09:21 2016 (r298714) +++ stable/10/sys/x86/x86/local_apic.c Wed Apr 27 19:12:49 2016 (r298715) @@ -56,6 +56,7 @@ __FBSDID("$FreeBSD$"); #include #include +#include #include #include #include @@ -158,6 +159,9 @@ volatile lapic_t *lapic; vm_paddr_t lapic_paddr; static u_long lapic_timer_divisor; static struct eventtimer lapic_et; +#ifdef SMP +static uint64_t lapic_ipi_wait_mult; +#endif static void lapic_enable(void); static void lapic_resume(struct pic *pic, bool suspend_cancelled); @@ -221,6 +225,9 @@ lvt_mode(struct lapic *la, u_int pin, ui void lapic_init(vm_paddr_t addr) { +#ifdef SMP + uint64_t r, r1, r2, rx; +#endif u_int regs[4]; int i, arat; @@ -275,6 +282,38 @@ lapic_init(vm_paddr_t addr) lapic_et.et_priv = NULL; et_register(&lapic_et); } + +#ifdef SMP +#define LOOPS 1000000 + /* + * Calibrate the busy loop waiting for IPI ack in xAPIC mode. + * lapic_ipi_wait_mult contains the number of iterations which + * approximately delay execution for 1 microsecond (the + * argument to native_lapic_ipi_wait() is in microseconds). + * + * We assume that TSC is present and already measured. + * Possible TSC frequency jumps are irrelevant to the + * calibration loop below, the CPU clock management code is + * not yet started, and we do not enter sleep states. + */ + KASSERT((cpu_feature & CPUID_TSC) != 0 && tsc_freq != 0, + ("TSC not initialized")); + r = rdtsc(); + for (rx = 0; rx < LOOPS; rx++) { + (void)lapic->icr_lo; + ia32_pause(); + } + r = rdtsc() - r; + r1 = tsc_freq * LOOPS; + r2 = r * 1000000; + lapic_ipi_wait_mult = r1 >= r2 ? r1 / r2 : 1; + if (bootverbose) { + printf("LAPIC: ipi_wait() us multiplier %ju (r %ju tsc %ju)\n", + (uintmax_t)lapic_ipi_wait_mult, (uintmax_t)r, + (uintmax_t)tsc_freq); + } +#undef LOOPS +#endif /* SMP */ } /* @@ -1381,25 +1420,20 @@ SYSINIT(apic_setup_io, SI_SUB_INTR, SI_O * private to the MD code. The public interface for the rest of the * kernel is defined in mp_machdep.c. */ + +/* + * Wait delay microseconds for IPI to be sent. If delay is -1, we + * wait forever. + */ int lapic_ipi_wait(int delay) { - int x; - - /* - * Wait delay microseconds for IPI to be sent. If delay is - * -1, we wait forever. - */ - if (delay == -1) { - while ((lapic->icr_lo & APIC_DELSTAT_MASK) != APIC_DELSTAT_IDLE) - ia32_pause(); - return (1); - } + uint64_t rx; - for (x = 0; x < delay; x += 5) { + for (rx = 0; delay == -1 || rx < lapic_ipi_wait_mult * delay; rx++) { if ((lapic->icr_lo & APIC_DELSTAT_MASK) == APIC_DELSTAT_IDLE) return (1); - DELAY(5); + ia32_pause(); } return (0); }