From owner-svn-src-head@freebsd.org Tue Mar 29 08:44:58 2016 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2C8B5AE16C0; Tue, 29 Mar 2016 08:44:58 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EEFB615ED; Tue, 29 Mar 2016 08:44:57 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id u2T8ivUO030046; Tue, 29 Mar 2016 08:44:57 GMT (envelope-from kib@FreeBSD.org) Received: (from kib@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id u2T8ivrN030045; Tue, 29 Mar 2016 08:44:57 GMT (envelope-from kib@FreeBSD.org) Message-Id: <201603290844.u2T8ivrN030045@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: kib set sender to kib@FreeBSD.org using -f From: Konstantin Belousov Date: Tue, 29 Mar 2016 08:44:57 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r297374 - head/sys/x86/x86 X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2016 08:44:58 -0000 Author: kib Date: Tue Mar 29 08:44:56 2016 New Revision: 297374 URL: https://svnweb.freebsd.org/changeset/base/297374 Log: Calibrate the frequency of the of the native_lapic_ipi_wait() loop, and avoid a delay while waiting for IPI delivery acknowledgement in xAPIC mode. This makes the loop exit immediately after the delivery bit in APIC_ICR register is set, instead of waiting for some microseconds. We only need to ensure that some amount of time is allowed for the LAPIC to react to the command, and we need that the wait time is finite and reasonable. For that reasons, it is irrelevant if the CPU frequency or throttling decrease the speed and make the loop, calibrated for full CPU speed at boot time, execute somewhat slower. Discussed with: bde, jhb Tested by: pho Sponsored by: The FreeBSD Foundation Modified: head/sys/x86/x86/local_apic.c Modified: head/sys/x86/x86/local_apic.c ============================================================================== --- head/sys/x86/x86/local_apic.c Tue Mar 29 08:31:34 2016 (r297373) +++ head/sys/x86/x86/local_apic.c Tue Mar 29 08:44:56 2016 (r297374) @@ -172,6 +172,7 @@ int lapic_eoi_suppression; static int lapic_timer_tsc_deadline; static u_long lapic_timer_divisor; static struct eventtimer lapic_et; +static uint64_t lapic_ipi_wait_mult; SYSCTL_NODE(_hw, OID_AUTO, apic, CTLFLAG_RD, 0, "APIC options"); SYSCTL_INT(_hw_apic, OID_AUTO, x2apic_mode, CTLFLAG_RD, &x2apic_mode, 0, ""); @@ -403,6 +404,7 @@ lvt_mode(struct lapic *la, u_int pin, ui static void native_lapic_init(vm_paddr_t addr) { + uint64_t r; uint32_t ver; u_int regs[4]; int i, arat; @@ -503,6 +505,34 @@ native_lapic_init(vm_paddr_t addr) TUNABLE_INT_FETCH("hw.lapic_eoi_suppression", &lapic_eoi_suppression); } + +#define LOOPS 1000000 + /* + * Calibrate the busy loop waiting for IPI ack in xAPIC mode. + * lapic_ipi_wait_mult contains the number of iterations which + * approximately delay execution for 1 microsecond (the + * argument to native_lapic_ipi_wait() is in microseconds). + * + * We assume that TSC is present and already measured. + * Possible TSC frequency jumps are irrelevant to the + * calibration loop below, the CPU clock management code is + * not yet started, and we do not enter sleep states. + */ + KASSERT((cpu_feature & CPUID_TSC) != 0 && tsc_freq != 0, + ("TSC not initialized")); + r = rdtsc(); + for (r = 0; r < LOOPS; r++) { + (void)lapic_read_icr_lo(); + ia32_pause(); + } + r = rdtsc() - r; + lapic_ipi_wait_mult = (r * 1000000) / tsc_freq / LOOPS; + if (bootverbose) { + printf("LAPIC: ipi_wait() us multiplier %jd (r %jd tsc %jd)\n", + (uintmax_t)lapic_ipi_wait_mult, (uintmax_t)r, + (uintmax_t)tsc_freq); + } +#undef LOOPS } /* @@ -1716,31 +1746,26 @@ SYSINIT(apic_setup_io, SI_SUB_INTR, SI_O * private to the MD code. The public interface for the rest of the * kernel is defined in mp_machdep.c. */ + +/* + * Wait delay microseconds for IPI to be sent. If delay is -1, we + * wait forever. + */ static int native_lapic_ipi_wait(int delay) { - int x; + uint64_t i, counter; /* LAPIC_ICR.APIC_DELSTAT_MASK is undefined in x2APIC mode */ - if (x2apic_mode) + if (x2apic_mode || delay == -1) return (1); - /* - * Wait delay microseconds for IPI to be sent. If delay is - * -1, we wait forever. - */ - if (delay == -1) { - while ((lapic_read_icr_lo() & APIC_DELSTAT_MASK) != - APIC_DELSTAT_IDLE) - ia32_pause(); - return (1); - } - - for (x = 0; x < delay; x++) { + counter = lapic_ipi_wait_mult * delay; + for (i = 0; i < counter; i++) { if ((lapic_read_icr_lo() & APIC_DELSTAT_MASK) == APIC_DELSTAT_IDLE) return (1); - DELAY(1); + ia32_pause(); } return (0); }