From owner-freebsd-stable@freebsd.org Wed Sep 14 12:49:53 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5DA44BDA011 for ; Wed, 14 Sep 2016 12:49:53 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 485A819C9 for ; Wed, 14 Sep 2016 12:49:53 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: by mailman.ysv.freebsd.org (Postfix) id 4442BBDA010; Wed, 14 Sep 2016 12:49:53 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 43D88BDA00F for ; Wed, 14 Sep 2016 12:49:53 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0545B19C7; Wed, 14 Sep 2016 12:49:53 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1bk9dP-0000lW-1j; Wed, 14 Sep 2016 15:49:51 +0300 Date: Wed, 14 Sep 2016 15:49:51 +0300 From: Slawa Olhovchenkov To: Andriy Gapon Cc: Konstantin Belousov , stable@FreeBSD.org Subject: Re: X2APIC support Message-ID: <20160914124950.GE2840@zxy.spb.ru> References: <20160913124239.GZ34394@zxy.spb.ru> <20160913142118.GA34394@zxy.spb.ru> <37f5cebc-3fa1-9e95-5123-f3d8daa3130a@FreeBSD.org> <20160913152240.GE38409@kib.kiev.ua> <20160914113634.GF38409@kib.kiev.ua> <763df55a-4b69-9f7a-1042-0f631a729881@FreeBSD.org> <20160914123313.GC2840@zxy.spb.ru> <966c58d6-db64-2879-6709-0aa596bec24b@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <966c58d6-db64-2879-6709-0aa596bec24b@FreeBSD.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Sep 2016 12:49:53 -0000 On Wed, Sep 14, 2016 at 03:35:39PM +0300, Andriy Gapon wrote: > On 14/09/2016 15:33, Slawa Olhovchenkov wrote: > > On Wed, Sep 14, 2016 at 03:22:17PM +0300, Andriy Gapon wrote: > > > >> On 14/09/2016 14:36, Konstantin Belousov wrote: > >>> On Tue, Sep 13, 2016 at 06:52:19PM +0300, Andriy Gapon wrote: > >>>> On 13/09/2016 18:22, Konstantin Belousov wrote: > >>>>> Any access > >>>>> to the LAPIC registers page in x2APIC mode faults. > >>>> > >>>> Is this a fact? > >>>> I read the following in the specification: > >>>> > >>>> In x2APIC mode, the memory mapped interface is not available and any > >>>> access to the MMIO interface will behave similar to that of a legacy xAPIC > >>>> in globally disabled state. > >>>> > >>>> But I couldn't find what actually happens for the legacy xAPIC in globally > >>>> disabled state. For AMD processors it is documented that if xAPIC is disabled > >>>> then accessing the APIC memory range works the same as accessing the regular > >>>> memory. That can be different for Intel processors, of course. > >>> > >>> Look at the native_lapic_init(), very beginning of the function. If > >>> x2apic_mode is non-zero, lapic_map is cleared after DMAP page cache > >>> attributes are changed. > >> > >> Right, but we are talking about the case where x2apic_mode *is zero* while the > >> hardware in x2APIC mode. > >> > >>> Right now, if BIOS pass control to OS in x2APIC mode, thinks just work > >>> because no LAPIC accesses are done until native_lapic_enable_x2apic() is > >>> called. This just happens, I thought about more organized receipt of the > >>> current LAPIC mode. Issue is that decision for LAPIC mode is performed > >>> by madt enumerator which pref. should not read LAPIC config (too early). > >>> And it is not clear at all, what to do if there is reason to use xAPIC, > >>> as checked by madt_setup_local(), but we are in x2APIC mode already. > >> > >> Yes. As I mentioned earlier we should at least panic with an informative > >> message. Maybe we could add some code to so something smarter later. > >> > >>> Anyway, returning to the original issue, I do not believe that the > >>> hand-off on that machine occured in the x2APIC mode when X2APIC_OPT_OUT > >>> is specified. Kernel would fault in that case. > >> > >> I think that it should be easy to check that if Slawa is still willing to try > >> another diagnostic patch. > > > > What combination of BIOS setting need? > > The one that causes the crash. VT(vga): text 80x25 CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.05-MHz K8-class CPU) Origin="GenuineIntel" Id=0x406f1 Family=0x6 Model=0x4f Stepping=1 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x121 Structured Extended Features=0x21cbfbb XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 137438953472 (131072 MB) avail memory = 133407973376 (127227 MB) MSR_APICBASE = 0x fee00d00 x2APIC is prohibited but turned on by BIOS Event timer "LAPIC" quality 600 ACPI APIC Table: boot_cpu_id = 255 kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = ff fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80537e74 stack pointer = 0x28:0xffffffff814b3a60 frame pointer = 0x28:0xffffffff814b3a70 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 0 () trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xffffffff805272e7 at kdb_backtrace+0x67 #1 0xffffffff804dd662 at vpanic+0x182 #2 0xffffffff804dd4d3 at panic+0x43 #3 0xffffffff807a37a1 at trap_fatal+0x351 #4 0xffffffff807a3993 at trap_pfault+0x1e3 #5 0xffffffff807a2f1c at trap+0x26c #6 0xffffffff80787ca1 at calltrap+0x8 #7 0xffffffff8083b57a at topo_probe+0x61a #8 0xffffffff8078fe93 at cpu_mp_start+0x1c3 #9 0xffffffff805382ca at mp_start+0x3a #10 0xffffffff80465cd8 at mi_startup+0x118 #11 0xffffffff8028dfac at btext+0x2c Uptime: 1s > >> diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c > >> index 203e9d00e8acc..37ac03fb9d811 100644 > >> --- a/sys/x86/x86/local_apic.c > >> +++ b/sys/x86/x86/local_apic.c > >> @@ -429,6 +429,12 @@ native_lapic_init(vm_paddr_t addr) > >> if (x2apic_mode) { > >> native_lapic_enable_x2apic(); > >> lapic_map = NULL; > >> + } else if ((cpu_feature2 & CPUID2_X2APIC) != 0) { > >> + r = rdmsr(MSR_APICBASE); > >> + printf("MSR_APICBASE = 0x%16jx\n", (uintmax_t)r); > >> + if ((r & (APICBASE_X2APIC | APICBASE_ENABLED)) == > >> + (APICBASE_X2APIC | APICBASE_ENABLED)) > >> + printf("x2APIC is prohibited but turned on by BIOS\n"); > >> } > >> > >> /* Setup the spurious interrupt handler. */ > >> > >> > >> > >> -- > >> Andriy Gapon > > > -- > Andriy Gapon