Date: Wed, 14 Sep 2016 15:49:51 +0300 From: Slawa Olhovchenkov <slw@zxy.spb.ru> To: Andriy Gapon <avg@FreeBSD.org> Cc: Konstantin Belousov <kostikbel@gmail.com>, stable@FreeBSD.org Subject: Re: X2APIC support Message-ID: <20160914124950.GE2840@zxy.spb.ru> In-Reply-To: <966c58d6-db64-2879-6709-0aa596bec24b@FreeBSD.org> References: <20160913124239.GZ34394@zxy.spb.ru> <c6e1a647-344b-a6d7-9f48-65092533ed21@FreeBSD.org> <20160913142118.GA34394@zxy.spb.ru> <37f5cebc-3fa1-9e95-5123-f3d8daa3130a@FreeBSD.org> <20160913152240.GE38409@kib.kiev.ua> <cb35f671-95e7-820f-6a78-ee60612cc1ad@FreeBSD.org> <20160914113634.GF38409@kib.kiev.ua> <763df55a-4b69-9f7a-1042-0f631a729881@FreeBSD.org> <20160914123313.GC2840@zxy.spb.ru> <966c58d6-db64-2879-6709-0aa596bec24b@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Sep 14, 2016 at 03:35:39PM +0300, Andriy Gapon wrote:
> On 14/09/2016 15:33, Slawa Olhovchenkov wrote:
> > On Wed, Sep 14, 2016 at 03:22:17PM +0300, Andriy Gapon wrote:
> >
> >> On 14/09/2016 14:36, Konstantin Belousov wrote:
> >>> On Tue, Sep 13, 2016 at 06:52:19PM +0300, Andriy Gapon wrote:
> >>>> On 13/09/2016 18:22, Konstantin Belousov wrote:
> >>>>> Any access
> >>>>> to the LAPIC registers page in x2APIC mode faults.
> >>>>
> >>>> Is this a fact?
> >>>> I read the following in the specification:
> >>>>
> >>>> In x2APIC mode, the memory mapped interface is not available and any
> >>>> access to the MMIO interface will behave similar to that of a legacy xAPIC
> >>>> in globally disabled state.
> >>>>
> >>>> But I couldn't find what actually happens for the legacy xAPIC in globally
> >>>> disabled state. For AMD processors it is documented that if xAPIC is disabled
> >>>> then accessing the APIC memory range works the same as accessing the regular
> >>>> memory. That can be different for Intel processors, of course.
> >>>
> >>> Look at the native_lapic_init(), very beginning of the function. If
> >>> x2apic_mode is non-zero, lapic_map is cleared after DMAP page cache
> >>> attributes are changed.
> >>
> >> Right, but we are talking about the case where x2apic_mode *is zero* while the
> >> hardware in x2APIC mode.
> >>
> >>> Right now, if BIOS pass control to OS in x2APIC mode, thinks just work
> >>> because no LAPIC accesses are done until native_lapic_enable_x2apic() is
> >>> called. This just happens, I thought about more organized receipt of the
> >>> current LAPIC mode. Issue is that decision for LAPIC mode is performed
> >>> by madt enumerator which pref. should not read LAPIC config (too early).
> >>> And it is not clear at all, what to do if there is reason to use xAPIC,
> >>> as checked by madt_setup_local(), but we are in x2APIC mode already.
> >>
> >> Yes. As I mentioned earlier we should at least panic with an informative
> >> message. Maybe we could add some code to so something smarter later.
> >>
> >>> Anyway, returning to the original issue, I do not believe that the
> >>> hand-off on that machine occured in the x2APIC mode when X2APIC_OPT_OUT
> >>> is specified. Kernel would fault in that case.
> >>
> >> I think that it should be easy to check that if Slawa is still willing to try
> >> another diagnostic patch.
> >
> > What combination of BIOS setting need?
>
> The one that causes the crash.
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.05-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x406f1 Family=0x6 Model=0x4f Stepping=1
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x21cbfbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,PQE,RDSEED,ADX,SMAP,PROCTRACE>
XSAVE Features=0x1<XSAVEOPT>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
TSC: P-state invariant, performance statistics
real memory = 137438953472 (131072 MB)
avail memory = 133407973376 (127227 MB)
MSR_APICBASE = 0x fee00d00
x2APIC is prohibited but turned on by BIOS
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
boot_cpu_id = 255
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = ff
fault virtual address = 0x0
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80537e74
stack pointer = 0x28:0xffffffff814b3a60
frame pointer = 0x28:0xffffffff814b3a70
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 0 ()
trap number = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff805272e7 at kdb_backtrace+0x67
#1 0xffffffff804dd662 at vpanic+0x182
#2 0xffffffff804dd4d3 at panic+0x43
#3 0xffffffff807a37a1 at trap_fatal+0x351
#4 0xffffffff807a3993 at trap_pfault+0x1e3
#5 0xffffffff807a2f1c at trap+0x26c
#6 0xffffffff80787ca1 at calltrap+0x8
#7 0xffffffff8083b57a at topo_probe+0x61a
#8 0xffffffff8078fe93 at cpu_mp_start+0x1c3
#9 0xffffffff805382ca at mp_start+0x3a
#10 0xffffffff80465cd8 at mi_startup+0x118
#11 0xffffffff8028dfac at btext+0x2c
Uptime: 1s
> >> diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c
> >> index 203e9d00e8acc..37ac03fb9d811 100644
> >> --- a/sys/x86/x86/local_apic.c
> >> +++ b/sys/x86/x86/local_apic.c
> >> @@ -429,6 +429,12 @@ native_lapic_init(vm_paddr_t addr)
> >> if (x2apic_mode) {
> >> native_lapic_enable_x2apic();
> >> lapic_map = NULL;
> >> + } else if ((cpu_feature2 & CPUID2_X2APIC) != 0) {
> >> + r = rdmsr(MSR_APICBASE);
> >> + printf("MSR_APICBASE = 0x%16jx\n", (uintmax_t)r);
> >> + if ((r & (APICBASE_X2APIC | APICBASE_ENABLED)) ==
> >> + (APICBASE_X2APIC | APICBASE_ENABLED))
> >> + printf("x2APIC is prohibited but turned on by BIOS\n");
> >> }
> >>
> >> /* Setup the spurious interrupt handler. */
> >>
> >>
> >>
> >> --
> >> Andriy Gapon
>
>
> --
> Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160914124950.GE2840>