Date: Wed, 14 Dec 2016 18:26:27 +0300 From: Slawa Olhovchenkov <slw@zxy.spb.ru> To: Konstantin Belousov <kostikbel@gmail.com> Cc: freebsd-current@freebsd.org Subject: Re: Enabling NUMA in BIOS stop booting FreeBSD Message-ID: <20161214152627.GF98176@zxy.spb.ru> In-Reply-To: <20161214121336.GD98176@zxy.spb.ru> References: <20161213143401.GK90287@zxy.spb.ru> <20161213150139.GZ54029@kib.kiev.ua> <20161213152838.GL90287@zxy.spb.ru> <20161213172529.GC54029@kib.kiev.ua> <20161213174345.GB98176@zxy.spb.ru> <20161214095350.GE94325@kib.kiev.ua> <20161214102711.GF94325@kib.kiev.ua> <20161214105211.GC98176@zxy.spb.ru> <20161214113927.GG94325@kib.kiev.ua> <20161214121336.GD98176@zxy.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Dec 14, 2016 at 03:13:36PM +0300, Slawa Olhovchenkov wrote:
> On Wed, Dec 14, 2016 at 01:39:27PM +0200, Konstantin Belousov wrote:
>
> > In other words, it is almost certainly the hang and not a fault causing
> > hang. This means that the machine is not compliant with the IA32
> > architecture, in particular, the region reported as normal memory by
> > E820 BIOS service does not behave as normal memory.
> >
> > Since regardless of the option setting, the memory map is same, and
> > bootstrap page table only depend on the memory map, we use the same page
> > table when hanging and when operating correctly. We do not fault or hang
> > when the option is turned off, which together with the improved early
> > fault handling in the patch, makes it almost certain that the problem is
> > in hardware configuration and not in our early setup.
> >
> > Of course, the most puzzling part is that memory test makes the hang
> > go away, while repeating memory test operation only on the msgbuf region
> > does not. msgbuf is special in that it is located at TOHM (top of high
> > memory). It spans 128KB from below it to the last byte of the last
> > physical segment.
> >
> > The only ideas I have right now is that there is either a bug in the
> > Caching Agent/Home agent/IMC configuration in BIOS, in which case there
> > is nothing OS can do to mitigate it. Or it might be that the memory
> > map reported by CMS is wrong (you said that you use legacy boot, right
> > ?). This is not too surprising if true, because non-EFI boot code path
> > definitely get less and less testing.
> >
> > For the later case (potential bug in CMS), could you switch to EFI boot
> > mode and see whether the issue magically healths itself ? You could boot
> > from USB stick in EFI mode without reinstalling for test.
>
> I can't boot from USB stick -- this is remote DC and IPMI allow only
> CDROM emulation.
>
> OK, I am boot in UEFI 12.0 snapshot ISO.
> Boot ok.
Sorry. Overload bu work and test wrong combination (NUMA=ON,
interleave=OFF)
snapshot iso don't boot with NUMA=ON interleave=ON
For test hardware setup (NUMA+interleave), what ISO I can try to boot?
PS: memmaps:
NUMA=ON interleave=OFF
OK memmap
Type Physical Virtual #Pages Attr
BootServicesCode 000000000000 000000000000 00000008 UC WC WT WB
ConventionalMemory 000000008000 000000000000 00000027 UC WC WT WB
BootServicesData 00000002f000 000000000000 00000011 UC WC WT WB
BootServicesCode 000000040000 000000000000 00000060 UC WC WT WB
ConventionalMemory 000000100000 000000000000 000660a3 UC WC WT WB
BootServicesData 0000661a3000 000000000000 00000080 UC WC WT WB
ConventionalMemory 000066223000 000000000000 000076b8 UC WC WT WB
LoaderData 00006d8db000 000000000000 00008000 UC WC WT WB
LoaderCode 0000758db000 000000000000 00000070 UC WC WT WB
BootServicesData 00007594b000 000000000000 00003220 UC WC WT WB
ConventionalMemory 000078b6b000 000000000000 0000028e UC WC WT WB
BootServicesCode 000078df9000 000000000000 00000372 UC WC WT WB
Reserved 00007916b000 000000000000 00000817 UC WC WT WB
ConventionalMemory 000079982000 000000000000 0000011f UC WC WT WB
ACPIMemoryNVS 000079aa1000 000000000000 00000509 UC WC WT WB
RuntimeServicesData 000079faa000 000000000000 00001dbd UC WC WT WB
RuntimeServicesCode 00007bd67000 000000000000 00000061 UC WC WT WB
BootServicesData 00007bdc8000 000000000000 00000001 UC WC WT WB
RuntimeServicesData 00007bdc9000 000000000000 00000086 UC WC WT WB
BootServicesData 00007be4f000 000000000000 000001b1 UC WC WT WB
ConventionalMemory 000100000000 000000000000 01f80000 UC WC WT WB
Reserved 00007c000000 000000000000 00004000
MemoryMappedIO 000080000000 000000000000 00010000 UC
MemoryMappedIO 0000fed1c000 000000000000 00000029 UC
MemoryMappedIO 0000ff000000 000000000000 00001000 UC
NUMA=ON interleave=ON
Type Physical Virtual #Pages Attr
BootServicesCode 000000000000 000000000000 00000008 UC WC WT WB
ConventionalMemory 000000008000 000000000000 00000027 UC WC WT WB
BootServicesData 00000002f000 000000000000 00000011 UC WC WT WB
BootServicesCode 000000040000 000000000000 00000060 UC WC WT WB
ConventionalMemory 000000100000 000000000000 000660a3 UC WC WT WB
BootServicesData 0000661a3000 000000000000 00000080 UC WC WT WB
ConventionalMemory 000066223000 000000000000 000076b8 UC WC WT WB
LoaderData 00006d8db000 000000000000 00008000 UC WC WT WB
LoaderCode 0000758db000 000000000000 00000070 UC WC WT WB
BootServicesData 00007594b000 000000000000 00003220 UC WC WT WB
ConventionalMemory 000078b6b000 000000000000 0000028e UC WC WT WB
BootServicesCode 000078df9000 000000000000 00000372 UC WC WT WB
Reserved 00007916b000 000000000000 00000817 UC WC WT WB
ConventionalMemory 000079982000 000000000000 0000011f UC WC WT WB
ACPIMemoryNVS 000079aa1000 000000000000 00000509 UC WC WT WB
RuntimeServicesData 000079faa000 000000000000 00001dbd UC WC WT WB
RuntimeServicesCode 00007bd67000 000000000000 00000061 UC WC WT WB
BootServicesData 00007bdc8000 000000000000 00000001 UC WC WT WB
RuntimeServicesData 00007bdc9000 000000000000 00000086 UC WC WT WB
BootServicesData 00007be4f000 000000000000 000001b1 UC WC WT WB
ConventionalMemory 000100000000 000000000000 01f80000 UC WC WT WB
Reserved 00007c000000 000000000000 00004000
MemoryMappedIO 000080000000 000000000000 00010000 UC
MemoryMappedIO 0000fed1c000 000000000000 00000029 UC
MemoryMappedIO 0000ff000000 000000000000 00001000 UC
NUMA=OFF interleave=ON
Type Physical Virtual #Pages Attr
BootServicesCode 000000000000 000000000000 00000008 UC WC WT WB
ConventionalMemory 000000008000 000000000000 00000027 UC WC WT WB
BootServicesData 00000002f000 000000000000 00000011 UC WC WT WB
BootServicesCode 000000040000 000000000000 00000060 UC WC WT WB
ConventionalMemory 000000100000 000000000000 000660a3 UC WC WT WB
BootServicesData 0000661a3000 000000000000 00000080 UC WC WT WB
ConventionalMemory 000066223000 000000000000 000076b8 UC WC WT WB
LoaderData 00006d8db000 000000000000 00008000 UC WC WT WB
LoaderCode 0000758db000 000000000000 00000070 UC WC WT WB
BootServicesData 00007594b000 000000000000 00003220 UC WC WT WB
ConventionalMemory 000078b6b000 000000000000 0000028e UC WC WT WB
BootServicesCode 000078df9000 000000000000 00000372 UC WC WT WB
Reserved 00007916b000 000000000000 00000817 UC WC WT WB
ConventionalMemory 000079982000 000000000000 00000123 UC WC WT WB
ACPIMemoryNVS 000079aa5000 000000000000 00000505 UC WC WT WB
RuntimeServicesData 000079faa000 000000000000 00001dbd UC WC WT WB
RuntimeServicesCode 00007bd67000 000000000000 00000061 UC WC WT WB
BootServicesData 00007bdc8000 000000000000 00000001 UC WC WT WB
RuntimeServicesData 00007bdc9000 000000000000 00000086 UC WC WT WB
BootServicesData 00007be4f000 000000000000 000001b1 UC WC WT WB
ConventionalMemory 000100000000 000000000000 01f80000 UC WC WT WB
Reserved 00007c000000 000000000000 00004000
MemoryMappedIO 000080000000 000000000000 00010000 UC
MemoryMappedIO 0000fed1c000 000000000000 00000029 UC
MemoryMappedIO 0000ff000000 000000000000 00001000 UC
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20161214152627.GF98176>