Date: Sun, 11 Dec 2016 23:47:09 +0300 From: Slawa Olhovchenkov <slw@zxy.spb.ru> To: Konstantin Belousov <kostikbel@gmail.com> Cc: current@freebsd.org Subject: Re: Enabling NUMA in BIOS stop booting FreeBSD Message-ID: <20161211204709.GK31311@zxy.spb.ru> In-Reply-To: <20161211200654.GA54029@kib.kiev.ua> References: <20161126092124.GM57876@zxy.spb.ru> <20161126155747.GF54029@kib.kiev.ua> <20161211182111.GH31311@zxy.spb.ru> <20161211191626.GI31311@zxy.spb.ru> <20161211192656.GZ54029@kib.kiev.ua> <20161211194559.GJ31311@zxy.spb.ru> <20161211200654.GA54029@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Dec 11, 2016 at 10:06:54PM +0200, Konstantin Belousov wrote:
> On Sun, Dec 11, 2016 at 10:45:59PM +0300, Slawa Olhovchenkov wrote:
> > On Sun, Dec 11, 2016 at 09:26:56PM +0200, Konstantin Belousov wrote:
> >
> > > On Sun, Dec 11, 2016 at 10:16:26PM +0300, Slawa Olhovchenkov wrote:
> > > > On Sun, Dec 11, 2016 at 09:21:11PM +0300, Slawa Olhovchenkov wrote:
> > > >
> > > > > On Sat, Nov 26, 2016 at 05:57:47PM +0200, Konstantin Belousov wrote:
> > > > >
> > > > > > On Sat, Nov 26, 2016 at 12:21:24PM +0300, Slawa Olhovchenkov wrote:
> > > > > > > I am try to enable NUMA in bios and can't boot FreeBSD.
> > > > > > > Boot stoped after next messages:
> > > > > > >
> > > > > > > ===
> > > > > > > Booting...
> > > > > > > KDB: debugger backends: ddb
> > > > > > > KDB: current backend: ddb
> > > > > > So at least the hammer_time() has a chance to initialize the console.
> > > > > > Do you have serial console ? Set the loader tunable debug.late_console
> > > > > > to 1 and see if any NMI reaction appear.
> > > > > >
> > > > > > > ===
> > > > > > >
> > > > > > > This is verbose boot.
> > > > > > > No reaction to ~^B, NMI.
> > > > > > >
> > > > > > > Same for head and 10.3-RELEASE.
> > > > > > >
> > > > > > > Hardware is Supermicro X10DRi, Dual E5-2650v4, 256GB RAM.
> > > > > > Is there a BIOS option for 'on-chip cluster' or 'HPC computing' ?
> > > > > > What if you try to frob it ?
> > > > > >
> > > > > > >
> > > > > > > On slight different hardware
> > > > > > > (Supermicro X10DRi w/ old BIOS, Dual E5-2640v3, 128GB RAM)
> > > > > > > 10.3 boot ok w/ BIOS NUMA enabled.
> > > > > >
> > > > > > I think the only way to debug this is to add printf() lines to hammer_time()
> > > > > > to see where does it break. Note that amd64_kdb_init() call succeeded,
> > > > > > so you can start bisect the code from there.
> > > > > >
> > > > >
> > > > > Hang in next two lines:
> > > > >
> > > > > msgbufinit(msgbufp, msgbufsize);
> > > > > fpuinit();
> > >
> > > Can you show the verbose dmesg up to the failure point ?
> > > In particular, the SMAP lines should be relevant.
> >
> > KDB: debugger backends: ddb
> > KDB: current backend: ddb
> > exit from kdb_init
> > KDB: enter: Boot flags requested debugger
> > [ thread pid 0 tid 0 ]
> > Stopped at 0xffffffff805361eb = kdb_enter+0x3b: movq
> > $0,0xffffffff80dcef20 = kdb_why
> >
> > No SMAP print, boot_verbose enabled.
> The log above shows that you used boot -d. What are the pristine boot
> messages, with debug.late_console set to 0, of course ?
This is stable/11, no debug.late_console.
With ANSI ESC, captured from SOL:
ESC[01;00HType '?' for a /boot/kernel.VSTREAM/opensolaris.ko size 0xcb10 at 0x13d3000
ESC[01;00HOK smap /boot/kernel.VSTREAM/if_igb.ko size 0x69f10 at 0x13e0000
ESC[02;00HSMAP type=02 base=000000000009ESC[01;00HSMAP type=01 base=0000000000000000 len=0000000000099c00 attr=01 ESC[02;00HSMAP type=02 base=0000000000099c00 len=0000000000006400 attr=01 ESC[03;00Hcan't find 'if_ixgbe'
ESC[01;00H/boot/kernel.VSTREAM/if_lagg.ko size 0x150c0 at 0x144a000
ESC[01;00HSMAP type=02 base=00000000000e/boot/kernel.VSTREAM/ukbd.ko size 0xe280 at 0x1460000
loading required moduleESC[01;00H 'usb'^MSMAP type=01 base=000000007916b000 len=0000000000936000 attr
=01 ESC[02;00H/boot/kernel.VSTREAM/usb.ko size 0x45d40 at 0x146f000
ESC[01;00HSMAP type=04 base=0000000079aa1000 len=000000/boot/kernel.VSTREAM/umass.ko size 0xaa10 at 0x14b5000
ESC[01;00HSMAP type=02 base=0000000079faa000 len=0000000002056000 attr=01 ESC[02;00HSMAP type=01 ba/boot/kernel.VSTREAM/accf_http.ko size 0x2710 at 0x14c0000
ESC[01;00HSMAP type=01 base=0000000100000000 len=0000001f80000000 attr=01 /boot/kernel.VSTREAM/sfxge.ko ESC[02;00HSMAP type=02 base=000000007c000000 len=0000000014000000 attr=01 size 0x1a8ee0 at 0x14c3000
ESC[03;00HSMAP type=02 base=00000000ff000000 len=0000000001000000 attr/boot/kernel.VSTREAM/uhci.ko size 0xd448 at 0x166c000
ESC[01;00HSMAP type=02 base=00000000fed1c000 len=0000000000029000 attr=01 /boot/kernel.VSTREAM/ohci.ko size 0xc900 at 0x167a000
ESC[02;00HOK memmap ESC[03;00Hmemmap not foun/boot/kernel.VSTREAM/ehci.ko size 0xfb60 at 0x1687000
ESC[01;00HOK memmap /boot/kernel.VSTREAM/xhci.ko size 0x11010 at 0x1697000
ESC[01;00Hmemmap not found /boot/kernel.VSTREAM/if_ix.ko size 0x51358 at 0x16a9000^MESC[01;00HOK boot
ESC[01;00HOK boot /boot/kernel.VSTREAM/cc_htcp.ko size 0x3a70 at 0x16fb000
Booting...
ESC[01;00H8+0x8+0xe9bdc] KDB: debugger backends: ddb
KDB: current backend: ddb
exit from kdb_init
KDB: enter: Boot flags requested debugger
[ thread pid 0 tid 0 ]
> Hm, might be also show the output of the 'smap' and 'memmap' output from
> the loader. If any of them worked, could be useful to see the same output
> with the NUMA option disabled as well.
NUMA disabled:
OK smap
SMAP type=01 base=0000000000000000 len=0000000000099c00 attr=01
SMAP type=02 base=0000000000099c00 len=0000000000006400 attr=01
SMAP type=02 base=00000000000e0000 len=0000000000020000 attr=01
SMAP type=01 base=0000000000100000 len=000000007906b000 attr=01
SMAP type=02 base=000000007916b000 len=000000000093a000 attr=01
SMAP type=04 base=0000000079aa5000 len=0000000000505000 attr=01
SMAP type=02 base=0000000079faa000 len=0000000002056000 attr=01
SMAP type=01 base=0000000100000000 len=0000001f80000000 attr=01
SMAP type=02 base=000000007c000000 len=0000000014000000 attr=01
SMAP type=02 base=00000000fed1c000 len=0000000000029000 attr=01
SMAP type=02 base=00000000ff000000 len=0000000001000000 attr=01
OK memmap
memmap not found
NUMA enabled:
OK smap
SMAP type=01 base=0000000000000000 len=0000000000099c00 attr=01
SMAP type=02 base=0000000000099c00 len=0000000000006400 attr=01
SMAP type=02 base=00000000000e0000 len=0000000000020000 attr=01
SMAP type=01 base=0000000000100000 len=000000007906b000 attr=01
SMAP type=02 base=000000007916b000 len=0000000000936000 attr=01
SMAP type=04 base=0000000079aa1000 len=0000000000509000 attr=01
SMAP type=02 base=0000000079faa000 len=0000000002056000 attr=01
SMAP type=01 base=0000000100000000 len=0000001f80000000 attr=01
SMAP type=02 base=000000007c000000 len=0000000014000000 attr=01
SMAP type=02 base=00000000fed1c000 len=0000000000029000 attr=01
SMAP type=02 base=00000000ff000000 len=0000000001000000 attr=01
OK memmap
memmap not found
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20161211204709.GK31311>