Date: Fri, 4 Dec 2009 10:35:59 -0500 From: John Baldwin <jhb@freebsd.org> To: Hiroki Sato <hrs@freebsd.org> Cc: freebsd-stable@freebsd.org Subject: Re: loader(8) readin failed on 7.2R and later including 8.0R Message-ID: <200912041035.59173.jhb@freebsd.org> In-Reply-To: <20091204.062008.155444535.hrs@allbsd.org> References: <200912020948.05698.jhb@freebsd.org> <200912030803.29797.jhb@freebsd.org> <20091204.062008.155444535.hrs@allbsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--Boundary-00=_fxSGLLhaI8YOBsF Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit On Thursday 03 December 2009 4:20:08 pm Hiroki Sato wrote: > John Baldwin <jhb@freebsd.org> wrote > in <200912030803.29797.jhb@freebsd.org>: > > jh> On Thursday 03 December 2009 5:29:13 am Hiroki Sato wrote: > jh> > John Baldwin <jhb@freebsd.org> wrote > jh> > in <200912020948.05698.jhb@freebsd.org>: > jh> > > jh> > jh> On Tuesday 01 December 2009 12:13:39 pm Hiroki Sato wrote: > jh> > jh> > While the "load" command seemed to finish, the box got stuck just > jh> > jh> > after entering "boot" command. > jh> > jh> > > jh> > jh> > Curious to say, I have got this symptom only on a specific box in > jh> > jh> > more than ten different boxes I upgraded so far; it is based on an > jh> > jh> > old motherboard Supermicro P4DPE[*]. > jh> > jh> > > jh> > jh> > [*] > jh> http://www.supermicro.com/products/motherboard/Xeon/E7500/P4DPE.cfm > jh> > jh> > > jh> > jh> > Any workaround? Booting from release CDROMs (7.2R and 8.0R) also > jh> > jh> > fail. On the box "7.1R" or "7.1R's loader + 7.2R kernel" worked > jh> > jh> > fine. It is possible something in changes of loader(8) between 7.1R > jh> > jh> > and 7.2R is the cause, but I am still not sure what it is... > jh> > jh> > jh> > jh> It may be related to the loader switching to using memory > 1MB for its > jh> > jh> malloc(). Maybe try building the loader with > jh> 'LOADER_NO_GPT_SUPPORT=yes' in > jh> > jh> /etc/src.conf? > jh> > > jh> > Thanks, a recompiled loader with LOADER_NO_GPT_SUPPORT=yes' displayed > jh> > "elf32_loadimage: could not read symbols - skipped!" for 8.0R kernel. > jh> > This is the same as 7.1R's loader + 8.0R kernel case. > jh> > jh> Can you get the output of 'smap' from the loader? Is the 8.0 kernel bigger > jh> than the 7.x kernel? If so, can you try trimming the 8.0 kernel a bit to see > jh> if that changes things? > > Sure. Output of smap on an 8.0R loader with LOADER_NO_GPT_SUPPORT=yes > was: > > | OK smap > | SMAP type=01 base=0000000000000000 len=000000000009f400 > | SMAP type=02 base=000000000009f400 len=0000000000000c00 > | SMAP type=02 base=00000000000dc000 len=0000000000024000 > | SMAP type=01 base=0000000000100000 len=0000000000e00000 So this is the region that ends up getting used for malloc: /* look for the first segment in 'extended' memory */ if ((smap.type == SMAP_TYPE_MEMORY) && (smap.base == 0x100000)) { bios_extmem = smap.length; ... /* Set memtop to actual top of memory */ memtop = memtop_copyin = 0x100000 + bios_extmem; and then later: #if defined(LOADER_BZIP2_SUPPORT) || defined(LOADER_FIREWIRE_SUPPORT) || defined(LOADER_GPT_SUPPORT) || defined(LOADER_ZFS_SUPPORT) heap_top = PTOV(memtop_copyin); memtop_copyin -= 0x300000; heap_bottom = PTOV(memtop_copyin); #else So memtop_copyin would start off as 0xf00000 but would end up as 0xc00000, and since the kernel starts at 4MB, I think that only leaves about 8MB for the kernel. Probably the loader needs to be more intelligent about using high memory for malloc by using the largest region > 1MB but < 4GB for malloc() instead of stealing memory from bios_extmem in the SMAP case. Try the attached patch which tries to make the loader use better smarts when picking a memory region for the heap (warning, I haven't tested it myself yet). > | SMAP type=02 base=0000000000f00000 len=0000000000100000 > | SMAP type=01 base=0000000001000000 len=00000000beef0000 > | SMAP type=03 base=00000000bfef0000 len=000000000000c000 > | SMAP type=04 base=00000000bfefc000 len=0000000000004000 > | SMAP type=01 base=00000000bff00000 len=0000000000080000 > | SMAP type=02 base=00000000bff80000 len=0000000000080000 > | SMAP type=02 base=00000000fec00000 len=0000000000010000 > | SMAP type=02 base=00000000fee00000 len=0000000000001000 > | SMAP type=02 base=00000000ff800000 len=0000000000400000 > | SMAP type=02 base=00000000fff00000 len=0000000000100000 > | OK > > Size difference between the two kernels was: > > | -r-xr-xr-x 1 root wheel 9708240 Dec 1 16:22 kernel.7/kernel > | -r-xr-xr-x 1 root wheel 11492703 Nov 21 15:48 kernel.8/kernel > > Then I rebuilt a smaller 8.0 kernel by removing some entries from the > kernel configuration file. The size is now smaller than 7.1R kernel: > > | -r-xr-xr-x 1 root wheel 7710491 Dec 3 21:10 /boot/kernel.8X/kernel > > Loading the new kernel seemed to work fine with the recompiled 8.0R > loader, but it got stuck just after entering "boot": > > | OK load /boot/kernel.8X/kernel > | /boot/kernel.8X/kernel text=0x5a7664 data=0x88d74+0x82f04 syms=[0x4+0x6d290+0x4+0x987e3] > | OK boot > | / I'm not sure why it would get stuck. Can you add some debug printfs to see how far it gets before it dies? E.g. does it get to the point of calling exec() (in which case the hang is in the kernel in locore.S rather than in the loader). -- John Baldwin --Boundary-00=_fxSGLLhaI8YOBsF Content-Type: text/x-patch; charset="ISO-8859-1"; name="loader_heap.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="loader_heap.patch" --- //depot/vendor/freebsd/src/sys/boot/i386/libi386/biosmem.c 2007/10/28 21:26:35 +++ //depot/user/jhb/boot/sys/boot/i386/libi386/biosmem.c 2009/12/04 15:33:59 @@ -35,14 +35,27 @@ #include "libi386.h" #include "btxv86.h" -vm_offset_t memtop, memtop_copyin; -u_int32_t bios_basemem, bios_extmem; +vm_offset_t memtop, memtop_copyin, high_heap_base; +uint32_t bios_basemem, bios_extmem, high_heap_size; static struct bios_smap smap; +/* + * The minimum amount of memory to reserve in bios_extmem for the kernel + * and modules. + */ +#define KERNEL_SPACE (16 * 1024 * 1024) + +/* + * The minimum amount of memory to reserve in bios_extmem for the heap. + */ +#define HEAP_MIN (3 * 1024 * 1024) + void bios_getmem(void) { + vm_offset_t base; + uint32_t size; /* Parse system memory map */ v86.ebx = 0; @@ -65,6 +78,38 @@ if ((smap.type == SMAP_TYPE_MEMORY) && (smap.base == 0x100000)) { bios_extmem = smap.length; } + + /* Look for the largest segment in 'extended' memory below 4GB. */ + if ((smap.type == SMAP_TYPE_MEMORY) && (smap.base > 0x100000) && + (smap.base < 0x100000000ull)) { + /* + * We want to preserve at least 16MB for kernel and modules, + * so take that into account for the first extended segment. + */ + if (smap.base == 0x100000) { + if (smap.length < KERNEL_SPACE) + size = 0; + else + size = smap.length - KERNEL_SPACE; + base = smap.base + KERNEL_SPACE; + } else { + base = smap.base; + size = smap.length; + } + + /* + * If this segment crosses the 4GB boundary, ignore it. Note + * that we also have to factor in the PTOV() offset as well as + * we don't want the virtual address to exceed 4GB. + */ + if ((uint64_t)base + size > VTOP(0x100000000ull)) + size -= ((uint64_t)base + size - VTOP(0x100000000ull)); + + if (size > high_heap_size) { + high_heap_size = size; + high_heap_base = base; + } + } } while (v86.ebx != 0); /* Fall back to the old compatibility function for base memory */ @@ -97,5 +142,13 @@ /* Set memtop to actual top of memory */ memtop = memtop_copyin = 0x100000 + bios_extmem; + /* + * If we have extended memory and did not find a suitable heap + * region in the SMAP, use the last 3MB of 'extended' memory as a + * high heap candidate. + */ + if (bios_extmem >= HEAP_MIN && high_heap_size == 0) { + high_heap_size = HEAP_MIN; + high_heap_base = memtop - HEAP_MIN; + } } - --- //depot/vendor/freebsd/src/sys/boot/i386/libi386/libi386.h 2009/03/12 20:45:15 +++ //depot/user/jhb/boot/sys/boot/i386/libi386/libi386.h 2009/12/04 15:33:59 @@ -78,9 +78,9 @@ int bc_getdev(struct i386_devdesc *dev); /* return dev_t for (dev) */ int bc_bios2unit(int biosdev); /* xlate BIOS device -> bioscd unit */ int bc_unit2bios(int unit); /* xlate bioscd unit -> BIOS device */ -u_int32_t bd_getbigeom(int bunit); /* return geometry in bootinfo format */ -int bd_bios2unit(int biosdev); /* xlate BIOS device -> biosdisk unit */ -int bd_unit2bios(int unit); /* xlate biosdisk unit -> BIOS device */ +uint32_t bd_getbigeom(int bunit); /* return geometry in bootinfo format */ +int bd_bios2unit(int biosdev); /* xlate BIOS device -> biosdisk unit */ +int bd_unit2bios(int unit); /* xlate biosdisk unit -> BIOS device */ int bd_getdev(struct i386_devdesc *dev); /* return dev_t for (dev) */ ssize_t i386_copyin(const void *src, vm_offset_t dest, const size_t len); @@ -92,12 +92,15 @@ void bios_getsmap(void); void bios_getmem(void); -extern u_int32_t bios_basemem; /* base memory in bytes */ -extern u_int32_t bios_extmem; /* extended memory in bytes */ +extern uint32_t bios_basemem; /* base memory in bytes */ +extern uint32_t bios_extmem; /* extended memory in bytes */ extern vm_offset_t memtop; /* last address of physical memory + 1 */ extern vm_offset_t memtop_copyin; /* memtop less heap size for the cases */ - /* when heap is at the top of extended memory */ - /* for other cases - just the same as memtop */ + /* when heap is at the top of */ + /* extended memory; for other cases */ + /* just the same as memtop */ +extern uint32_t high_heap_size; /* extended memory region available */ +extern vm_offset_t high_heap_base; /* for use as the heap */ int biospci_find_devclass(uint32_t class, int index, uint32_t *locator); int biospci_write_config(uint32_t locator, int offset, int width, uint32_t val); --- //depot/vendor/freebsd/src/sys/boot/i386/loader/main.c 2009/03/09 17:20:15 +++ //depot/user/jhb/boot/sys/boot/i386/loader/main.c 2009/12/04 15:33:59 @@ -102,13 +102,19 @@ */ bios_getmem(); -#if defined(LOADER_BZIP2_SUPPORT) || defined(LOADER_FIREWIRE_SUPPORT) || defined(LOADER_GPT_SUPPORT) || defined(LOADER_ZFS_SUPPORT) - heap_top = PTOV(memtop_copyin); - memtop_copyin -= 0x300000; - heap_bottom = PTOV(memtop_copyin); +#if defined(LOADER_BZIP2_SUPPORT) || defined(LOADER_FIREWIRE_SUPPORT) || \ + defined(LOADER_GPT_SUPPORT) || defined(LOADER_ZFS_SUPPORT) + if (high_heap_size > 0) { + heap_top = PTOV(high_heap_base) + high_heap_size; + heap_bottom = PTOV(high_heap_base); + if (high_heap_base < memtop_copyin) + memtop_copyin = high_heap_base; + } else #else - heap_top = (void *)bios_basemem; - heap_bottom = (void *)end; + { + heap_top = (void *)PTOV(bios_basemem); + heap_bottom = (void *)end; + } #endif setheap(heap_bottom, heap_top); --Boundary-00=_fxSGLLhaI8YOBsF--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200912041035.59173.jhb>