Date: Thu, 10 Jul 2008 12:40:19 +0200 From: Marek Woloszyn <Marek.Woloszyn@comp-css.pl> To: freebsd-ppc@freebsd.org Cc: grehan@FreeBSD.org Subject: Re: FreeBSD on MPC8349 (e300 core) Message-ID: <57ea49861343a439.48760333@comp.waw.pl>
index | next in thread | raw e-mail
[-- Attachment #1 --] Peter Grehan wrote: > Hi Marek, > >> We have FreeBSD 6.3 running on our Freescale MPC8349 board (e300 >> core). > > Excellent ! > >> But, unfortunately, it is not stable and we don't know what to >> do now. >> >> We have added several things to FreeBSD 6.3 to make it work on our >> board: 1. Some workarounds for OpenBIOS. 2. IMISS, DLMISS and DSMISS >> exceptions handlers in trap_subr.S. 3. Other hardware dependent >> stuff. > > There is a SoC project this year to port to the Efika platform: > > http://wiki.freebsd.org/Porting_FreeBSD_to_Efika_%28PPC_bring_up%29 > > The *MISS exception handlers are in that, via some patches that Andrew > Turner contributed many moons back. We will look into it. >> We have noticed that there were some errors in pmap: 1. Unnecessary >> ptegidx shift in pmap_pte_insert function (already fixed in 7.0). > > Yep, needs to be MFC'd back to RELENG_6. > >> 2. Hash table overflows when the system works with high load - caused >> by non-uniform PTE entries distribution in the hash table. > > Have you tried increasing the size of the hash table ? That is a simple > workaround. As you may have seen, there is no facility for overflow of > the secondary hash bucket: that is something that could be added. > Yes. We have already tried that. It helps, but the PTE usage distribution in PTEG table is still non-uniform. For normal PTEG table size the secondary hash bucket hits PTEGs that are already filled by the primary hash bucket and then we get a panic. >> Unfortunately we still have problems with the system. We experience >> random processes crashes when the system is starting. It happens >> approximately once for 20 boots. If the system does not crash at the >> startup, everything works fine. We do not know where to look for the >> solution: pmap? vm? > > Is it a hang ? A panic ? If the latter, do you have a console trace ? > It's a panic. I've attached three example backtraces from sh, tail and pkill. They have all appeared during the boot process. We have many more core dumps from various system tools, but they are all similar to these. Suddenly a pointer points to 0x0 or an index in a table is invalid. Maybe something in the kernel overwrites user pages or maps a wrong page for a process? We have also tried some tricks with <pmap_init(pmap_t pmap)> in pmap.c. There is a variable <entropy> witch is initialized with timebase register values and used to make VSID values more random. As our problem seems to be random and the booting process is rather deterministic, we have initialized the entropy with a constant value to check if it would have an impact on our problem. The first chosen value 0x12345678 didn't help, but 0x35913521chosen at random seems to help (we do not observe any crashed at boot, but we still haven't tried any long runs). Never the less, we haven't found the heart of this problem and we hardly have a clue where to look. We have been wondering if anybody else had expierienced similar program crashes. Ps. I'm not sure if my last e-mail reached your server, so i'm sending it once again. Please don't treat it as spam. Kind Regards, Marcin Ligenza [-- Attachment #2 --] gdb tail 225.tail.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "powerpc-semihalf-freebsd"... Attaching to program: /usr/bin/tail, process 225 ptrace: No such process. Core was generated by `tail'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x219ae740 in __sclose (cookie=0x0) at /usr/home/marcinl/p4/etacli/src/lib/libc/stdio/stdio.c:94 94 return (_close(((FILE *)cookie)->_file)); (gdb) bt #0 0x219ae740 in __sclose (cookie=0x0) at /usr/home/marcinl/p4/etacli/src/lib/libc/stdio/stdio.c:94 (gdb) print cookie $1 = (void *) 0x0 (gdb) ====================================================================================================== Comment: cookie is pointing to address 0x0 which doesn't seem to be a valid address [-- Attachment #3 --] %gdb sh 371.sh.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "powerpc-semihalf-freebsd"... Attaching to program: /bin/sh, process 371 ptrace: No such process. warning: exec file is newer than core file. Core was generated by `sh'. Program terminated with signal 6, Aborted. Reading symbols from /lib/libedit.so.5...done. Loaded symbols for /lib/libedit.so.5 Reading symbols from /lib/libncurses.so.6...done. Loaded symbols for /lib/libncurses.so.6 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 __raise (s=6) at /usr/home/marcinl/p4/etacli/src/lib/libc/gen/raise.c:51 51 } (gdb) bt #0 __raise (s=6) at /usr/home/marcinl/p4/etacli/src/lib/libc/gen/raise.c:51 #1 0x21a7723c in abort () at /usr/home/marcinl/p4/etacli/src/lib/libc/stdlib/abort.c:52 #2 0x219c88cc in wrtmessage ( p1=0x21a829f4 "��e���e���e���e���d���e���e0��e���e���e���e���e���e���e���e\020��e���e���eP��ep��e���e\220��e���e�junk pointer, too low to make sense\n", p2=0x21881000 "", p3=0x1802768 "\001\203��", p4=0x183c448 "Uk\020:=k\001\204\201k��}i\003�N\200\004 ") at /usr/home/marcinl/p4/etacli/src/lib/libc/stdlib/malloc.c:294 #3 0x219c8938 in wrterror (p=0x0) at /usr/home/marcinl/p4/etacli/src/lib/libc/stdlib/malloc.c:305 #4 0x219ca090 in free_bytes (ptr=0x1841000, index=14, info=0x0) at /usr/home/marcinl/p4/etacli/src/lib/libc/stdlib/malloc.c:1046 #5 0x219c9d20 in irealloc (ptr=0x1841000, size=0) at /usr/home/marcinl/p4/etacli/src/lib/libc/stdlib/malloc.c:854 #6 0x219ca784 in pubrealloc (ptr=0x1841000, size=0, func=0x21a82a8c "gh to make sense\n") at /usr/home/marcinl/p4/etacli/src/lib/libc/stdlib/malloc.c:1115 #7 0x219ca93c in pubrealloc (ptr=0x1841000, size=25412492, func=0x183c0d8 "") at /usr/home/marcinl/p4/etacli/src/lib/libc/stdlib/malloc.c:1149 #8 0x01818918 in ckfree (p=0x1841000) at /usr/home/marcinl/p4/etacli/src/bin/sh/memalloc.c:88 #9 0x018142dc in popfile () at /usr/home/marcinl/p4/etacli/src/bin/sh/input.c:485 #10 0x018143a8 in popallfiles () at /usr/home/marcinl/p4/etacli/src/bin/sh/input.c:502 #11 0x018143dc in closescript () at /usr/home/marcinl/p4/etacli/src/bin/sh/input.c:515 #12 0x01816540 in forkshell (jp=0x1843000, n=0x1842034, mode=0) at /usr/home/marcinl/p4/etacli/src/bin/sh/jobs.c:760 #13 0x01809bcc in evalcommand (cmd=0x1842034, flags=0, backcmd=0x0) at /usr/home/marcinl/p4/etacli/src/bin/sh/eval.c:747 #14 0x01808808 in evaltree (n=0x1842034, flags=0) at /usr/home/marcinl/p4/etacli/src/bin/sh/eval.c:267 #15 0x0180868c in evaltree (n=0x1842060, flags=0) at /usr/home/marcinl/p4/etacli/src/bin/sh/eval.c:221 #16 0x018084c8 in evalstring (s=0x7fffdf8a "{ setkey -D; } 2>&1") at /usr/home/marcinl/p4/etacli/src/bin/sh/eval.c:169 #17 0x01818134 in main (argc=3, argv=0x7fffdefc) at /usr/home/marcinl/p4/etacli/src/bin/sh/main.c:179 (gdb) f 4 #4 0x219ca090 in free_bytes (ptr=0x1841000, index=14, info=0x0) at /usr/home/marcinl/p4/etacli/src/lib/libc/stdlib/malloc.c:1046 1046 *mp = info->next; (gdb) list 1041 #ifdef MALLOC_EXTRA_SANITY 1042 if (!*mp) 1043 wrterror("(ES): Not on queue\n"); 1044 #endif /* MALLOC_EXTRA_SANITY */ 1045 } 1046 *mp = info->next; 1047 1048 /* Free the page & the info structure if need be */ 1049 page_dir[ptr2index(info->page)] = MALLOC_FIRST; 1050 vp = info->page; /* Order is important ! */ (gdb) print info $1 = (struct pginfo *) 0x0 ====================================================================================================== Comment: info is pointing to address 0x0 which doesn't seem to be a valid address [-- Attachment #4 --] %gdb pkill 467.pkill.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "powerpc-semihalf-freebsd"... Attaching to program: /bin/pkill, process 467 ptrace: No such process. warning: exec file is newer than core file. Core was generated by `pkill'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libkvm.so.3...done. Loaded symbols for /lib/libkvm.so.3 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x21820e14 in symlook_obj (name=0x2186a4e7 "__cxa_finalize", hash=199910549, obj=0x21854200, in_plt=0 '\0') at /usr/home/marcinl/p4/etacli/src/libexec/rtld-elf/rtld.c:2443 2443 unsigned long symnum = obj->buckets[hash % obj->nbuckets]; (gdb) bt #0 0x21820e14 in symlook_obj (name=0x2186a4e7 "__cxa_finalize", hash=199910549, obj=0x21854200, in_plt=0 '\0') at /usr/home/marcinl/p4/etacli/src/libexec/rtld-elf/rtld.c:2443 #1 0x21820ad4 in symlook_list (name=0x2186a4e7 "__cxa_finalize", hash=199910549, objlist=0x21862008, defobj_out=0x7fffdcd8, in_plt=0 '\0', dlp=0x7fffdcc0) at /usr/home/marcinl/p4/etacli/src/libexec/rtld-elf/rtld.c:2366 #2 0x21820774 in symlook_default (name=0x2186a4e7 "__cxa_finalize", hash=199910549, refobj=0x21854100, defobj_out=0x7fffdd24, in_plt=0 '\0') at /usr/home/marcinl/p4/etacli/src/libexec/rtld-elf/rtld.c:2299 #3 0x2181d48c in find_symdef (symnum=65, refobj=0x21854100, defobj_out=0x7fffdd60, in_plt=0 '\0', cache=0x2185a000) at /usr/home/marcinl/p4/etacli/src/libexec/rtld-elf/rtld.c:1005 #4 0x2181adc8 in reloc_nonplt_object (obj_rtld=0x21864b20, obj=0x21854100, rela=0x2186b1cc, cache=0x2185a000) at /usr/home/marcinl/p4/etacli/src/libexec/rtld-elf/powerpc/reloc.c:163 #5 0x2181b154 in reloc_non_plt (obj=0x21854100, obj_rtld=0x21864b20) at /usr/home/marcinl/p4/etacli/src/libexec/rtld-elf/powerpc/reloc.c:298 #6 0x2181e9f0 in relocate_objects (first=0x21854000, bind_now=0 '\0', rtldobj=0x21864b20) at /usr/home/marcinl/p4/etacli/src/libexec/rtld-elf/rtld.c:1549 #7 0x2181c1f8 in _rtld (sp=0x7fffdf10, exit_proc=0x7fffdeb8, objp=0x7fffdebc) at /usr/home/marcinl/p4/etacli/src/libexec/rtld-elf/rtld.c:475 #8 0x2181a988 in .rtld_start () at /usr/home/marcinl/p4/etacli/src/libexec/rtld-elf/powerpc/rtld_start.S:98 (gdb) f 0 #0 0x21820e14 in symlook_obj (name=0x2186a4e7 "__cxa_finalize", hash=199910549, obj=0x21854200, in_plt=0 '\0') at /usr/home/marcinl/p4/etacli/src/libexec/rtld-elf/rtld.c:2443 2443 unsigned long symnum = obj->buckets[hash % obj->nbuckets]; (gdb) list 2438 const Elf_Sym * 2439 symlook_obj(const char *name, unsigned long hash, const Obj_Entry *obj, 2440 bool in_plt) 2441 { 2442 if (obj->buckets != NULL) { 2443 unsigned long symnum = obj->buckets[hash % obj->nbuckets]; 2444 2445 while (symnum != STN_UNDEF) { 2446 const Elf_Sym *symp; 2447 const char *strp; (gdb) print hash % obj->nbuckets $1 = 199910549 (gdb) ====================================================================================================== Comment: (print hash % obj->nbucketsinfo) value 199910549 seems to large for an indexhome | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?57ea49861343a439.48760333>