Date: Thu, 13 May 2004 07:48:17 -0700 (PDT) From: M_SPAHZgORN <m_spahzgorn@yahoo.com> To: freebsd-questions@freebsd.org Subject: Help interpreting Kernel (gdb) Debug output Message-ID: <20040513144817.52007.qmail@web50307.mail.yahoo.com>
next in thread | raw e-mail | index | archive | help
I'm having a kernel crash on a regular basis, but don't know how to interpret
the gdb output. Any help would be much, much appreciated as I have a deadline
fast approaching to colocate this box.
> what version of FreeBSD?
Running: FreeBSD 4_9
> what error message comes to the screen when it panics?
I don't know because it usually happens between 3:03 AM and
4:40 AM EST. I am sleeping at this time. ;-)
> does the panic occur regularly (when I run this it fails, fail once
> in a while, etc)?
Yes, it seems to happen every day or every other day at the
times stated above. I've reinstalled the OS at least 10 times,
doing the same process each time, so I think it has something
to do with my install process. There are no CRON jobs scheduled
at the times it happens so I don't believe it's program related.
> any hardware issues flaky RAM/powersupply, non-terminate SCSI bus
> or heat problems that could be the problem?
Running high-quality Kingston ECC RAM (2 GB) with a 4 GB swap,
three seagate scsi-raid drives in raid-5 on an adaptec 2100s
controller, tyan dual-xeon mobo with two CPUs, everything is
top of the line. Heat is not an issue, I have extremely good
airflow in the box (15 fans total). The case is a 4u rackmount,
power supply is high quality 500w, and all scsi devices are
terminated properly. I am co-locating this box as soon as I
can figure this problem out.
> do you have GDB compiled into the kernel? (nice to have the symbol
table)
I believe so, here is my Kernel config:
#####
machine i386
ident DEBUG
maxusers 0
options MAXDSIZ="(512*1024*1024)"
options SMP
options APIC_IO
cpu I686_CPU
options COMPAT_43
options SYSVSHM
options SYSVSEM
options SYSVMSG
options KTRACE
options INET
pseudo-device ether
pseudo-device loop
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_VERBOSE_LIMIT=10
options IPSTEALTH
options RANDOM_IP_ID
options ACCEPT_FILTER_DATA
options ACCEPT_FILTER_HTTP
options ICMP_BANDLIM
options FFS
options FFS_ROOT
options PROCFS
options SOFTUPDATES
options P1003_1B
options _KPOSIX_PRIORITY_SCHEDULING
device scbus
device da
device pass
options SCSI_DELAY=5000
pseudo-device pty
device isa
device atkbdc0 at isa? port IO_KBD
device atkbd0 at atkbdc? irq 1 flags 0x1
#options KBD_INSTALL_CDEV
device vga0 at isa?
device sc0 at isa? flags 0x100
options MAXCONS=1
options SC_DISABLE_DDBKEY
options SC_DISABLE_REBOOT
options SC_NO_CUTPASTE
options SC_NO_FONT_LOADING
options SC_NO_HISTORY
options SC_NO_SYSMOUSE
device npx0 at nexus? port IO_NPX irq 13
device ata
options ATA_STATIC_ID
device pci
device agp
device em
options NMBCLUSTERS=87040 device asr
options DDB
options DDB_UNATTENDED
makeoptions DEBUG=-g
options DIAGNOSTIC
#####
And in my rc.conf I added:
#####
dumpdev="/dev/da0s1b"
dumpdir="/var/crash"
#####
Then after it crashes I run:
shell> gdb -k /usr/obj/usr/src/sys/DEBUG/kernel.debug /var/crash/vmcore.0
... I also run ...
shell> gdb -k /usr/obj/usr/src/sys/DEBUG/kernel.debug.orig /var/crash/vmcore.0
... because I've been told to backup my original 'kernel.debug'
file because it is changed after a crash. I get the same output
when I run gdb on either 'kernel.debug' file.
Then...
(kgdb) where
... and here is the output (which is what I need help interpreting):
-------------------------------------------------------------------------------
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487 if (dumping++) {
(kgdb) where
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1 0xc014ba30 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
#2 0xc014beb1 in panic (fmt=0xc0230db9 "%s") at
/usr/src/sys/kern/kern_shutdown.c:595
#3 0xc01ffc82 in trap_fatal (frame=0xfe9fac2c, eva=0) at
/usr/src/sys/i386/i386/trap.c:974
#4 0xc01ff8d5 in trap_pfault (frame=0xfe9fac2c, usermode=0, eva=0) at
/usr/src/sys/i386/i386/trap.c:867
#5 0xc01ff41b in trap (frame={tf_fs = -752156648, tf_es = -1071316976, tf_ds
=
16, tf_edi = 0,
tf_esi = -737259520, tf_ebp = -23090016, tf_isp = -23090088, tf_ebx = 0,
tf_edx = -1744879617,
tf_ecx = 42, tf_eax = 0, tf_trapno = 12, tf_err = 2, tf_eip
= -1071651613, tf_cs = 8,
tf_eflags = 66050, tf_esp = -24734848, tf_ss = -1072191432}) at
/usr/src/sys/i386/i386/trap.c:466
#6 0xc01fe4e3 in generic_bzero ()
#7 0xc01b7bd0 in ffs_vget (mp=0xd34da200, ino=739027, vpp=0xfe9fad50)
at /usr/src/sys/ufs/ffs/ffs_vfsops.c:1109
#8 0xc01bab2f in ufs_lookup (ap=0xfe9fada8) at
/usr/src/sys/ufs/ufs/ufs_lookup.c:611
#9 0xc01bf595 in ufs_vnoperate (ap=0xfe9fada8) at
/usr/src/sys/ufs/ufs/ufs_vnops.c:2376
#10 0xc017684a in vfs_cache_lookup (ap=0xfe9fae00) at vnode_if.h:77
#11 0xc01bf595 in ufs_vnoperate (ap=0xfe9fae00) at
/usr/src/sys/ufs/ufs/ufs_vnops.c:2376
#12 0xc0179921 in lookup (ndp=0xfe9fae7c) at vnode_if.h:52
#13 0xc017940c in namei (ndp=0xfe9fae7c) at /usr/src/sys/kern/vfs_lookup.c:153
#14 0xc017f93d in lstat (p=0xfe869380, uap=0xfe9faf80) at
/usr/src/sys/kern/vfs_syscalls.c:1824
#15 0xc01fffed in syscall2 (frame={tf_fs = 142082095, tf_es = 47, tf_ds
= -1078001617,
tf_edi = 136761280, tf_esi = 142748160, tf_ebp = -1077946144, tf_isp
= -23089196,
tf_ebx = 136761432, tf_edx = -1077945820, tf_ecx = 142568712, tf_eax =
190,
tf_trapno = -1077945552, tf_err = 2, tf_eip = 674403276, tf_cs = 31,
tf_eflags = 646,
tf_esp = -1077946524, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1175
#16 0xc01ecf8b in Xint0x80_syscall ()
#17 0x80e336e in ?? ()
#18 0x8111cbb in ?? ()
#19 0x804e2dd in ?? ()
#20 0x804fbab in ?? ()
#21 0x804ed51 in ?? ()
#22 0x804fbab in ?? ()
#23 0x804ed51 in ?? ()
#24 0x804fbab in ?? ()
#25 0x804ed51 in ?? ()
#26 0x804fbab in ?? ()
#27 0x804ed51 in ?? ()
#28 0x8050930 in ?? ()
#29 0x807b819 in ?? ()
#30 0x806a029 in ?? ()
#31 0x804adfe in ?? ()
-------------------------------------------------------------------------------
... and here is the relevant part from my all.log (5-13-04 (FRESH INSTALL) -
4:40:10):
#####
May 13 04:13:00 tycobb /usr/sbin/cron[3366]: (root) CMD
(/usr/local/sbin/tripwire --check | mail -s "Cron <
May 13 04:15:00 tycobb /usr/sbin/cron[3370]: (root) CMD (/usr/libexec/atrun)
May 13 04:29:56 tycobb syslogd: restart
May 13 04:29:56 tycobb /kernel: Checking for core dump:
May 13 04:29:56 tycobb /kernel: savecore: reboot after panic: page fault
May 13 04:29:56 tycobb savecore: reboot after panic: page fault
May 13 04:29:57 tycobb /kernel: savecore: system went down at Thu May 13
04:15:35 2004
May 13 04:29:57 tycobb /kernel: savecore: /var/crash/bounds: No such file or
directory
#####
Anything I'm forgetting?
__________________________________
Do you Yahoo!?
Yahoo! Movies - Buy advance tickets for 'Shrek 2'
http://movies.yahoo.com/showtimes/movie?mid=1808405861
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040513144817.52007.qmail>