Date: Thu, 10 May 2007 19:00:00 -0600 From: Scott Swanson <scotts@inetz.com> To: Kris Kennaway <kris@obsecurity.org> Cc: freebsd-stable@freebsd.org Subject: Re: panic: spin lock held too long (w/ backtrace) Message-ID: <4643C010.7060302@inetz.com> In-Reply-To: <20070510214348.GA68925@xor.obsecurity.org> References: <46438569.1000506@inetz.com> <20070510214348.GA68925@xor.obsecurity.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Kris Kennaway wrote: > On Thu, May 10, 2007 at 02:49:45PM -0600, Scott Swanson wrote: >> Hello all, >> >> I have a couple dozen SuperMicro servers in production and have had >> reoccurring crashing issues on a couple of them that are under higher >> load. >> >> After managing to pry one out of production, I have been able to >> re-create the problem fairly consistently just by looping a script that >> tars up a large directory and writes the the file to a different >> partition on the same raid array. >> >> Can anyone point me towards the next step to debug this issue? I can >> pursue any other debugging info that may be required. >> >> Regards; >> Scott Swanson >> Inetz Media >> >> >>> uname -a >> FreeBSD pyrite.inetz.com 6.2-RELEASE-p4 FreeBSD 6.2-RELEASE-p4 #3: Tue >> May 8 13:08:22 MDT 2007 >> root@biotite.inetz.com:/usr/obj/usr/src/sys/PYRITE i386 >> >> >> root@pyrite /usr/obj/usr/src/sys/PYRITE > kgdb kernel.debug >> /var/crash/vmcore.0 >> [GDB will not be able to debug user-mode threads: >> /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] >> GNU gdb 6.1.1 [FreeBSD] >> >> Unread portion of the kernel message buffer: >> spin lock smp rendezvous held by 0xc95a5900 for > 5 seconds > > What is thread 0xc95a5900 doing? > > Kris > Is this the best way to determine the action of the thread? Regards; Scott (kgdb) info threads 90 Thread 100123 (PID=18303: bsdtar) 0xc0644f5b in sched_switch (td=0xc95a5900, newtd=0xc92aad80, flags=0) at /usr/src/sys/kern/sched_4bsd.c:973 89 Thread 100140 (PID=17447: sh) 0xc0644f5b in sched_switch (td=0xc9a67c00, newtd=0xc95a5900, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 88 Thread 100144 (PID=17441: tcsh) 0xc0644f5b in sched_switch (td=0xc9a67600, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 87 Thread 100125 (PID=17440: sshd) 0xc0644f5b in sched_switch (td=0xc95a5600, newtd=0xc95a5900, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 86 Thread 100109 (PID=17437: sshd) 0xc0644f5b in sched_switch (td=0xc9a41480, newtd=0xc9a67600, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 85 Thread 100039 (PID=898: getty) 0xc0644f5b in sched_switch (td=0xc93e8900, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 84 Thread 100067 (PID=897: getty) 0xc0644f5b in sched_switch (td=0xc95a6d80, newtd=0xc93e8900, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 83 Thread 100087 (PID=896: getty) 0xc0644f5b in sched_switch (td=0xc9604180, newtd=0xc9903180, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 82 Thread 100088 (PID=895: getty) 0xc0644f5b in sched_switch (td=0xc9903300, newtd=0xc9832780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 81 Thread 100089 (PID=894: getty) 0xc0644f5b in sched_switch (td=0xc9903180, newtd=0xc92aa780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 80 Thread 100082 (PID=893: getty) 0xc0644f5b in sched_switch (td=0xc9604900, newtd=0xc9903180, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 79 Thread 100073 (PID=892: getty) 0xc0644f5b in sched_switch (td=0xc9832780, newtd=0xc9832180, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 78 Thread 100077 (PID=891: getty) 0xc0644f5b in sched_switch (td=0xc9832180, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 77 Thread 100078 (PID=890: getty) 0xc0644f5b in sched_switch (td=0xc9832000, newtd=0xc9604900, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 76 Thread 100090 (PID=849: cron) 0xc0644f5b in sched_switch (td=0xc9903000, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 75 Thread 100056 (PID=843: sendmail) 0xc0644f5b in sched_switch (td=0xc9604000, newtd=0xc92aa780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 74 Thread 100069 (PID=839: sendmail) 0xc0644f5b in sched_switch (td=0xc95a6a80, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 73 Thread 100063 (PID=833: sshd) 0xc0644f5b in sched_switch (td=0xc9603480, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 72 Thread 100107 (PID=832: mysqld) 0xc0644f5b in sched_switch (td=0xc9a41780, newtd=0xc92aa780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 71 Thread 100106 (PID=832: mysqld) 0xc0644f5b in sched_switch (td=0xc9a41900, newtd=0xc92aa780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 70 Thread 100105 (PID=832: mysqld) 0xc0644f5b in sched_switch (td=0xc9a41a80, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 69 Thread 100102 (PID=832: mysqld) 0xc0644f5b in sched_switch (td=0xc95a5c00, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 68 Thread 100104 (PID=832: mysqld) 0xc0644f5b in sched_switch (td=0xc9a41c00, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 67 Thread 100103 (PID=832: mysqld) 0xc0644f5b in sched_switch (td=0xc9832a80, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 66 Thread 100101 (PID=832: mysqld) 0xc0644f5b in sched_switch (td=0xc9832d80, newtd=0xc9a41780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 65 Thread 100100 (PID=832: mysqld) 0xc0644f5b in sched_switch (td=0xc9902000, newtd=0xc9a67c00, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 64 Thread 100099 (PID=832: mysqld) 0xc0644f5b in sched_switch (td=0xc9902180, newtd=0xc9a67c00, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 63 Thread 100098 (PID=832: mysqld) 0xc0644f5b in sched_switch (td=0xc9902300, newtd=0xc92aa780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 62 Thread 100060 (PID=832: mysqld) 0xc0644f5b in sched_switch (td=0xc9603900, newtd=0xc9a41780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 61 Thread 100059 (PID=796: sh) 0xc0644f5b in sched_switch (td=0xc9603a80, newtd=0xc9603900, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 60 Thread 100070 (PID=784: usbd) 0xc0644f5b in sched_switch (td=0xc95a6900, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 59 Thread 100076 (PID=764: ntpd) 0xc0644f5b in sched_switch (td=0xc9832300, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 58 Thread 100051 (PID=734: nrpe2) 0xc0644f5b in sched_switch (td=0xc95a6480, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 57 Thread 100074 (PID=720: nfsd) 0xc0644f5b in sched_switch (td=0xc9832600, newtd=0xc95a5d80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 56 Thread 100061 (PID=718: nfsd) 0xc0644f5b in sched_switch (td=0xc9603780, newtd=0xc95a6d80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 55 Thread 100057 (PID=717: nfsd) 0xc0644f5b in sched_switch (td=0xc9603d80, newtd=0xc93e8780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 54 Thread 100038 (PID=716: nfsd) 0xc0644f5b in sched_switch (td=0xc93e8a80, newtd=0xc93e8780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 53 Thread 100040 (PID=715: nfsd) 0xc0644f5b in sched_switch (td=0xc93e8780, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 52 Thread 100052 (PID=713: mountd) 0xc0644f5b in sched_switch (td=0xc95a6300, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 51 Thread 100075 (PID=652: rpcbind) 0xc0644f5b in sched_switch (td=0xc9832480, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 50 Thread 100071 (PID=634: syslogd) 0xc0644f5b in sched_switch (td=0xc95a6780, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 49 Thread 100050 (PID=568: devd) 0xc0644f5b in sched_switch (td=0xc95a6600, newtd=0xc92aa780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 48 Thread 100058 (PID=136: adjkerntz) 0xc0644f5b in sched_switch (td=0xc9603c00, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 47 Thread 100042 (PID=46: schedcpu) 0xc0644f5b in sched_switch (td=0xc93e8480, newtd=0xc92aad80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 46 Thread 100043 (PID=45: softdepflush) 0xc0644f5b in sched_switch (td=0xc93e8300, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 ---Type <return> to continue, or q <return> to quit--- 45 Thread 100044 (PID=44: vnlru) 0xc0644f5b in sched_switch (td=0xc93e8180, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 * 44 Thread 100045 (PID=43: syncer) doadump () at pcpu.h:165 43 Thread 100046 (PID=42: bufdaemon) 0xc0644f5b in sched_switch (td=0xc93abd80, newtd=0xc93ab000, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 42 Thread 100047 (PID=41: pagezero) 0xc0644f5b in sched_switch (td=0xc93abc00, newtd=0xc92aa780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 41 Thread 100048 (PID=40: vmdaemon) 0xc0644f5b in sched_switch (td=0xc93aba80, newtd=0xc93e8480, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 40 Thread 100049 (PID=39: pagedaemon) 0xc0644f5b in sched_switch (td=0xc93ab900, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 39 Thread 100027 (PID=38: irq7: ppc0) fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 38 Thread 100028 (PID=37: fdc0) 0xc0644f5b in sched_switch (td=0xc93ab000, newtd=0xc93e8180, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 37 Thread 100029 (PID=36: swi0: sio) 0xc0644f5b in sched_switch (td=0xc92afd80, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 36 Thread 100030 (PID=35: irq1: atkbd0) 0xc0644f5b in sched_switch (td=0xc92afc00, newtd=0xc92af300, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 35 Thread 100031 (PID=34: irq15: ata1) 0xc0644f5b in sched_switch (td=0xc92afa80, newtd=0xc92af180, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 34 Thread 100032 (PID=33: irq14: ata0) fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 33 Thread 100033 (PID=32: usb2) 0xc0644f5b in sched_switch (td=0xc92af780, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 32 Thread 100034 (PID=31: irq18: uhci2) fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 31 Thread 100035 (PID=30: usb1) 0xc0644f5b in sched_switch (td=0xc92af480, newtd=0xc92af780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 30 Thread 100036 (PID=29: irq19: uhci1) fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 29 Thread 100037 (PID=28: usbtask) 0xc0644f5b in sched_switch (td=0xc93e8c00, newtd=0xc92af480, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 28 Thread 100017 (PID=27: usb0) 0xc0644f5b in sched_switch (td=0xc92ab900, newtd=0xc92af480, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 27 Thread 100018 (PID=26: irq16: uhci0) fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 26 Thread 100019 (PID=25: irq72: asr0) 0xc0644f5b in sched_switch (td=0xc92ab600, newtd=0xc93ab780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 25 Thread 100020 (PID=24: irq29: em1) fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 24 Thread 100021 (PID=23: irq28: em0) 0xc0644f5b in sched_switch (td=0xc92ab300, newtd=0xc92aa780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 23 Thread 100022 (PID=22: irq9: acpi0) fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 22 Thread 100023 (PID=21: swi2: cambio) 0xc0644f5b in sched_switch (td=0xc93ab780, newtd=0xc92abc00, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 21 Thread 100024 (PID=9: kqueue taskq) 0xc0644f5b in sched_switch (td=0xc93ab600, newtd=0xc09475c0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 20 Thread 100025 (PID=8: acpi_task_2) 0xc0644f5b in sched_switch (td=0xc93ab480, newtd=0xc93ab600, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 19 Thread 100026 (PID=7: acpi_task_1) 0xc0644f5b in sched_switch (td=0xc93ab300, newtd=0xc93ab480, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 18 Thread 100008 (PID=6: acpi_task_0) 0xc0644f5b in sched_switch (td=0xc92aa300, newtd=0xc93ab300, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 17 Thread 100009 (PID=20: swi5: +) fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 16 Thread 100010 (PID=5: thread taskq) 0xc0644f5b in sched_switch (td=0xc92aa000, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 15 Thread 100011 (PID=19: swi6: Giant taskq) 0xc0644f5b in sched_switch (td=0xc92af300, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 14 Thread 100012 (PID=18: swi6: task queue) 0xc0644f5b in sched_switch (td=0xc92af180, newtd=0xc92aa780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 13 Thread 100013 (PID=17: yarrow) 0xc0644f5b in sched_switch (td=0xc92af000, newtd=0xc92aba80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 12 Thread 100014 (PID=4: g_down) 0xc0644f5b in sched_switch (td=0xc92abd80, newtd=0xc93e8000, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 11 Thread 100015 (PID=3: g_up) 0xc0644f5b in sched_switch (td=0xc92abc00, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 10 Thread 100016 (PID=2: g_event) 0xc0644f5b in sched_switch (td=0xc92aba80, newtd=0xc92aaa80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 9 Thread 100000 (PID=16: swi3: vm) fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 8 Thread 100001 (PID=15: swi4: clock sio) 0xc0644f5b in sched_switch (td=0xc92aad80, newtd=0xc93e8000, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 7 Thread 100002 (PID=14: swi1: net) 0xc0644f5b in sched_switch (td=0xc92aac00, newtd=0xc92aa780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 6 Thread 100003 (PID=13: idle: cpu0) 0xc0644f5b in sched_switch (td=0xc92aaa80, newtd=0xc92aad80, flags=2) at /usr/src/sys/kern/sched_4bsd.c:973 5 Thread 100004 (PID=12: idle: cpu1) fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 4 Thread 100005 (PID=11: idle: cpu2) 0xc0644f5b in sched_switch (td=0xc92aa780, newtd=0xc95a5900, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 3 Thread 100006 (PID=10: idle: cpu3) fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 2 Thread 100007 (PID=1: init) 0xc0644f5b in sched_switch (td=0xc92aa480, newtd=0xc9603480, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973 1 Thread 0 (PID=0: swapper) 0xc0644f5b in sched_switch (td=0xc09475c0, newtd=0xc93ab780, flags=1) at /usr/src/sys/kern/sched_4bsd.c:973
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4643C010.7060302>