Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 21 Jan 2006 07:05:10 +0100
From:      Thierry Herbelot <thierry@herbelot.com>
To:        freebsd-current@freebsd.org
Subject:   Re: panic: spin lock held too long (while rebooting)
Message-ID:  <200601210705.11539.thierry@herbelot.com>
In-Reply-To: <200601040838.49663.jhb@freebsd.org>
References:  <200601040806.37953.thierry@herbelot.com> <200601040838.49663.jhb@freebsd.org>

next in thread | previous in thread | raw e-mail | index | archive | help
Le Wednesday 4 January 2006 14:38, John Baldwin a écrit :
> On Wednesday 04 January 2006 02:06 am, Thierry Herbelot wrote:

[SNIP previous similar panic]
>
> Next time you get this, can you use 'show threads' to figure out the tid
> for the thread whose pointer is in the printf (0xc16de480 in this case) and
> then do a trace of that thread?

Hello,

Here is a more detailed crash session :

is this (zomb)  problematic ? (in ps) :
    8 c182e228    0     1     0 0002204 zomb[INACTIVE] g_mirror gm0s1

I keep the machine in DDB, if there are more detailed commands to investigate 
the panic (the machine is an SMP BP6, runs a GENERIC current kernel, and 
stores its local files in two g_mirror partitions).

The problematic spinlock is held by 0xc16de340 which is cpustop_handler.

	TfH

PS : printout of the crash :

# reboot
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...3 2 2 2 0 0 done
All buffers synced.
Uptime: 39m52s
GEOM_MIRROR: Device files1: provider mirror/files1 destroyed.
GEOM_MIRROR: Device files1 destroyed.
GEOM_MIRROR: Device gm0s1: provider mirror/gm0s1 destroyed.
GEOM_MIRROR: Device gm0s1 destroyed.
Rebooting...
cpu_reset: Stopping other CPUs
spin lock sched lock held by 0xc16de340 for > 5 seconds
panic: spin lock held too long
cpuid = 0
KDB: enter: panic
[thread pid 18874 tid 100079 ]
Stopped at      kdb_enter+0x2b: nop

db> trace
Tracing pid 18874 tid 100079 td 0xc197c1a0
kdb_enter(c0882824) at kdb_enter+0x2b
panic(c0881c7e,c094e970,c16de340,c0881c55,c0881daf) at panic+0x127
_mtx_lock_spin(c094e970,c197c1a0,2,c087e5ea,cb) at _mtx_lock_spin+0xa8
_mtx_lock_spin_flags(c094e970,2,c087e5ea,cb) at _mtx_lock_spin_flags+0xaa
hardclock_cpu(0,c09cef20,0,c7644ba0,c0808371) at hardclock_cpu+0x25
hardclock(0,c081810e) at hardclock+0x13
lapic_handle_timer(8,28,c16b0028,c16ba00c,0) at lapic_handle_timer+0x71
Xtimerint() at Xtimerint+0x1e
--- interrupt, eip = 0xc081810e, esp = 0xc7644be8, ebp = 0xc7644c0c ---
DELAY(f4240) at DELAY+0x92
cpu_reset(f4240,c0882a0b,c7644c64,c065418b,0) at cpu_reset+0xc8
shutdown_reset(0,0,c16ba00c,0,c0882848) at shutdown_reset+0x1c
boot(0,c094e988,0,c0882848,a1) at boot+0x553
reboot(c197c1a0,c7644d04,c197c1a0,0,2) at reboot+0x32
syscall(3b,3b,3b,0,2) at syscall+0x27e
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (55, FreeBSD ELF32, reboot), eip = 0x280add0f, esp = 0xbfbfedcc, 
ebp = 0xbfbfee18 ---

db> ps
  pid   proc     uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
18874 c1be0450    0     1 18874 0004002 [CPU 0] reboot
   41 c18a5ac8    0     0     0 0000204 [SLPQ - 0xc72cfd04][SLP] schedcpu
   40 c18a5cf0    0     0     0 0000204 [SLPQ - 0xc09a2dec][SLP] nfsiod 3
   39 c18aa000    0     0     0 0000204 [SLPQ - 0xc09a2de8][SLP] nfsiod 2
   38 c17a8cf0    0     0     0 0000204 [SLPQ - 0xc09a2de4][SLP] nfsiod 1
   37 c182b000    0     0     0 0000204 [SLPQ - 0xc09a2de0][SLP] nfsiod 0
   36 c182b228    0     0     0 0000204 [SLPQ ktsusp 0xc182b2ac][SLP] vnlru
   35 c182b450    0     0     0 0000204 [SLPQ ktsusp 0xc182b4d4][SLP] syncer
   34 c182b678    0     0     0 0000204 [SLPQ ktsusp 0xc182b6fc][SLP] 
bufdaemon
   33 c182b8a0    0     0     0 000020c [SLPQ pgzero 0xc09b0d10][SLP] pagezero
   32 c182bac8    0     0     0 0000204 [SLPQ psleep 0xc09a8c3c][SLP] vmdaemon
   31 c182bcf0    0     0     0 0000204 [SLPQ psleep 0xc09a8bfc][SLP] 
pagedaemon
   30 c182e450    0     0     0 0000204 [IWAIT] irq5: pcm1
    7 c16f5678    0     0     0 0000204 [SLPQ - 0xc16cbe3c][SLP] fdc0
   29 c16f58a0    0     0     0 0000204 [IWAIT] swi0: sio
   28 c16f5ac8    0     0     0 0000204 [IWAIT] irq7: ppc0
   27 c16f5cf0    0     0     0 0000204 [IWAIT] irq1: atkbd0
   26 c17a8000    0     0     0 0000204 [IWAIT] irq17: atapci1
   25 c17a8228    0     0     0 0000204 [IWAIT] irq18: pcm0++
   24 c17a8450    0     0     0 0000204 [SLPQ usbtsk 0xc094c744][SLP] usbtask
   23 c17a8678    0     0     0 0000204 [SLPQ usbevt 0xc17b1210][SLP] usb0
   22 c17a88a0    0     0     0 0000204 [IWAIT] irq19: dc0 uhci0
   21 c17a8ac8    0     0     0 0000204 [IWAIT] irq15: ata1
   20 c16e2228    0     0     0 0000204 [IWAIT] irq14: ata0
   19 c16e2450    0     0     0 0000204 [IWAIT] swi6: task queue
    6 c16e2678    0     0     0 0000204 [SLPQ - 0xc1792400][SLP] kqueue taskq
   18 c16e28a0    0     0     0 0000204 [IWAIT] swi2: cambio
   17 c16e2ac8    0     0     0 0000204 [IWAIT] swi5: +
    5 c16e2cf0    0     0     0 0000204 [SLPQ - 0xc16dc500][SLP] thread taskq
   16 c16f5000    0     0     0 0000204 [IWAIT] swi6: Giant taskq
   15 c16f5228    0     0     0 0000204 [SLPQ - 0xc094a3a0][SLP] yarrow
    4 c16f5450    0     0     0 0000204 [SLPQ - 0xc094cebc][SLP] g_down
    3 c16dd000    0     0     0 0000204 [SLPQ - 0xc094ceb8][SLP] g_up
    2 c16dd228    0     0     0 0000204 [RUNQ] g_event
   14 c16dd450    0     0     0 0000204 [IWAIT] swi1: net
   13 c16dd678    0     0     0 0000204 [IWAIT] swi3: vm
   12 c16dd8a0    0     0     0 000020c [LOCK  Giant c1824480] swi4: clock sio
   11 c16ddac8    0     0     0 000020c [Can run] idle: cpu0
   10 c16ddcf0    0     0     0 000020c [Can run] idle: cpu1
    1 c16e2000    0     0     1 0004200 [CPU 1] init
    0 c094cfc0    0     0     0 0000200 [IWAIT] swapper
    8 c182e228    0     1     0 0002204 zomb[INACTIVE] g_mirror gm0s1


db> show threads
  100079 (0xc197c1a0)  kdb_enter(c0882824) at kdb_enter+0x2b
  100047 (0xc18a9680)  sched_switch(c18a9680,0,1) at sched_switch+0x183
  100048 (0xc18a94e0)  sched_switch(c18a94e0,0,1) at sched_switch+0x183
  100049 (0xc18a9340)  sched_switch(c18a9340,0,1) at sched_switch+0x183
  100027 (0xc16e3d00)  sched_switch(c16e3d00,0,1) at sched_switch+0x183
  100028 (0xc16e3b60)  sched_switch(c16e3b60,0,1) at sched_switch+0x183
  100029 (0xc16e39c0)  sched_switch(c16e39c0,0,1) at sched_switch+0x183
  100030 (0xc16e3820)  sched_switch(c16e3820,0,1) at sched_switch+0x183
  100031 (0xc16e3680)  sched_switch(c16e3680,0,1) at sched_switch+0x183
  100032 (0xc16e34e0)  sched_switch(c16e34e0,0,1) at sched_switch+0x183
  100033 (0xc182d000)  sched_switch(c182d000,0,1) at sched_switch+0x183
  100034 (0xc182cd00)  sched_switch(c182cd00,0,1) at sched_switch+0x183
  100037 (0xc182c820)  fork_trampoline() at fork_trampoline
  100017 (0xc16df680)  sched_switch(c16df680,0,1) at sched_switch+0x183
  100018 (0xc16df4e0)  sched_switch(c16df4e0,0,1) at sched_switch+0x183
  100019 (0xc16df340)  fork_trampoline() at fork_trampoline
  100020 (0xc16df1a0)  fork_trampoline() at fork_trampoline
  100021 (0xc17a9820)  fork_trampoline() at fork_trampoline
  100022 (0xc17a9680)  sched_switch(c17a9680,0,1) at sched_switch+0x183
  100023 (0xc17a94e0)  sched_switch(c17a94e0,0,1) at sched_switch+0x183
  100024 (0xc17a9340)  sched_switch(c17a9340,0,1) at sched_switch+0x183
  100025 (0xc17a91a0)  fork_trampoline() at fork_trampoline
  100026 (0xc17a9000)  sched_switch(c17a9000,0,1) at sched_switch+0x183
  100008 (0xc16de1a0)  sched_switch(c16de1a0,0,1) at sched_switch+0x183
  100009 (0xc16de000)  sched_switch(c16de000,0,1) at sched_switch+0x183
  100010 (0xc16e3340)  sched_switch(c16e3340,0,1) at sched_switch+0x183
  100011 (0xc16e31a0)  fork_trampoline() at fork_trampoline
  100012 (0xc16e3000)  fork_trampoline() at fork_trampoline
  100013 (0xc16dfd00)  sched_switch(c16dfd00,0,1) at sched_switch+0x183
  100014 (0xc16dfb60)  fork_trampoline() at fork_trampoline
  100015 (0xc16df9c0)  sched_switch(c16df9c0,0,1) at sched_switch+0x183
  100016 (0xc16df820)  sched_switch(c16df820,0,1) at sched_switch+0x183
  100000 (0xc16df000)  sched_switch(c16df000,0,1) at sched_switch+0x183
  100001 (0xc16ded00)  sched_switch(c16ded00,0,1) at sched_switch+0x183
  100002 (0xc16deb60)  fork_trampoline() at fork_trampoline
  100003 (0xc16de9c0)  fork_trampoline() at fork_trampoline
  100004 (0xc16de820)  sched_switch(c16de820,0,1) at sched_switch+0x183
  100005 (0xc16de680)  sched_switch(c16de680,c16de1a0,6) at sched_switch+0x183
  100006 (0xc16de4e0)  sched_switch(c16de4e0,0,1) at sched_switch+0x183
  100007 (0xc16de340)  cpustop_handler(c6cccba8,c08149f1,c6cccb4c,c0678738,0) 
at cpustop_handler+0x31
       0 (0xc094d200)  sched_switch(c094d200,0,1) at sched_switch+0x183



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200601210705.11539.thierry>