Date: Fri, 8 Feb 2002 16:17:41 -0800 From: "Eugene M. Kim" <gene@nttmcl.com> To: Julian Elischer <julian@elischer.org> Cc: David Wolfskill <david@catwhisker.org>, current@FreeBSD.ORG Subject: Re: Hang on flushing buffers w/today's -CURRENT, SMP system Message-ID: <20020208161740.A28661@alicia.nttmcl.com> In-Reply-To: <Pine.BSF.4.21.0202081556080.3881-100000@InterJet.elischer.org>; from julian@elischer.org on Fri, Feb 08, 2002 at 03:56:21PM -0800 References: <20020208152145.B24821@alicia.nttmcl.com> <Pine.BSF.4.21.0202081556080.3881-100000@InterJet.elischer.org>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
On Fri, Feb 08, 2002 at 03:56:21PM -0800, Julian Elischer wrote:
>
> In your case we need totrace proc 1 I think..
>
I got the `reboot' process at this session, so I traced that process.
Before I had used `shutdown -r', which probably SIGINT'ed the init
process so it's init (pid 1) calling reboot()... The attached log also
has its trace JFYI.
One more bit of info: as you see from the pcpu output, mine is not an
SMP but an UP box.
Thanks,
Eugene
[-- Attachment #2 --]
show locks
exclusive (sleep mutex) Giant (0xc02e60c0) locked @ /usr/src/sys/kern/kern_intr.c:532
db> ps
pid proc addr uid ppid pgrp flag stat wmesg wchan cmd
279 cdbdc500 cdc6e000 0 1 279 0004002 2 reboot
185 cc988900 cdbbc000 0 0 0 0000204 3 nfsidl c1d053ac nfsiod 3
184 cc988c00 cdbb8000 0 0 0 0000204 3 nfsidl c1d053a8 nfsiod 2
183 cc988f00 cdbb4000 0 0 0 0000204 3 nfsidl c1d053a4 nfsiod 1
182 cc989200 cdbb0000 0 0 0 0000204 3 nfsidl c1d053a0 nfsiod 0
7 cc98b600 cd1a6000 0 0 0 0000204 3 ktsusp cc98b800 syncer
6 cc98b900 cd1a2000 0 0 0 0000204 3 ktsusp cc98bb00 vnlru
5 cc98bc00 cd19e000 0 0 0 0000204 3 ktsusp cc98be00 bufdaemon
4 cc98bf00 cd19a000 0 0 0 0000204 3 pgzero c0327f88 pagezero
3 cc98c200 cd196000 0 0 0 0000204 3 psleep c0327f9c vmdaemon
2 cc98c500 cd192000 0 0 0 0000204 3 psleep c02e0698 pagedaemon
31 cc98c800 cc991000 0 0 0 0000204 6 irq8: rtc
30 cc98cb00 cc98d000 0 0 0 0000204 6 irq0: clk
29 cc321f00 cc984000 0 0 0 0000204 6 irq4: sio0
28 cc322200 cc980000 0 0 0 0000204 6 swi0: tty:sio
27 cc322500 cc97c000 0 0 0 0000204 6 irq7: ppc0
26 cc322800 cc978000 0 0 0 0000204 6 irq12: psm0
25 cc322b00 cc974000 0 0 0 0000204 2 irq1: atkbd0
24 cc322e00 cc970000 0 0 0 0000204 3 usbevt c1b60210 usb0
23 cc323100 cc96c000 0 0 0 0000204 6 irq11: uhci0
--More--
22 cc323400 cc968000 0 0 0 0000204 6 irq15: ata1
21 cc323700 cc964000 0 0 0 0000204 6 irq14: ata0
20 cc323a00 cc95b000 0 0 0 0000204 6 irq5: pcm0
19 cc323d00 cc953000 0 0 0 0000204 6 irq13:
18 cc324000 cc94f000 0 0 0 0000204 6 swi5: acpitaskq
17 cc324300 cc94b000 0 0 0 0000204 6 swi5: task queue
16 cc324600 cc947000 0 0 0 0000204 6 swi3: cambio
15 cc324900 cc943000 0 0 0 0000204 6 swi2: camnet
14 cc324c00 cc93f000 0 0 0 0000204 3 sleep c04141c0 random
13 cc324f00 cc93b000 0 0 0 0000204 6 swi4: vm
12 cc325200 cc937000 0 0 0 000020c 2 swi6: tty:sio clock
11 cc325500 cc933000 0 0 0 0000204 6 swi1: net
10 cc325800 cc32e000 0 0 0 000020c 2 idle
1 cc325b00 cc32a000 0 0 1 0004200 3 wait cc325b00 init
0 c02c41c0 c047c000 0 0 0 0000200 3 sched c02c41c0 swapper
db> tr 279
mi_switch(0,cdbdc500,cdbdc604,10,0) at mi_switch+0x153
boot(0,cdbdc714,cdc71d40,c0262b80,cdbdc604) at boot+0x200
reboot(cdbdc604,cdc71d20,2,0,0) at reboot+0x37
syscall(2f,2f,2f,0,0) at syscall+0x254
syscall_with_err_pushed() at syscall_with_err_pushed+0x1b
--- syscall (55, FreeBSD ELF, reboot), eip = 0x8048b8b, esp = 0xbfbffb1c, ebp = 0xbfbffb48 ---
db> tr 1
mi_switch(1,0,cc32dd20,1,0) at mi_switch+0x153
msleep(cc325b00,0,15c,c0287e85,0) at msleep+0x322
wait1(cc325c04,cc32dd20,0,cc32dd40,c0262b80) at wait1+0x617
wait4(cc325c04,cc32dd20,0,bfbffe18,bfbffe24) at wait4+0x12
syscall(2f,2f,2f,bfbffe24,bfbffe18) at syscall+0x254
syscall_with_err_pushed() at syscall_with_err_pushed+0x1b
--- syscall (7, FreeBSD ELF, wait4), eip = 0x8050c37, esp = 0xbfbffcf8, ebp = 0xbfbffd14 ---
db> tr 0
mi_switch(c02def10,0,483000,1,0) at mi_switch+0x153
msleep(c02c41c0,0,44,c02a7570,3e8) at msleep+0x322
scheduler(0,47bc00,47b000,0,c0121d1c) at scheduler+0x146
mi_startup() at mi_startup+0x95
begin() at begin+0x43
db> ~~ show witness
Sleep locks:
0 (dead) -- last acquired @ (dead):0
0 (dead) -- last acquired @ (dead):0
0 Giant -- last acquired @ /usr/src/sys/kern/kern_intr.c:532
1 ithread -- last acquired @ /usr/src/sys/kern/kern_intr.c:269
2 struct filedesc -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1170
2 fork list -- last acquired @ /usr/src/sys/kern/kern_fork.c:649
3 lockmgr -- last acquired @ /usr/src/sys/kern/kern_lock.c:227
2 proctree -- last acquired @ /usr/src/sys/kern/kern_exit.c:491
3 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:303
3 allproc -- last acquired @ /usr/src/sys/kern/kern_synch.c:267
4 zone -- last acquired @ /usr/src/sys/vm/vm_zone.c:506
4 process lock -- last acquired @ /usr/src/sys/kern/kern_kthread.c:178
5 ucred -- last acquired @ /usr/src/sys/kern/kern_prot.c:1601
5 uidinfo hash -- last acquired @ /usr/src/sys/kern/kern_resource.c:844
6 uidinfo struct -- last acquired @ order list:0
1 mbuf PCPU list lock -- last acquired @ /usr/src/sys/kern/subr_mbuf.c:784
1 eventhandler -- last acquired @ /usr/src/sys/kern/subr_eventhandler.c:162
3 lockmgr -- last acquired @ /usr/src/sys/kern/kern_lock.c:227
3 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:303
4 zone -- last acquired @ /usr/src/sys/vm/vm_zone.c:506
1 sndstat -- last acquired @ /usr/src/sys/modules/sound/pcm/../../../dev/sound/pcm/sndstat.c:231
1 sf_bufs list lock -- last acquired @ /usr/src/sys/kern/uipc_syscalls.c:1556
3 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:303
3 lockmgr -- last acquired @ /usr/src/sys/kern/kern_lock.c:227
4 zone -- last acquired @ /usr/src/sys/vm/vm_zone.c:506
1 zone subsystem -- last acquired @ /usr/src/sys/vm/vm_zone.c:179
4 zone -- last acquired @ /usr/src/sys/vm/vm_zone.c:506
1 rman -- last acquired @ /usr/src/sys/kern/subr_rman.c:194
3 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:303
1 bpf global lock -- last acquired @ /usr/src/sys/net/bpf.c:1270
1 taskqueue list -- last acquired @ /usr/src/sys/kern/subr_taskqueue.c:85
1 rman head -- last acquired @ /usr/src/sys/kern/subr_rman.c:107
1 ACPI semaphore -- last acquired @ /usr/src/sys/modules/acpi/../../dev/acpica/Osd/OsdSynch.c:296
1 vm86 lock -- last acquired @ /usr/src/sys/i386/i386/vm86.c:578
1 pcm0:play:2 -- last acquired @ /usr/src/sys/modules/sound/pcm/../../../dev/sound/pcm/channel.c:674
3 lockmgr -- last acquired @ /usr/src/sys/kern/kern_lock.c:227
3 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:303
4 zone -- last acquired @ /usr/src/sys/vm/vm_zone.c:506
1 pcm0:play:3 -- last acquired @ /usr/src/sys/modules/sound/pcm/../../../dev/sound/pcm/channel.c:674
3 lockmgr -- last acquired @ /usr/src/sys/kern/kern_lock.c:227
3 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:303
4 zone -- last acquired @ /usr/src/sys/vm/vm_zone.c:506
1 pcm0:play:1 -- last acquired @ /usr/src/sys/modules/sound/pcm/../../../dev/sound/pcm/channel.c:674
3 lockmgr -- last acquired @ /usr/src/sys/kern/kern_lock.c:227
3 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:303
4 zone -- last acquired @ /usr/src/sys/vm/vm_zone.c:506
1 pcm0:record:0 -- last acquired @ /usr/src/sys/modules/sound/pcm/../../../dev/sound/pcm/channel.c:674
3 lockmgr -- last acquired @ /usr/src/sys/kern/kern_lock.c:227
3 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:303
4 zone -- last acquired @ /usr/src/sys/vm/vm_zone.c:506
1 pcm0:play:0 -- last acquired @ /usr/src/sys/modules/sound/pcm/../../../dev/sound/pcm/channel.c:674
3 lockmgr -- last acquired @ /usr/src/sys/kern/kern_lock.c:227
3 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:303
4 zone -- last acquired @ /usr/src/sys/vm/vm_zone.c:506
1 pcm0 -- last acquired @ /usr/src/sys/modules/sound/pcm/../../../dev/sound/pcm/sound.c:134
2 pcm0:fake -- last acquired @ /usr/src/sys/modules/sound/pcm/../../../dev/sound/pcm/channel.c:674
3 lockmgr -- last acquired @ /usr/src/sys/kern/kern_lock.c:227
3 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:303
4 zone -- last acquired @ /usr/src/sys/vm/vm_zone.c:506
1 mntid -- last acquired @ /usr/src/sys/kern/vfs_subr.c:440
2 mountlist -- last acquired @ /usr/src/sys/kern/vfs_syscalls.c:674
1 pbuf mutex -- last acquired @ /usr/src/sys/vm/vm_pager.c:466
5 ucred -- last acquired @ /usr/src/sys/kern/kern_prot.c:1601
1 random reseed -- last acquired @ /usr/src/sys/modules/random/../../dev/random/yarrow.c:172
1 pcm0:mixer -- last acquired @ /usr/src/sys/modules/sound/pcm/../../../dev/sound/pcm/sound.c:134
2 pcm0:ac97 -- last acquired @ /usr/src/sys/modules/sound/pcm/../../../dev/sound/pcm/sound.c:134
3 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:303
4 zone -- last acquired @ /usr/src/sys/vm/vm_zone.c:506
1 ufs ihash -- last acquired @ /usr/src/sys/ufs/ufs/ufs_ihash.c:148
3 vnode interlock -- last acquired @ /usr/src/sys/kern/vfs_vnops.c:697
4 spechash -- last acquired @ /usr/src/sys/kern/vfs_subr.c:2287
4 vnode_free_list -- last acquired @ /usr/src/sys/kern/vfs_subr.c:2238
4 mntvnode -- last acquired @ /usr/src/sys/kern/vfs_subr.c:2598
1 buftime lock -- last acquired @ /usr/src/sys/sys/buf.h:278
1 dirhash list -- last acquired @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:334
2 dirhash -- last acquired @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:353
1 filelist lock -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1074
2 filedesc structure -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1259
3 vnode interlock -- last acquired @ /usr/src/sys/kern/vfs_vnops.c:697
4 spechash -- last acquired @ /usr/src/sys/kern/vfs_subr.c:2287
4 vnode_free_list -- last acquired @ /usr/src/sys/kern/vfs_subr.c:2238
4 mntvnode -- last acquired @ /usr/src/sys/kern/vfs_subr.c:2598
5 ucred -- last acquired @ /usr/src/sys/kern/kern_prot.c:1601
4 zone -- last acquired @ /usr/src/sys/vm/vm_zone.c:506
1 dksort -- last acquired @ /usr/src/sys/kern/subr_disklabel.c:95
1 ifsvgt -- last acquired @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1223
1 vnode pollinfo -- last acquired @ /usr/src/sys/kern/vfs_subr.c:2776
1 vm object_list -- last acquired @ /usr/src/sys/vm/vm_object.c:553
1 pseudofs_vncache -- last acquired @ /usr/src/sys/modules/pseudofs/../../fs/pseudofs/pseudofs_vncache.c:211
1 pseudofs -- last acquired @ /usr/src/sys/modules/pseudofs/../../fs/pseudofs/pseudofs_fileno.c:87
1 msdosfs dehash -- last acquired @ /usr/src/sys/modules/msdosfs/../../fs/msdosfs/msdosfs_denode.c:133
3 vnode interlock -- last acquired @ /usr/src/sys/kern/vfs_vnops.c:697
4 spechash -- last acquired @ /usr/src/sys/kern/vfs_subr.c:2287
4 vnode_free_list -- last acquired @ /usr/src/sys/kern/vfs_subr.c:2238
4 mntvnode -- last acquired @ /usr/src/sys/kern/vfs_subr.c:2598
1 ip_inq -- last acquired @ /usr/src/sys/netinet/ip_input.c:857
Spin locks:
0 sio -- last acquired @ /usr/src/sys/dev/sio/sio.c:3137
1 cy -- last acquired @ order list:0
2 ng_node -- last acquired @ order list:0
3 ng_worklist -- last acquired @ order list:0
4 ithread table lock -- last acquired @ /usr/src/sys/i386/isa/intr_machdep.c:618
5 sched lock -- last acquired @ /usr/src/sys/kern/kern_clock.c:470
6 callout -- last acquired @ /usr/src/sys/kern/kern_clock.c:232
7 allpmaps -- last acquired @ order list:0
8 icu -- last acquired @ /usr/src/sys/i386/isa/intr_machdep.c:584
9 clk -- last acquired @ /usr/src/sys/i386/isa/clock.c:1232
Locks which were never acquired:
gif
pseudofs_fileno
arp_inq
ip6_inq
bpf interface lock
lo
ACPI global lock
taskqueue
mbuf subsystem general lists lock
phys_pager list
dev_pager list
dev_pager create
swap_pager list
vm buckets hash mutexes
vm pageq mutex
db> p show pcpu
cpuid = 0
curthread = 0xcc322c04: pid 25 "irq1: atkbd0"
curpcb = 0xcc977da0
fpcurthread = none
idlethread = 0xcc325904: pid 10 "idle"
currentldt = 0x28
spin locks held:
db>
help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020208161740.A28661>