Date: Thu, 10 Feb 2005 03:04:12 -0800 (PST) From: Don Lewis <truckman@FreeBSD.org> To: jhb@FreeBSD.org Cc: freebsd-current@FreeBSD.org Subject: Re: odd -CURRENT performance issue Message-ID: <200502101104.j1AB4C54009051@gw.catspoiler.org> In-Reply-To: <200502082329.j18NT9DE004558@gw.catspoiler.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 8 Feb, Don Lewis wrote:
> On 8 Feb, John Baldwin wrote:
>> WITNESS has especially poor scalability issues and can get much worse as more
>> lock classes and relationships are added to its tree due to how it tries to
>> rebalance the tree every time it adds a new relationship (and each rebalance
>> is O(n) I think). If you turn off witness via sysctl debug.witness.watch=0,.
>> does the same weirdness persist?
>
> I'm in the middle of a portupgrade run to catch up with the latest perl
> upgrade, so it'll be a while before I can try that experiment. Earlier
> today I broke into DDB a bunch of times to get some stack traces to try
> to figure out where the kernel was spending most of its time. Most of
> the stack traces had this in common:
>
> critical_exit(c094d12c,eb27bbac,c0643c27,c0906a20,0) at critical_exit+0xb2
> _mtx_unlock_spin_flags(c0906a20,0,c083c0bd,325,c28515c0) at _mtx_unlock_spin_flags+0x8d
> witness_checkorder(c2dad7bc,9,c0852ab7,9f) at witness_checkorder+0x29f
> _mtx_lock_flags(c2dad7bc,0,c0852ab7,9f,22) at _mtx_lock_flags+0x5b
>
> where witness_checkorder+0x29f is here:
>
> /*
> * If we know that the the lock we are acquiring comes after
> * the lock we most recently acquired in the lock order tree,
> * then there is no need for any further checks.
> */
> if (isitmydescendant(w1, w)) {
> mtx_unlock_spin(&w_mtx);
>
> and by far the majority of the callers were calling VM_OBJECT_LOCK().
>
> I suspect that the tree has gotten rebalanced so that the "vm object"
> lock suddenly takes a much longer time to find. I'll try to get
> snapshots of the tree before and after the problem occurs.
I forgot to try turning off debug.witness.watch, but I did a "show
witness" when the system was slow and then after a reboot which made the
system was fast again. It looks like the discovery of the
FFS Lock -> standard object
relationship, which causes witness to delete the direct link
Giant -> standard object
is what is throwing out the boat anchor. Without this direct
parent/child relationship, isitmydescendant() has a much longer search
to do.
I just looked closely at the rebalancing code. Gak!
Here are the outputs of show witness in the fast and slow cases:
Fast:
Sleep locks:
0 g_xdown -- last acquired @ /usr/src/sys/geom/geom_io.c:423
3 Giant -- last acquired @ /usr/src/sys/kern/kern_timeout.c:247
4 UMA lock -- last acquired @ /usr/src/sys/vm/uma_core.c:1491
12 Malloc Stats -- last acquired @ /usr/src/sys/kern/kern_malloc.c:228
16 UMA pcpu -- last acquired @ /usr/src/sys/vm/uma_core.c:1825
17 KMAP ENTRY -- last acquired @ /usr/src/sys/vm/uma_core.c:418
18 UMA zone -- last acquired @ /usr/src/sys/vm/uma_core.c:1842
4 standard object -- last acquired @ /usr/src/sys/vm/vm_object.c:449
5 vm object_list -- last acquired @ /usr/src/sys/vm/vm_object.c:643
5 swap_pager swhash -- last acquired @ /usr/src/sys/vm/swap_pager.c:1961
14 vm page queue mutex -- last acquired @ /usr/src/sys/vm/vm_fault.c:902
15 vnode interlock -- last acquired @ /usr/src/sys/kern/vfs_subr.c:1898
16 Syncer mtx -- last acquired @ /usr/src/sys/kern/vfs_subr.c:1617
16 vnode_free_list -- last acquired @ /usr/src/sys/kern/vfs_subr.c:590
16 cdev -- last acquired @ /usr/src/sys/kern/kern_conf.c:70
15 pmap -- last acquired @ /usr/src/sys/i386/i386/pmap.c:2688
16 UMA pcpu -- (already displayed)
16 uma object -- last acquired @ /usr/src/sys/vm/uma_core.c:988
16 SYSMAPS -- last acquired @ /usr/src/sys/i386/i386/pmap.c:2422
4 kernel linker -- last acquired @ /usr/src/sys/kern/kern_linker.c:461
4 eventhandler -- last acquired @ /usr/src/sys/kern/subr_eventhandler.c:213
5 eventhandler list -- last acquired @ /usr/src/sys/kern/kern_exit.c:199
4 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:584
4 filedesc structure -- last acquired @ /usr/src/sys/kern/sys_generic.c:814
6 accept -- last acquired @ /usr/src/sys/kern/uipc_socket.c:460
7 so_snd -- last acquired @ /usr/src/sys/kern/uipc_socket.c:2112
8 tcp_hc_entry -- last acquired @ /usr/src/sys/netinet/tcp_hostcache.c:286
8 so_rcv -- last acquired @ /usr/src/sys/kern/uipc_socket.c:2113
9 radix node head -- last acquired @ /usr/src/sys/net/route.c:148
10 ifnet -- last acquired @ /usr/src/sys/net/if.c:1056
10 rtentry -- last acquired @ /usr/src/sys/netinet/ip_output.c:822
11 network driver -- last acquired @ /usr/src/sys/dev/fxp/if_fxp.c:1756
12 if send queue -- last acquired @ /usr/src/sys/dev/fxp/if_fxp.c:1268
12 Malloc Stats -- (already displayed)
12 knlist lock for lockless objects -- last acquired @ /usr/src/sys/kern/kern_event.c:1453
12 system map -- last acquired @ /usr/src/sys/vm/vm_map.c:2317
13 kmem object -- last acquired @ /usr/src/sys/vm/vm_kern.c:397
14 vm page queue mutex -- (already displayed)
13 kernel object -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3705
14 vm page queue mutex -- (already displayed)
11 rts_inq -- last acquired @ /usr/src/sys/net/netisr.c:232
11 ifaddr -- last acquired @ /usr/src/sys/net/route.c:788
9 sellck -- last acquired @ /usr/src/sys/kern/sys_generic.c:713
9 process lock -- last acquired @ /usr/src/sys/kern/kern_resource.c:850
10 ktrace -- last acquired @ /usr/src/sys/kern/kern_exit.c:354
10 sigacts -- last acquired @ /usr/src/sys/kern/subr_sleepqueue.c:369
10 struct pargs.ref -- last acquired @ /usr/src/sys/kern/kern_proc.c:1110
10 session -- last acquired @ /usr/src/sys/kern/kern_proc.c:450
11 tty -- last acquired @ /usr/src/sys/kern/kern_event.c:1453
15 vnode interlock -- (already displayed)
11 uidinfo hash -- last acquired @ /usr/src/sys/kern/kern_resource.c:1034
12 sleep mtxpool -- last acquired @ /usr/src/sys/kern/kern_prot.c:1749
12 uidinfo struct -- last acquired @ order list:0
13 allprison -- last acquired @ /usr/src/sys/kern/kern_jail.c:460
5 pipe mutex -- last acquired @ /usr/src/sys/kern/sys_pipe.c:1340
9 sellck -- (already displayed)
6 sigio lock -- last acquired @ /usr/src/sys/kern/kern_descrip.c:739
7 process group -- last acquired @ /usr/src/sys/kern/kern_proc.c:257
9 process lock -- (already displayed)
4 rman -- last acquired @ /usr/src/sys/kern/subr_rman.c:448
12 Malloc Stats -- (already displayed)
12 system map -- (already displayed)
4 kobj -- last acquired @ /usr/src/sys/kern/subr_kobj.c:298
4 devd -- last acquired @ /usr/src/sys/kern/subr_bus.c:497
9 sellck -- (already displayed)
4 bounce pages lock -- last acquired @ /usr/src/sys/i386/i386/busdma_machdep.c:1054
4 TID lock -- last acquired @ /usr/src/sys/kern/kern_thread.c:204
4 vm86 lock -- last acquired @ /usr/src/sys/i386/i386/vm86.c:581
4 acpica subsystem lock -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSynch.c:360
4 ithread -- last acquired @ /usr/src/sys/kern/kern_intr.c:275
4 ttylist -- last acquired @ /usr/src/sys/kern/tty.c:2799
11 tty -- (already displayed)
4 GEOM orphanage -- last acquired @ /usr/src/sys/geom/geom_event.c:188
4 ACPI semaphore -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSynch.c:300
4 taskqueue list -- last acquired @ /usr/src/sys/kern/subr_taskqueue.c:85
4 domain list -- last acquired @ /usr/src/sys/kern/uipc_domain.c:224
4 rman head -- last acquired @ /usr/src/sys/kern/subr_rman.c:111
4 bdone lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3031
4 sf_buf -- last acquired @ /usr/src/sys/i386/i386/vm_machdep.c:680
4 taskqueue -- last acquired @ /usr/src/sys/kern/subr_taskqueue.c:194
4 bpf global lock -- last acquired @ /usr/src/sys/net/bpf.c:1446
5 bpf interface lock -- last acquired @ order list:0
6 bpf cdev lock -- last acquired @ order list:0
4 pseudofs -- last acquired @ /usr/src/sys/fs/pseudofs/pseudofs_fileno.c:86
4 nfsd_mtx -- last acquired @ /usr/src/sys/nfsserver/nfs_srvsock.c:811
8 so_rcv -- (already displayed)
4 if_cloners lock -- last acquired @ /usr/src/sys/net/if_clone.c:199
4 Name Cache -- last acquired @ /usr/src/sys/kern/vfs_cache.c:347
15 vnode interlock -- (already displayed)
16 UMA pcpu -- (already displayed)
4 if_clone lock -- last acquired @ /usr/src/sys/net/if_clone.c:321
4 tcp -- last acquired @ /usr/src/sys/netinet/tcp_subr.c:1359
5 tcpinp -- last acquired @ /usr/src/sys/netinet/tcp_input.c:747
6 so_glabel -- last acquired @ /usr/src/sys/kern/uipc_socket.c:273
6 random reseed -- last acquired @ /usr/src/sys/dev/random/yarrow.c:193
6 ip_inq -- last acquired @ /usr/src/sys/net/netisr.c:232
6 arc4_mtx -- last acquired @ /usr/src/sys/libkern/arc4random.c:137
6 accept -- (already displayed)
4 pbuf mutex -- last acquired @ /usr/src/sys/vm/vm_pager.c:410
4 lo_mtx -- last acquired @ /usr/src/sys/net/if_loop.c:154
4 pfil_head_mtx -- last acquired @ /usr/src/sys/net/pfil.c:174
5 pfil_head_list lock -- last acquired @ /usr/src/sys/net/pfil.c:180
4 pseudofs_vncache -- last acquired @ /usr/src/sys/fs/pseudofs/pseudofs_vncache.c:243
15 vnode interlock -- (already displayed)
4 needsbuffer lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:313
4 Softdep Lock -- last acquired @ /usr/src/sys/ufs/ffs/ffs_softdep.c:4420
5 buffer daemon lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:421
12 Malloc Stats -- (already displayed)
15 vnode interlock -- (already displayed)
16 UMA pcpu -- (already displayed)
4 runningbufspace lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:332
4 devstat -- last acquired @ /usr/src/sys/kern/subr_devstat.c:190
4 ACPI task queue -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSchedule.c:118
4 mntid -- last acquired @ /usr/src/sys/kern/vfs_subr.c:415
5 mountlist -- last acquired @ /usr/src/sys/kern/vfs_subr.c:3078
6 struct mount mtx -- last acquired @ /usr/src/sys/kern/vfs_vnops.c:957
15 vnode interlock -- (already displayed)
4 buf queue lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:1544
15 vnode interlock -- (already displayed)
4 rtsock route_cb lock -- last acquired @ /usr/src/sys/net/rtsock.c:234
4 FFS Lock -- last acquired @ /usr/src/sys/ufs/ffs/ffs_softdep.c:2316
4 fdesc -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1425
4 g_disk_done -- last acquired @ /usr/src/sys/geom/geom_disk.c:198
5 bio queue -- last acquired @ /usr/src/sys/geom/geom_io.c:66
16 UMA pcpu -- (already displayed)
4 dirhash list -- last acquired @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:348
5 dirhash -- last acquired @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:456
4 rawcb -- last acquired @ /usr/src/sys/net/raw_cb.c:105
8 so_rcv -- (already displayed)
4 udp -- last acquired @ /usr/src/sys/netinet/udp_usrreq.c:246
5 udp6inp -- last acquired @ /usr/src/sys/netinet6/udp6_usrreq.c:670
6 if_afdata -- last acquired @ /usr/src/sys/netinet6/scope6.c:272
6 ip6_inq -- last acquired @ /usr/src/sys/net/netisr.c:232
6 accept -- (already displayed)
5 udpinp -- last acquired @ /usr/src/sys/netinet/udp_usrreq.c:386
6 arc4_mtx -- (already displayed)
6 NFS reqq lock -- last acquired @ /usr/src/sys/nfsclient/nfs_socket.c:1049
7 Synch NFS reply posting -- last acquired @ /usr/src/sys/nfsclient/nfs_socket.c:584
9 process lock -- (already displayed)
6 accept -- (already displayed)
4 ufs ihash -- last acquired @ /usr/src/sys/ufs/ufs/ufs_ihash.c:156
15 vnode interlock -- (already displayed)
4 semid -- last acquired @ /usr/src/sys/kern/sysv_sem.c:1002
5 sem -- last acquired @ /usr/src/sys/kern/sysv_sem.c:1258
4 unp -- last acquired @ /usr/src/sys/kern/uipc_usrreq.c:329
6 accept -- (already displayed)
0 g_xup -- last acquired @ /usr/src/sys/geom/geom_io.c:484
3 Giant -- (already displayed)
0 arp_inq -- last acquired @ /usr/src/sys/net/netisr.c:232
0 igmp_mtx -- last acquired @ /usr/src/sys/netinet/igmp.c:431
0 ipqlock -- last acquired @ /usr/src/sys/netinet/ip_input.c:1110
16 UMA pcpu -- (already displayed)
0 ip6qlock -- last acquired @ /usr/src/sys/netinet6/frag6.c:682
0 nfslock -- last acquired @ /usr/src/sys/nfsclient/nfs_lock.c:123
0 GEOM topology -- last acquired @ /usr/src/sys/geom/geom_event.c:220
1 fdc lock -- last acquired @ /usr/src/sys/dev/fdc/fdc.c:777
2 callout_wait_lock -- last acquired @ /usr/src/sys/kern/kern_timeout.c:289
1 swapdev -- last acquired @ /usr/src/sys/vm/swap_pager.c:2102
3 Giant -- (already displayed)
0 p_peers -- last acquired @ /usr/src/sys/kern/kern_exit.c:250
0 ACPI PCI power methods -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pci.c:207
0 ACPI root bus -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi.c:969
4 rman -- (already displayed)
4 ACPI semaphore -- (already displayed)
0 ACPI PCI bus methods -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pcib.c:214
1 ACPI PCI link -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pci_link.c:972
2 kernel environment -- last acquired @ /usr/src/sys/kern/kern_environment.c:285
4 ACPI semaphore -- (already displayed)
12 Malloc Stats -- (already displayed)
16 UMA pcpu -- (already displayed)
0 ACPI CPU -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_cpu.c:817
0 module subsystem sx lock -- last acquired @ /usr/src/sys/kern/kern_module.c:115
0 protect sysfilt_ops -- last acquired @ /usr/src/sys/kern/kern_event.c:669
0 sysctl lock -- last acquired @ /usr/src/sys/kern/kern_sysctl.c:1335
1 filelist lock -- last acquired @ /usr/src/sys/kern/kern_descrip.c:2128
4 filedesc structure -- (already displayed)
1 addrsel_lock -- last acquired @ /usr/src/sys/netinet6/in6_src.c:1137
1 rip -- last acquired @ /usr/src/sys/netinet/raw_ip.c:866
1 allproc -- last acquired @ /usr/src/sys/kern/sched_4bsd.c:435
2 user map -- last acquired @ /usr/src/sys/vm/vm_map.c:2998
3 Giant -- (already displayed)
0 ddp_list_mtx -- last acquired @ order list:0
1 ddp_mtx -- last acquired @ order list:0
0 slip_mtx -- last acquired @ order list:0
1 slip sc_mtx -- last acquired @ order list:0
0 proctree -- last acquired @ /usr/src/sys/kern/tty.c:2076
1 allproc -- (already displayed)
Slow:
Sleep locks:
0 g_xdown -- last acquired @ /usr/src/sys/geom/geom_io.c:423
3 Giant -- last acquired @ /usr/src/sys/kern/kern_timeout.c:247
4 UMA lock -- last acquired @ /usr/src/sys/vm/uma_core.c:1491
12 Malloc Stats -- last acquired @ /usr/src/sys/kern/kern_malloc.c:228
12 system map -- last acquired @ /usr/src/sys/vm/vm_map.c:2317
13 kmem object -- last acquired @ /usr/src/sys/vm/vm_kern.c:397
14 vm page queue mutex -- last acquired @ /usr/src/sys/vm/vm_pageout.c:1433
15 vnode interlock -- last acquired @ /usr/src/sys/kern/vfs_subr.c:1489
16 vnode_free_list -- last acquired @ /usr/src/sys/kern/vfs_subr.c:590
16 Syncer mtx -- last acquired @ /usr/src/sys/kern/vfs_subr.c:1617
16 cdev -- last acquired @ /usr/src/sys/kern/kern_conf.c:70
15 pmap -- last acquired @ /usr/src/sys/i386/i386/pmap.c:1626
16 UMA pcpu -- last acquired @ /usr/src/sys/vm/uma_core.c:2251
17 KMAP ENTRY -- last acquired @ /usr/src/sys/vm/uma_core.c:418
18 UMA zone -- last acquired @ /usr/src/sys/vm/uma_core.c:2269
16 uma object -- last acquired @ /usr/src/sys/vm/uma_core.c:988
16 SYSMAPS -- last acquired @ /usr/src/sys/i386/i386/pmap.c:2422
13 kernel object -- last acquired @ /usr/src/sys/vm/vm_object.c:449
14 vm page queue mutex -- (already displayed)
4 vm86 lock -- last acquired @ /usr/src/sys/i386/i386/vm86.c:581
4 eventhandler -- last acquired @ /usr/src/sys/kern/subr_eventhandler.c:213
5 eventhandler list -- last acquired @ /usr/src/sys/kern/kern_exit.c:199
4 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:584
4 kernel linker -- last acquired @ /usr/src/sys/kern/kern_linker.c:1061
4 filedesc structure -- last acquired @ /usr/src/sys/kern/sys_generic.c:814
6 accept -- last acquired @ /usr/src/sys/kern/uipc_socket.c:460
7 so_snd -- last acquired @ /usr/src/sys/kern/uipc_socket.c:2112
8 tcp_hc_entry -- last acquired @ /usr/src/sys/netinet/tcp_hostcache.c:654
12 system map -- (already displayed)
8 so_rcv -- last acquired @ /usr/src/sys/kern/uipc_socket.c:2113
9 sellck -- last acquired @ /usr/src/sys/kern/sys_generic.c:713
9 radix node head -- last acquired @ /usr/src/sys/net/route.c:148
10 ifnet -- last acquired @ /usr/src/sys/net/if.c:1056
11 random reseed -- last acquired @ /usr/src/sys/dev/random/yarrow.c:193
11 arc4_mtx -- last acquired @ /usr/src/sys/libkern/arc4random.c:137
10 rtentry -- last acquired @ /usr/src/sys/netinet/ip_output.c:822
11 rts_inq -- last acquired @ /usr/src/sys/net/netisr.c:232
11 network driver -- last acquired @ /usr/src/sys/dev/fxp/if_fxp.c:1756
12 Malloc Stats -- (already displayed)
12 knlist lock for lockless objects -- last acquired @ /usr/src/sys/kern/kern_event.c:1453
12 if send queue -- last acquired @ /usr/src/sys/dev/fxp/if_fxp.c:1268
12 system map -- (already displayed)
11 ifaddr -- last acquired @ /usr/src/sys/net/route.c:788
9 process lock -- last acquired @ /usr/src/sys/kern/subr_sleepqueue.c:368
10 struct pargs.ref -- last acquired @ /usr/src/sys/kern/kern_proc.c:1110
10 ktrace -- last acquired @ /usr/src/sys/kern/kern_exit.c:354
10 sigacts -- last acquired @ /usr/src/sys/kern/subr_sleepqueue.c:369
10 session -- last acquired @ /usr/src/sys/kern/kern_proc.c:450
15 vnode interlock -- (already displayed)
11 tty -- last acquired @ /usr/src/sys/kern/kern_event.c:1453
11 uidinfo hash -- last acquired @ /usr/src/sys/kern/kern_resource.c:1034
12 sleep mtxpool -- last acquired @ /usr/src/sys/kern/kern_descrip.c:2099
12 uidinfo struct -- last acquired @ order list:0
13 allprison -- last acquired @ /usr/src/sys/kern/kern_jail.c:460
5 pipe mutex -- last acquired @ /usr/src/sys/kern/sys_pipe.c:1340
14 vm page queue mutex -- (already displayed)
9 sellck -- (already displayed)
6 sigio lock -- last acquired @ /usr/src/sys/kern/kern_descrip.c:739
7 process group -- last acquired @ /usr/src/sys/kern/kern_proc.c:257
9 process lock -- (already displayed)
4 devd -- last acquired @ /usr/src/sys/kern/subr_bus.c:497
9 sellck -- (already displayed)
4 bounce pages lock -- last acquired @ /usr/src/sys/i386/i386/busdma_machdep.c:1054
4 kobj -- last acquired @ /usr/src/sys/kern/subr_kobj.c:298
4 TID lock -- last acquired @ /usr/src/sys/kern/kern_thread.c:204
4 rman -- last acquired @ /usr/src/sys/kern/subr_rman.c:448
12 Malloc Stats -- (already displayed)
12 system map -- (already displayed)
4 bpf global lock -- last acquired @ /usr/src/sys/net/bpf.c:1446
5 bpf interface lock -- last acquired @ order list:0
6 bpf cdev lock -- last acquired @ order list:0
4 ACPI semaphore -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSynch.c:300
4 acpica subsystem lock -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSynch.c:360
4 GEOM orphanage -- last acquired @ /usr/src/sys/geom/geom_event.c:188
4 ithread -- last acquired @ /usr/src/sys/kern/kern_intr.c:275
4 ttylist -- last acquired @ /usr/src/sys/kern/tty.c:2799
11 tty -- (already displayed)
4 taskqueue list -- last acquired @ /usr/src/sys/kern/subr_taskqueue.c:85
4 rman head -- last acquired @ /usr/src/sys/kern/subr_rman.c:111
4 taskqueue -- last acquired @ /usr/src/sys/kern/subr_taskqueue.c:194
4 domain list -- last acquired @ /usr/src/sys/kern/uipc_domain.c:224
4 bdone lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3031
4 sf_buf -- last acquired @ /usr/src/sys/i386/i386/vm_machdep.c:680
4 tcp -- last acquired @ /usr/src/sys/netinet/tcp_subr.c:1359
5 tcpinp -- last acquired @ /usr/src/sys/netinet/tcp_input.c:747
6 ip6_inq -- last acquired @ /usr/src/sys/net/netisr.c:232
6 if_afdata -- last acquired @ /usr/src/sys/netinet6/scope6.c:272
6 so_glabel -- last acquired @ /usr/src/sys/kern/uipc_socket.c:273
6 ip_inq -- last acquired @ /usr/src/sys/net/netisr.c:232
6 accept -- (already displayed)
4 if_cloners lock -- last acquired @ /usr/src/sys/net/if_clone.c:199
4 pseudofs -- last acquired @ /usr/src/sys/fs/pseudofs/pseudofs_fileno.c:86
4 Name Cache -- last acquired @ /usr/src/sys/kern/vfs_cache.c:347
12 system map -- (already displayed)
4 if_clone lock -- last acquired @ /usr/src/sys/net/if_clone.c:321
4 nfsd_mtx -- last acquired @ /usr/src/sys/nfsserver/nfs_srvsock.c:811
8 so_rcv -- (already displayed)
4 udp -- last acquired @ /usr/src/sys/netinet/udp_usrreq.c:246
5 udp6inp -- last acquired @ /usr/src/sys/netinet6/udp6_usrreq.c:670
6 ip6_inq -- (already displayed)
6 if_afdata -- (already displayed)
6 accept -- (already displayed)
5 udpinp -- last acquired @ /usr/src/sys/netinet/udp_usrreq.c:386
6 ip_inq -- (already displayed)
6 NFS reqq lock -- last acquired @ /usr/src/sys/nfsclient/nfs_socket.c:1049
7 Synch NFS reply posting -- last acquired @ /usr/src/sys/nfsclient/nfs_socket.c:584
6 accept -- (already displayed)
4 runningbufspace lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:332
4 pfil_head_mtx -- last acquired @ /usr/src/sys/net/pfil.c:174
5 pfil_head_list lock -- last acquired @ /usr/src/sys/net/pfil.c:180
4 pbuf mutex -- last acquired @ /usr/src/sys/vm/vm_pager.c:410
4 lo_mtx -- last acquired @ /usr/src/sys/net/if_loop.c:154
4 pseudofs_vncache -- last acquired @ /usr/src/sys/fs/pseudofs/pseudofs_vncache.c:243
15 vnode interlock -- (already displayed)
4 rtsock route_cb lock -- last acquired @ /usr/src/sys/net/rtsock.c:234
4 ACPI task queue -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSchedule.c:118
4 devstat -- last acquired @ /usr/src/sys/kern/subr_devstat.c:83
4 fdesc -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1425
4 mntid -- last acquired @ /usr/src/sys/kern/vfs_subr.c:415
5 mountlist -- last acquired @ /usr/src/sys/kern/vfs_subr.c:3078
6 struct mount mtx -- last acquired @ /usr/src/sys/kern/vfs_vnops.c:984
15 vnode interlock -- (already displayed)
4 rawcb -- last acquired @ /usr/src/sys/net/raw_cb.c:105
8 so_rcv -- (already displayed)
4 unp -- last acquired @ /usr/src/sys/kern/uipc_usrreq.c:329
6 accept -- (already displayed)
4 FFS Lock -- last acquired @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1044
5 needsbuffer lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:313
5 buf queue lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:1544
6 struct mount mtx -- (already displayed)
6 Softdep Lock -- last acquired @ /usr/src/sys/ufs/ffs/ffs_softdep.c:402
7 buffer daemon lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:2151
12 Malloc Stats -- (already displayed)
12 system map -- (already displayed)
5 standard object -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3523
6 vm object_list -- last acquired @ /usr/src/sys/vm/vm_object.c:643
6 swap_pager swhash -- last acquired @ /usr/src/sys/vm/swap_pager.c:1961
7 swapdev -- last acquired @ /usr/src/sys/vm/swap_pager.c:2245
16 uma object -- (already displayed)
16 UMA pcpu -- (already displayed)
14 vm page queue mutex -- (already displayed)
4 g_disk_done -- last acquired @ /usr/src/sys/geom/geom_disk.c:198
5 bio queue -- last acquired @ /usr/src/sys/geom/geom_io.c:66
12 system map -- (already displayed)
4 rip -- last acquired @ /usr/src/sys/netinet/raw_ip.c:195
6 accept -- (already displayed)
4 semid -- last acquired @ /usr/src/sys/kern/sysv_sem.c:1286
5 sem -- last acquired @ /usr/src/sys/kern/sysv_sem.c:1258
4 dirhash list -- last acquired @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:348
5 dirhash -- last acquired @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:349
4 ufs ihash -- last acquired @ /usr/src/sys/ufs/ufs/ufs_ihash.c:156
15 vnode interlock -- (already displayed)
4 linux osname -- last acquired @ /usr/src/sys/modules/linux/../../compat/linux/linux_mib.c:231
4 ipqlock -- last acquired @ /usr/src/sys/netinet/ip_input.c:1110
16 UMA pcpu -- (already displayed)
0 g_xup -- last acquired @ /usr/src/sys/geom/geom_io.c:484
3 Giant -- (already displayed)
0 arp_inq -- last acquired @ /usr/src/sys/net/netisr.c:232
0 igmp_mtx -- last acquired @ /usr/src/sys/netinet/igmp.c:431
0 ip6qlock -- last acquired @ /usr/src/sys/netinet6/frag6.c:682
0 nfslock -- last acquired @ /usr/src/sys/nfsclient/nfs_lock.c:123
0 GEOM topology -- last acquired @ /usr/src/sys/geom/geom_event.c:220
1 fdc lock -- last acquired @ /usr/src/sys/dev/fdc/fdc.c:777
2 callout_wait_lock -- last acquired @ /usr/src/sys/kern/kern_timeout.c:289
3 Giant -- (already displayed)
0 p_peers -- last acquired @ /usr/src/sys/kern/kern_exit.c:250
0 ACPI PCI power methods -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pci.c:207
0 ACPI root bus -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi.c:969
4 rman -- (already displayed)
4 ACPI semaphore -- (already displayed)
0 ACPI PCI bus methods -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pcib.c:214
1 ACPI PCI link -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pci_link.c:972
4 ACPI semaphore -- (already displayed)
12 Malloc Stats -- (already displayed)
2 kernel environment -- last acquired @ /usr/src/sys/kern/kern_environment.c:285
16 UMA pcpu -- (already displayed)
0 ACPI CPU -- last acquired @ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_cpu.c:817
0 module subsystem sx lock -- last acquired @ /usr/src/sys/kern/kern_module.c:115
0 protect sysfilt_ops -- last acquired @ /usr/src/sys/kern/kern_event.c:669
0 sysctl lock -- last acquired @ /usr/src/sys/kern/kern_sysctl.c:1335
1 filelist lock -- last acquired @ /usr/src/sys/kern/kern_descrip.c:2128
4 filedesc structure -- (already displayed)
1 addrsel_lock -- last acquired @ /usr/src/sys/netinet6/in6_src.c:1137
1 allproc -- last acquired @ /usr/src/sys/kern/sched_4bsd.c:435
2 user map -- last acquired @ /usr/src/sys/vm/vm_glue.c:176
3 Giant -- (already displayed)
0 ddp_list_mtx -- last acquired @ order list:0
1 ddp_mtx -- last acquired @ order list:0
0 slip_mtx -- last acquired @ order list:0
1 slip sc_mtx -- last acquired @ order list:0
0 proctree -- last acquired @ /usr/src/sys/kern/kern_exit.c:591
1 allproc -- (already displayed)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200502101104.j1AB4C54009051>