From owner-freebsd-current@FreeBSD.ORG Wed Mar 19 02:25:11 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C27221065672 for ; Wed, 19 Mar 2008 02:25:11 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.182]) by mx1.freebsd.org (Postfix) with ESMTP id 90E138FC1B for ; Wed, 19 Mar 2008 02:25:11 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: by wa-out-1112.google.com with SMTP id k17so191026waf.3 for ; Tue, 18 Mar 2008 19:25:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=8Con8OOjGCXODleeoyao6JjM21BXc52/qqmdNgr4QPU=; b=ABpLDHTbxic7CrReHjpN2Nl2V0Hppd9AFE700nC36fKl9L6EqGYxHOGAv7sFEG3fYFuJoZADa2+1wZyo68QBSAX5WGt5Eq8F9Ukx+R8H1jzmKvQzmV8zKUdti04ifc5SmDO7pgszqElSkQqAIbKmJXivqWq7rOts9ZmUb/6Gvwk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=ARpOiIta3aL0M+KpL+DvwBn8to7Ka/UFUE2v/RQKWx/WZS2w3GPNf20LNld4Hs1C3WwOxLsXCdOrC/8fixRV2416RXxPDLeBGwJcZuLw0IDF1KTvBFAWvWQmz/XQvesrzHpz7Wtkt3p++1qeAWcMIK02ZClkpPpm/Jly0RlRGgE= Received: by 10.114.132.5 with SMTP id f5mr393658wad.50.1205893511181; Tue, 18 Mar 2008 19:25:11 -0700 (PDT) Received: by 10.115.22.10 with HTTP; Tue, 18 Mar 2008 19:25:11 -0700 (PDT) Message-ID: Date: Tue, 18 Mar 2008 19:25:11 -0700 From: "Kip Macy" To: "Robert Huff" In-Reply-To: <18400.30930.430744.931475@jerusalem.litteratus.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <18399.51985.110882.166457@jerusalem.litteratus.org> <18400.30930.430744.931475@jerusalem.litteratus.org> Cc: freebsd-current@freebsd.org, Alex Goncharov Subject: Re: Seeing lock order reversal X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Mar 2008 02:25:12 -0000 On Tue, Mar 18, 2008 at 7:22 PM, Robert Huff wrote: > Kip Macy writes: > > > > I'm gettig both of these as well. It doesn't stop the system > > > from booting, or _seem_ to affect operation ... but it would be nice > > > if they would go away. > < > > > See archives / UPDATING for an explanation. > WITNESS support was recently added for lockmgr - the locking primitive used by FreeBSD's file systems. If that fact is not in UPDATING it *should be* (this means you Attilio). Most of the VFS LORs you are seeing are likely pre-exisiting because but were kept hidden by lack of WITNESS support and only manifested themselves as periodic deadlocks. -Kip > 1) Am I correct in thinking you mean this: > > 20080301: > The layout of struct vmspace has changed. This affects libkvm > and any executables that link against libkvm and use the > kvm_getprocs() function. In particular, but not exclusively, > it affects ps(1), fstat(1), pkill(1), systat(1), top(1) and w(1). > The effects are minimal, but it's advisable to upgrade world > nonetheless. > > > 2) I updated/rebuilt/installed world today. Still getting the > LORs; see appended. > > > Robert Huff > > > Mar 18 22:10:04 jerusalem kernel: Copyright (c) 1992-2008 The FreeBSD Project. > Mar 18 22:10:04 jerusalem kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > Mar 18 22:10:04 jerusalem kernel: The Regents of the University of California. All rights reserved. > Mar 18 22:10:04 jerusalem kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. > Mar 18 22:10:04 jerusalem kernel: FreeBSD 8.0-CURRENT #0: Tue Mar 18 20:52:53 EDT 2008 > Mar 18 22:10:04 jerusalem kernel: huff@jerusalem.litteratus.org:/usr/obj/usr/src/sys/JERUSALEM > Mar 18 22:10:04 jerusalem kernel: WARNING: WITNESS option enabled, expect reduced performance. > Mar 18 22:10:04 jerusalem kernel: Timecounter "i8254" frequency 1193182 Hz quality 0 > Mar 18 22:10:04 jerusalem kernel: CPU: Intel(R) Pentium(R) 4 CPU 2.26GHz (2266.76-MHz 686-class CPU) > Mar 18 22:10:04 jerusalem kernel: Origin = "GenuineIntel" Id = 0xf24 Stepping = 4 > Mar 18 22:10:04 jerusalem kernel: Features=0x3febfbff > Mar 18 22:10:04 jerusalem kernel: real memory = 536854528 (511 MB) > Mar 18 22:10:04 jerusalem kernel: avail memory = 516349952 (492 MB) > Mar 18 22:10:04 jerusalem kernel: acpi0: on motherboard > Mar 18 22:10:04 jerusalem kernel: acpi0: [ITHREAD] > Mar 18 22:10:04 jerusalem kernel: acpi0: Power Button (fixed) > Mar 18 22:10:04 jerusalem kernel: acpi0: reservation of 0, a0000 (3) failed > Mar 18 22:10:04 jerusalem kernel: acpi0: reservation of 100000, 1ff00000 (3) failed > Mar 18 22:10:04 jerusalem kernel: Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 > Mar 18 22:10:04 jerusalem kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0xe408-0xe40b on acpi0 > Mar 18 22:10:04 jerusalem kernel: pcib0: port 0xcf8-0xcff on acpi0 > Mar 18 22:10:04 jerusalem kernel: pci0: on pcib0 > Mar 18 22:10:04 jerusalem kernel: agp0: on hostb0 > Mar 18 22:10:04 jerusalem kernel: pcib1: at device 1.0 on pci0 > Mar 18 22:10:04 jerusalem kernel: pci1: on pcib1 > Mar 18 22:10:04 jerusalem kernel: vgapci0: mem 0xfc000000-0xfdffffff,0xf3800000-0xf3803fff,0xf3000000-0xf37fffff irq 11 at device 0.0 on pci1 > Mar 18 22:10:04 jerusalem kernel: drm0: on vgapci0 > Mar 18 22:10:04 jerusalem kernel: info: [drm] AGP at 0xf4000000 64MB > Mar 18 22:10:04 jerusalem kernel: info: [drm] Initialized mga 3.2.2 20060319 > Mar 18 22:10:04 jerusalem kernel: isab0: at device 2.0 on pci0 > Mar 18 22:10:04 jerusalem kernel: isa0: on isab0 > Mar 18 22:10:04 jerusalem kernel: ohci0: mem 0xf2800000-0xf2800fff irq 5 at device 2.2 on pci0 > Mar 18 22:10:04 jerusalem kernel: ohci0: [GIANT-LOCKED] > Mar 18 22:10:04 jerusalem kernel: ohci0: [ITHREAD] > Mar 18 22:10:04 jerusalem kernel: usb0: OHCI version 1.0, legacy support > Mar 18 22:10:04 jerusalem kernel: usb0: SMM does not respond, resetting > Mar 18 22:10:04 jerusalem kernel: usb0: on ohci0 > Mar 18 22:10:04 jerusalem kernel: usb0: USB revision 1.0 > Mar 18 22:10:04 jerusalem kernel: uhub0: on usb0 > Mar 18 22:10:04 jerusalem kernel: uhub0: 3 ports with 3 removable, self powered > Mar 18 22:10:04 jerusalem kernel: ohci1: mem 0xf2000000-0xf2000fff irq 9 at device 2.3 on pci0 > Mar 18 22:10:04 jerusalem kernel: ohci1: [GIANT-LOCKED] > Mar 18 22:10:04 jerusalem kernel: ohci1: [ITHREAD] > Mar 18 22:10:04 jerusalem kernel: usb1: OHCI version 1.0, legacy support > Mar 18 22:10:04 jerusalem kernel: usb1: SMM does not respond, resetting > Mar 18 22:10:04 jerusalem kernel: usb1: on ohci1 > Mar 18 22:10:04 jerusalem kernel: usb1: USB revision 1.0 > Mar 18 22:10:04 jerusalem kernel: uhub1: on usb1 > Mar 18 22:10:04 jerusalem kernel: uhub1: 3 ports with 3 removable, self powered > Mar 18 22:10:04 jerusalem kernel: pci0: at device 2.5 (no driver attached) > Mar 18 22:10:04 jerusalem kernel: pcm0: port 0xa800-0xa8ff irq 10 at device 5.0 on pci0 > Mar 18 22:10:04 jerusalem kernel: pcm0: [ITHREAD] > Mar 18 22:10:04 jerusalem kernel: ahc0: port 0xa400-0xa4ff mem 0xf1000000-0xf1000fff irq 11 at device 8.0 on pci0 > Mar 18 22:10:04 jerusalem kernel: ahc0: [ITHREAD] > Mar 18 22:10:04 jerusalem kernel: aic7890/91: Ultra2 Wide Channel A, SCSI Id=7, 32/253 SCBs > Mar 18 22:10:04 jerusalem kernel: ohci2: mem 0xf0800000-0xf0800fff irq 14 at device 10.0 on pci0 > Mar 18 22:10:04 jerusalem kernel: ohci2: [GIANT-LOCKED] > Mar 18 22:10:04 jerusalem kernel: ohci2: [ITHREAD] > Mar 18 22:10:04 jerusalem kernel: usb2: OHCI version 1.0, legacy support > Mar 18 22:10:04 jerusalem kernel: usb2: on ohci2 > Mar 18 22:10:04 jerusalem kernel: usb2: USB revision 1.0 > Mar 18 22:10:04 jerusalem kernel: uhub2: on usb2 > Mar 18 22:10:04 jerusalem kernel: uhub2: 2 ports with 2 removable, self powered > Mar 18 22:10:04 jerusalem kernel: ehci0: mem 0xf0000000-0xf00000ff irq 15 at device 10.3 on pci0 > Mar 18 22:10:04 jerusalem kernel: ehci0: [GIANT-LOCKED] > Mar 18 22:10:04 jerusalem kernel: ehci0: [ITHREAD] > Mar 18 22:10:04 jerusalem kernel: usb3: EHCI version 1.0 > Mar 18 22:10:04 jerusalem kernel: usb3: companion controller, 2 ports each: usb2 > Mar 18 22:10:04 jerusalem kernel: usb3: on ehci0 > Mar 18 22:10:04 jerusalem kernel: usb3: USB revision 2.0 > Mar 18 22:10:04 jerusalem kernel: uhub3: on usb3 > Mar 18 22:10:04 jerusalem kernel: uhub3: 6 ports with 6 removable, self powered > Mar 18 22:10:04 jerusalem kernel: umass0: on uhub3 > Mar 18 22:10:04 jerusalem kernel: em0: port 0xa000-0xa03f mem 0xef800000-0xef81ffff,0xef000000-0xef03ffff irq 11 at device 12.0 on pci0 > Mar 18 22:10:04 jerusalem kernel: em0: [FILTER] > Mar 18 22:10:04 jerusalem kernel: em0: Ethernet address: 00:0e:0c:a8:a7:e8 > Mar 18 22:10:04 jerusalem kernel: em1: port 0x9800-0x983f mem 0xee800000-0xee81ffff irq 10 at device 12.1 on pci0 > Mar 18 22:10:04 jerusalem kernel: em1: [FILTER] > Mar 18 22:10:04 jerusalem kernel: em1: Ethernet address: 00:0e:0c:a8:a7:e9 > Mar 18 22:10:04 jerusalem kernel: cpu0: on acpi0 > Mar 18 22:10:04 jerusalem kernel: acpi_throttle0: on cpu0 > Mar 18 22:10:04 jerusalem kernel: acpi_button0: on acpi0 > Mar 18 22:10:04 jerusalem kernel: fdc0: port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0 > Mar 18 22:10:04 jerusalem kernel: fdc0: [FILTER] > Mar 18 22:10:04 jerusalem kernel: fd0: <1440-KB 3.5" drive> on fdc0 drive 0 > Mar 18 22:10:04 jerusalem kernel: sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > Mar 18 22:10:04 jerusalem kernel: sio0: type 16550A > Mar 18 22:10:04 jerusalem kernel: sio0: [FILTER] > Mar 18 22:10:04 jerusalem kernel: sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 > Mar 18 22:10:04 jerusalem kernel: sio1: type 16550A > Mar 18 22:10:04 jerusalem kernel: sio1: [FILTER] > Mar 18 22:10:04 jerusalem kernel: orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xcbfff,0xcc000-0xd17ff,0xd4000-0xd4fff pnpid ORM0000 on isa0 > Mar 18 22:10:04 jerusalem kernel: ppc0: at port 0x378-0x37f irq 7 on isa0 > Mar 18 22:10:04 jerusalem kernel: ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > Mar 18 22:10:04 jerusalem kernel: ppc0: FIFO with 16/16/16 bytes threshold > Mar 18 22:10:04 jerusalem kernel: ppbus0: on ppc0 > Mar 18 22:10:04 jerusalem kernel: ppbus0: [ITHREAD] > Mar 18 22:10:04 jerusalem kernel: lpt0: on ppbus0 > Mar 18 22:10:04 jerusalem kernel: lpt0: Interrupt-driven port > Mar 18 22:10:04 jerusalem kernel: ppi0: on ppbus0 > Mar 18 22:10:04 jerusalem kernel: ppc0: [GIANT-LOCKED] > Mar 18 22:10:04 jerusalem kernel: ppc0: [ITHREAD] > Mar 18 22:10:04 jerusalem kernel: sc0: at flags 0x100 on isa0 > Mar 18 22:10:04 jerusalem kernel: sc0: VGA <16 virtual consoles, flags=0x300> > Mar 18 22:10:04 jerusalem kernel: vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > Mar 18 22:10:04 jerusalem kernel: uhub4: on uhub0 > Mar 18 22:10:04 jerusalem kernel: uhub4: 4 ports with 4 removable, bus powered > Mar 18 22:10:04 jerusalem kernel: ums0: on uhub4 > Mar 18 22:10:04 jerusalem kernel: ums0: 8 buttons and Z dir. > Mar 18 22:10:04 jerusalem kernel: ukbd0: on uhub4 > Mar 18 22:10:04 jerusalem kernel: kbd0 at ukbd0 > Mar 18 22:10:04 jerusalem kernel: ugen0: on uhub0 > Mar 18 22:10:04 jerusalem kernel: ugen1: on uhub2 > Mar 18 22:10:04 jerusalem kernel: Timecounter "TSC" frequency 2266762208 Hz quality 800 > Mar 18 22:10:04 jerusalem kernel: Timecounters tick every 1.000 msec > Mar 18 22:10:04 jerusalem kernel: ipfw2 (+ipv6) initialized, divert enabled, nat enabled, rule-based forwarding disabled, default to deny, logging limited to 100 packets/entry by default > Mar 18 22:10:04 jerusalem kernel: da0 at ahc0 bus 0 target 0 lun 0 > Mar 18 22:10:04 jerusalem kernel: da0: Fixed Direct Access SCSI-2 device > Mar 18 22:10:04 jerusalem kernel: da0: 80.000MB/s transfers (40.000MHz, offset 15, 16bit) > Mar 18 22:10:04 jerusalem kernel: da0: Command Queueing Enabled > Mar 18 22:10:04 jerusalem kernel: da0: 8683MB (17783240 512 byte sectors: 255H 63S/T 1106C) > Mar 18 22:10:04 jerusalem kernel: da1 at ahc0 bus 0 target 1 lun 0 > Mar 18 22:10:04 jerusalem kernel: da1: Fixed Direct Access SCSI-2 device > Mar 18 22:10:04 jerusalem kernel: da1: 80.000MB/s transfers (40.000MHz, offset 15, 16bit) > Mar 18 22:10:04 jerusalem kernel: da1: Command Queueing Enabled > Mar 18 22:10:04 jerusalem kernel: da1: 47702MB (97693755 512 byte sectors: 255H 63S/T 6081C) > Mar 18 22:10:04 jerusalem kernel: da2 at ahc0 bus 0 target 2 lun 0 > Mar 18 22:10:04 jerusalem kernel: da2: Fixed Direct Access SCSI-2 device > Mar 18 22:10:04 jerusalem kernel: da2: 10.000MB/s transfers (10.000MHz, offset 15) > Mar 18 22:10:04 jerusalem kernel: da2: Command Queueing Enabled > Mar 18 22:10:04 jerusalem kernel: da2: 47702MB (97693755 512 byte sectors: 255H 63S/T 6081C) > Mar 18 22:10:04 jerusalem kernel: da3 at umass-sim0 bus 0 target 0 lun 0 > Mar 18 22:10:04 jerusalem kernel: da3: Fixed Direct Access SCSI-0 device > Mar 18 22:10:04 jerusalem kernel: da3: 40.000MB/s transfers > Mar 18 22:10:04 jerusalem kernel: da3: 95611MB (195813072 512 byte sectors: 255H 63S/T 12188C) > Mar 18 22:10:04 jerusalem kernel: WARNING: WITNESS option enabled, expect reduced performance. > Mar 18 22:10:04 jerusalem kernel: cd0 at ahc0 bus 0 target 3 lun 0 > Mar 18 22:10:04 jerusalem kernel: cd0: Removable CD-ROM SCSI-2 device > Mar 18 22:10:04 jerusalem kernel: cd0: 3.300MB/s transfers > Mar 18 22:10:04 jerusalem kernel: cd0: cd present [331717 x 2048 byte records] > Mar 18 22:10:04 jerusalem kernel: lock order reversal: > Mar 18 22:10:04 jerusalem kernel: 1st 0xc2aa1e28 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2064 > Mar 18 22:10:04 jerusalem kernel: 2nd 0xc2b60194 devfsmount (devfsmount) @ /usr/src/sys/fs/devfs/devfs_vnops.c:201 > Mar 18 22:10:04 jerusalem kernel: KDB: stack backtrace: > Mar 18 22:10:04 jerusalem kernel: db_trace_self_wrapper(c070c2d9,d3ac8bbc,c058c09d,c070e5f6,c2b60194,...) at db_trace_self_wrapper+0x26 > Mar 18 22:10:04 jerusalem kernel: kdb_backtrace(c070e5f6,c2b60194,c0700b2f,c0700b2f,c0700b70,...) at kdb_backtrace+0x29 > Mar 18 22:10:04 jerusalem kernel: witness_checkorder(c2b60194,9,c0700b70,c9,c7,...) at witness_checkorder+0x6af > Mar 18 22:10:04 jerusalem kernel: _sx_xlock(c2b60194,0,c0700b70,c9,c2b60194,...) at _sx_xlock+0x77 > Mar 18 22:10:04 jerusalem kernel: devfs_allocv(c2b62180,c2b64000,d3ac8c28,c28ea000,c0714353,...) at devfs_allocv+0x13e > Mar 18 22:10:04 jerusalem kernel: devfs_root(c2b64000,2,c07d7ad8,c28ea000,ca,...) at devfs_root+0x51 > Mar 18 22:10:04 jerusalem kernel: set_rootvnode(c07d7ac0,0,c0714353,5f4,c05c52da,...) at set_rootvnode+0x2b > Mar 18 22:10:04 jerusalem kernel: vfs_mountroot(c078c830,4,c07047a9,264,c058611f,...) at vfs_mountroot+0x334 > Mar 18 22:10:04 jerusalem kernel: start_init(0,d3ac8d38,c0706143,30c,c28e7000,...) at start_init+0x65 > Mar 18 22:10:04 jerusalem kernel: fork_exit(c0526dcd,0,d3ac8d38) at fork_exit+0xb8 > Mar 18 22:10:04 jerusalem kernel: fork_trampoline() at fork_trampoline+0x8 > Mar 18 22:10:04 jerusalem kernel: --- trap 0, eip = 0, esp = 0xd3ac8d70, ebp = 0 --- > Mar 18 22:10:04 jerusalem kernel: Trying to mount root from ufs:/dev/da0s1a > Mar 18 22:10:04 jerusalem kernel: lock order reversal: > Mar 18 22:10:04 jerusalem kernel: 1st 0xc2aa19e8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2064 > Mar 18 22:10:04 jerusalem kernel: 2nd 0xc2b64000 vfslock (vfslock) @ /usr/src/sys/kern/vfs_subr.c:364 > Mar 18 22:10:04 jerusalem kernel: KDB: stack backtrace: > Mar 18 22:10:04 jerusalem kernel: db_trace_self_wrapper(c070c2d9,d3ac89dc,c058c09d,c070e5f6,c2b64000,...) at db_trace_self_wrapper+0x26 > Mar 18 22:10:04 jerusalem kernel: kdb_backtrace(c070e5f6,c2b64000,c0714456,c0714456,c07149ee,...) at kdb_backtrace+0x29 > Mar 18 22:10:04 jerusalem kernel: witness_checkorder(c2b64000,1,c07149ee,16c,d3ac8a1c,...) at witness_checkorder+0x6af > Mar 18 22:10:04 jerusalem kernel: _lockmgr_args(c2b64000,20001,c2b64030,0,ffffffff,...) at _lockmgr_args+0x1c9 > Mar 18 22:10:04 jerusalem kernel: vfs_busy(c2b64000,0,0,c28ea000,d3ac8b58,...) at vfs_busy+0x19f > Mar 18 22:10:04 jerusalem kernel: lookup(d3ac8b44,c071411b,c6,bf,c28cdb2c,...) at lookup+0x735 > Mar 18 22:10:04 jerusalem kernel: namei(d3ac8b44,c2b64030,1c1,c0714353,d3ac8b54,...) at namei+0x25e > Mar 18 22:10:04 jerusalem kernel: kern_unlink(c28ea000,c0714790,1,62f,0,...) at kern_unlink+0x39 > Mar 18 22:10:04 jerusalem kernel: vfs_mountroot_try(c071494a,c070dfd0,c06fecf9,1,c05c52da,...) at vfs_mountroot_try+0x466 > Mar 18 22:10:04 jerusalem kernel: vfs_mountroot(c078c830,4,c07047a9,264,c058611f,...) at vfs_mountroot+0x3f2 > Mar 18 22:10:04 jerusalem kernel: start_init(0,d3ac8d38,c0706143,30c,c28e7000,...) at start_init+0x65 > Mar 18 22:10:04 jerusalem kernel: fork_exit(c0526dcd,0,d3ac8d38) at fork_exit+0xb8 > Mar 18 22:10:04 jerusalem kernel: fork_trampoline() at fork_trampoline+0x8 > Mar 18 22:10:04 jerusalem kernel: --- trap 0, eip = 0, esp = 0xd3ac8d70, ebp = 0 --- > Mar 18 22:10:04 jerusalem kernel: lock order reversal: > Mar 18 22:10:04 jerusalem kernel: 1st 0xc28ec044 user map (user map) @ /usr/src/sys/vm/vm_map.c:3111 > Mar 18 22:10:04 jerusalem kernel: 2nd 0xc2aa17c8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2064 > Mar 18 22:10:04 jerusalem kernel: KDB: stack backtrace: > Mar 18 22:10:04 jerusalem kernel: db_trace_self_wrapper(c070c2d9,d3ac89c0,c058c09d,c070e5f6,c2aa17c8,...) at db_trace_self_wrapper+0x26 > Mar 18 22:10:04 jerusalem kernel: kdb_backtrace(c070e5f6,c2aa17c8,c0703ec1,c0703ec1,c07149ee,...) at kdb_backtrace+0x29 > Mar 18 22:10:04 jerusalem kernel: witness_checkorder(c2aa17c8,1,c07149ee,810,d3ac89e4,...) at witness_checkorder+0x6af > Mar 18 22:10:04 jerusalem kernel: _lockmgr_args(c2aa17c8,30041,c2aa17f8,0,ffffffff,...) at _lockmgr_args+0x1c9 > Mar 18 22:10:04 jerusalem kernel: ffs_lock(d3ac8a74,c054b23c,c0791354,30041,c2aa1770,...) at ffs_lock+0xa1 > Mar 18 22:10:04 jerusalem kernel: VOP_LOCK1_APV(c0767d60,d3ac8a74,c070dfce,3,c2aa17f8,...) at VOP_LOCK1_APV+0xab > Mar 18 22:10:04 jerusalem kernel: _vn_lock(c2aa1770,30041,c07149ee,810,0,...) at _vn_lock+0xed > Mar 18 22:10:04 jerusalem kernel: vget(c2aa1770,30041,c28ea000,4a9,c1046780,...) at vget+0x101 > Mar 18 22:10:04 jerusalem kernel: vnode_pager_lock(c1046600,0,c0722bee,127,d3ac8be8,...) at vnode_pager_lock+0x1a5 > Mar 18 22:10:04 jerusalem kernel: vm_fault(c28ec000,80b4000,2,8,80b4000,...) at vm_fault+0x1dc > Mar 18 22:10:04 jerusalem kernel: trap_pfault(5,0,c072b9a2,2c4,c,...) at trap_pfault+0xf9 > Mar 18 22:10:04 jerusalem kernel: trap(d3ac8d38) at trap+0x230 > Mar 18 22:10:04 jerusalem kernel: calltrap() at calltrap+0x6 > Mar 18 22:10:04 jerusalem kernel: --- trap 0xc, eip = 0x80480e5, esp = 0xbfbfef10, ebp = 0xbfbfef30 --- > Mar 18 22:10:05 jerusalem savecore: no dumps found > Mar 18 22:10:06 jerusalem kernel: lock order reversal: > Mar 18 22:10:06 jerusalem kernel: 1st 0xc2bfdaf8 isofs (isofs) @ /usr/src/sys/kern/vfs_subr.c:2064 > Mar 18 22:10:06 jerusalem kernel: 2nd 0xc2b63538 vfslock (vfslock) @ /usr/src/sys/kern/vfs_subr.c:364 > Mar 18 22:10:06 jerusalem kernel: KDB: stack backtrace: > Mar 18 22:10:06 jerusalem kernel: db_trace_self_wrapper(c070c2d9,d5f5ba18,c058c09d,c070e5f6,c2b63538,...) at db_trace_self_wrapper+0x26 > Mar 18 22:10:06 jerusalem kernel: kdb_backtrace(c070e5f6,c2b63538,c0714456,c0714456,c07149ee,...) at kdb_backtrace+0x29 > Mar 18 22:10:06 jerusalem kernel: witness_checkorder(c2b63538,1,c07149ee,16c,d5f5ba58,...) at witness_checkorder+0x6af > Mar 18 22:10:06 jerusalem kernel: _lockmgr_args(c2b63538,20001,c2b63568,0,ffffffff,...) at _lockmgr_args+0x1c9 > Mar 18 22:10:06 jerusalem kernel: vfs_busy(c2b63538,10,0,c2b9b220,0,...) at vfs_busy+0x19f > Mar 18 22:10:06 jerusalem kernel: vfs_donmount(810e080,c,d5f5bc74,c2dc3600,0,...) at vfs_donmount+0xdc8 > Mar 18 22:10:06 jerusalem kernel: nmount(c2b9b220,d5f5bcfc,c,c070f2c4,c0751e10,...) at nmount+0x8e > Mar 18 22:10:06 jerusalem kernel: syscall(d5f5bd38) at syscall+0x237 > Mar 18 22:10:06 jerusalem kernel: Xint0x80_syscall() at Xint0x80_syscall+0x20 > Mar 18 22:10:06 jerusalem kernel: --- syscall (378, FreeBSD ELF32, nmount), eip = 0x480dafcf, esp = 0xbfbfe97c, ebp = 0xbfbfedd8 --- > Mar 18 22:10:06 jerusalem kernel: lock order reversal: > Mar 18 22:10:06 jerusalem kernel: 1st 0xc2bfd278 pseudofs (pseudofs) @ /usr/src/sys/kern/vfs_subr.c:2064 > Mar 18 22:10:06 jerusalem kernel: 2nd 0xc2b6329c vfslock (vfslock) @ /usr/src/sys/kern/vfs_subr.c:364 > Mar 18 22:10:06 jerusalem kernel: KDB: stack backtrace: > Mar 18 22:10:06 jerusalem kernel: db_trace_self_wrapper(c070c2d9,d5f5ba18,c058c09d,c070e5f6,c2b6329c,...) at db_trace_self_wrapper+0x26 > Mar 18 22:10:06 jerusalem kernel: kdb_backtrace(c070e5f6,c2b6329c,c0714456,c0714456,c07149ee,...) at kdb_backtrace+0x29 > Mar 18 22:10:06 jerusalem kernel: witness_checkorder(c2b6329c,1,c07149ee,16c,d5f5ba58,...) at witness_checkorder+0x6af > Mar 18 22:10:06 jerusalem kernel: _lockmgr_args(c2b6329c,20001,c2b632cc,0,ffffffff,...) at _lockmgr_args+0x1c9 > Mar 18 22:10:06 jerusalem kernel: vfs_busy(c2b6329c,10,0,c2b9b220,0,...) at vfs_busy+0x19f > Mar 18 22:10:06 jerusalem kernel: vfs_donmount(810e080,c,d5f5bc74,c2dc3680,0,...) at vfs_donmount+0xdc8 > Mar 18 22:10:06 jerusalem kernel: nmount(c2b9b220,d5f5bcfc,c,c070f2c4,c0751e10,...) at nmount+0x8e > Mar 18 22:10:06 jerusalem kernel: syscall(d5f5bd38) at syscall+0x237 > Mar 18 22:10:06 jerusalem kernel: Xint0x80_syscall() at Xint0x80_syscall+0x20 > Mar 18 22:10:06 jerusalem kernel: --- syscall (378, FreeBSD ELF32, nmount), eip = 0x480dafcf, esp = 0xbfbfe97c, ebp = 0xbfbfedd8 --- >