From owner-freebsd-bugs@FreeBSD.ORG Thu Jul 23 15:20:01 2009 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4CAB1106566C for ; Thu, 23 Jul 2009 15:20:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 25EF68FC1A for ; Thu, 23 Jul 2009 15:20:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n6NFK1GW024048 for ; Thu, 23 Jul 2009 15:20:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n6NFK0Bp024047; Thu, 23 Jul 2009 15:20:00 GMT (envelope-from gnats) Resent-Date: Thu, 23 Jul 2009 15:20:00 GMT Resent-Message-Id: <200907231520.n6NFK0Bp024047@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Royce William Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E7EC1065677 for ; Thu, 23 Jul 2009 15:13:18 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 4AAA88FC08 for ; Thu, 23 Jul 2009 15:13:18 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n6NFDIVB096281 for ; Thu, 23 Jul 2009 15:13:18 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id n6NFDI1Y096280; Thu, 23 Jul 2009 15:13:18 GMT (envelope-from nobody) Message-Id: <200907231513.n6NFDI1Y096280@www.freebsd.org> Date: Thu, 23 Jul 2009 15:13:18 GMT From: Royce William To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/137034: lock-order reversal near vfs_bio.c, ufs_dirhash.c X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Jul 2009 15:20:01 -0000 >Number: 137034 >Category: kern >Synopsis: lock-order reversal near vfs_bio.c, ufs_dirhash.c >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Thu Jul 23 15:20:00 UTC 2009 >Closed-Date: >Last-Modified: >Originator: Royce William >Release: 8.0-BETA2 >Organization: >Environment: FreeBSD max.prv.tycho.org 8.0-BETA2 FreeBSD 8.0-BETA2 #0: Wed Jul 15 23:25:30 UTC 2009 root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386 >Description: Unloaded system upgrading from 7.2-SECURITY to 8.0-BETA2 via freebsd-update. Halfway through the second install phase, in which I was running the 8.0-BETA2 kernel but a 7.2-SECURITY userland, I got this LOR: lock order reversal: 1st 0xc557f310 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2558 2nd 0xc5d82600 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper(c0c6baf4,f4b9175c,c08bc995,c08ad6db,c0c6e989,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c08ad6db,c0c6e989,c592bef8,c592f500,f4b917b8,...) at kdb_backtrace+0x29 _witness_debugger(c0c6e989,c5d82600,c0c8f767,c592f500,c0c8f400,...) at _witness_debugger+0x25 witness_checkorder(c5d82600,9,c0c8f400,11d,0,...) at witness_checkorder+0x839 _sx_xlock(c5d82600,0,c0c8f400,11d,c5ff3d24,...) at _sx_xlock+0x85 ufsdirhash_acquire(c557f2b0,f4b918d0,16c,e581c8ac,f4b91888,...) at ufsdirhash_acquire+0x35 ufsdirhash_add(c5ff3d24,f4b918d0,8ac,f4b91874,f4b91878,...) at ufsdirhash_add+0x13 ufs_direnter(c5b96648,cceff324,f4b918d0,f4b91bd0,0,...) at ufs_direnter+0x729 ufs_makeinode(f4b91bd0,0,f4b91abc,f4b91a18,c0bb6745,...) at ufs_makeinode+0x508 ufs_create(f4b91abc,f4b91ad4,0,0,f4b91ba4,...) at ufs_create+0x30 VOP_CREATE_APV(c0d6e7e0,f4b91abc,f4b91bd0,f4b91a54,0,...) at VOP_CREATE_APV+0xa5 vn_open_cred(f4b91ba4,f4b91c5c,1a0,0,c8596900,...) at vn_open_cred+0x215 vn_open(f4b91ba4,f4b91c5c,1a0,c78877e0,0,...) at vn_open+0x3b kern_openat(d872e480,ffffff9c,28303514,0,602,...) at kern_openat+0x11f kern_open(d872e480,28303514,0,601,1b6,...) at kern_open+0x35 open(d872e480,f4b91cf8,c,d872e480,c0d4d44c,...) at open+0x30 syscall(f4b91d38) at syscall+0x2a3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (5, FreeBSD ELF32, open), eip = 0x281d8833, esp = 0xbfbfe16c, ebp = 0xbfbfe208 --- The system is using ZFS for some mounts (may not be relevant): royce@max$ mount /dev/ad4s1a on / (ufs, local) devfs on /dev (devfs, local, multilabel) tank/home on /home (zfs, local, nosuid) tank on /tank (zfs, local) tank/tmp on /tmp (zfs, local, noatime, nosuid) tank/usr on /usr (zfs, local, noatime) tank/var on /var (zfs, local, noatime, nosuid) royce@max$ zpool upgrade This system is currently running ZFS pool version 13. The following pools are out of date, and can be upgraded. After being upgraded, these pools will no longer be accessible by older software versions. VER POOL --- ------------ 6 tank >How-To-Repeat: Probably I/O related, but in brief: #### Install FreeBSD 7.2. # freebsd-update fetch install # shutdown -r now # ## System is now running 7.2-SECURITY as of 2009-07-21 #### Upgrade to 8.0-BETA2 kernel. # freebsd-update -r 8.0-BETA2 upgrade # freebsd-update install # shutdown -r now #### Perform second phase of install. # freebsd-update install #### LOR occurred here. >Fix: Patch attached with submission follows: # # royce@tycho.org 2009-07-22 # Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-BETA2 #0: Wed Jul 15 23:25:30 UTC 2009 root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (3000.12-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf43 Stepping = 3 Features=0xbfebfbff Features2=0x649d AMD Features=0x20100000 TSC: P-state invariant real memory = 2147483648 (2048 MB) avail memory = 2077204480 (1980 MB) ACPI APIC Table: This module (opensolaris) contains code covered by the Common Development and Distribution License (CDDL) see http://opensolaris.org/os/licensing/opensolaris_license/ ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, 7f5e0000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 pcib2: at device 0.0 on pci1 pci2: on pcib2 ath0: mem 0xd0000000-0xd000ffff irq 24 at device 1.0 on pci2 ath0: [ITHREAD] ath0: AR2413 mac 7.9 RF2413 phy 4.5 vgapci0: port 0xe000-0xe007 mem 0xd0400000-0xd047ffff,0xc0000000-0xcfffffff,0xd0480000-0xd04bffff irq 16 at device 2.0 on pci0 agp0: on vgapci0 agp0: detected 7932k stolen memory agp0: aperture size is 256M pcib3: irq 16 at device 28.0 on pci0 pci3: on pcib3 pcib4: irq 18 at device 28.2 on pci0 pci4: on pcib4 bge0: mem 0xd0200000-0xd020ffff irq 18 at device 0.0 on pci4 miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge0: Ethernet address: 00:30:48:85:d3:bc bge0: [ITHREAD] pcib5: irq 19 at device 28.3 on pci0 pci5: on pcib5 bge1: mem 0xd0300000-0xd030ffff irq 19 at device 0.0 on pci5 miibus1: on bge1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge1: Ethernet address: 00:30:48:85:d3:bd bge1: [ITHREAD] uhci0: port 0xe100-0xe11f irq 23 at device 29.0 on pci0 uhci0: [ITHREAD] uhci0: LegSup = 0x2f00 usbus0: on uhci0 uhci1: port 0xe200-0xe21f irq 19 at device 29.1 on pci0 uhci1: [ITHREAD] uhci1: LegSup = 0x2f00 usbus1: on uhci1 uhci2: port 0xe300-0xe31f irq 18 at device 29.2 on pci0 uhci2: [ITHREAD] uhci2: LegSup = 0x2f00 usbus2: on uhci2 uhci3: port 0xe400-0xe41f irq 16 at device 29.3 on pci0 uhci3: [ITHREAD] uhci3: LegSup = 0x2f00 usbus3: on uhci3 ehci0: mem 0xd04c0000-0xd04c03ff irq 23 at device 29.7 on pci0 ehci0: [ITHREAD] usbus4: EHCI version 1.0 usbus4: on ehci0 pcib6: at device 30.0 on pci0 pci6: on pcib6 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 31.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] atapci1: port 0xe900-0xe907,0xea00-0xea03,0xeb00-0xeb07,0xec00-0xec03,0xed00-0xed0f mem 0xd04c3000-0xd04c33ff irq 19 at device 31.2 on pci0 atapci1: [ITHREAD] ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] pci0: at device 31.3 (no driver attached) atrtc0: port 0x70-0x73 irq 8 on acpi0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] cpu0: on acpi0 acpi_perf0: on cpu0 p4tcc0: on cpu0 pmtimer0 on isa0 orm0: at iomem 0xcc000-0xcffff pnpid ORM0000 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: parallel port not found. WARNING: ZFS is considered to be an experimental feature in FreeBSD. Timecounter "TSC" frequency 3000120615 Hz quality 800 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 480Mbps High Speed USB v2.0 ZFS NOTICE: prefetch is disabled by default on i386 - add enable to tunable to change. ZFS filesystem version 13 ZFS storage pool version 13 ad4: 239372MB at ata2-master SATA150 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ad6: 239372MB at ata3-master SATA150 ar0: 239372MB status: READY ar0: disk0 READY (master) using ad4 at ata2-master ar0: disk1 READY (mirror) using ad6 at ata3-master WARNING: WITNESS option enabled, expect reduced performance. GEOM: ar0s1: geometry does not match label (16h,63s != 255h,63s). Root mount waiting for: usbus4 usbus3 usbus2 usbus1 usbus0 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered Root mount waiting for: usbus4 uhub4: 8 ports with 8 removable, self powered Trying to mount root from ufs:/dev/ad4s1a bge0: link state changed to UP >Release-Note: >Audit-Trail: >Unformatted: