From owner-freebsd-current@FreeBSD.ORG Fri Mar 5 01:47:11 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 30A97106564A for ; Fri, 5 Mar 2010 01:47:11 +0000 (UTC) (envelope-from nikolai-freebsd@fetissov.org) Received: from starfish.geekisp.com (mail.geekisp.com [216.168.135.169]) by mx1.freebsd.org (Postfix) with ESMTP id B3F698FC16 for ; Fri, 5 Mar 2010 01:47:10 +0000 (UTC) Received: (qmail 13671 invoked by uid 1003); 5 Mar 2010 01:19:07 -0000 Received: from 99.35.57.141 (proxying for 99.35.57.141) (SquirrelMail authenticated user nikolai@fetissov.org) by www.geekisp.com with HTTP; Thu, 4 Mar 2010 20:19:06 -0500 Message-ID: <18b0d217f1153d417b4ee5ed22cc1496.squirrel@www.geekisp.com> Date: Thu, 4 Mar 2010 20:19:06 -0500 From: "Nikolai Fetissov" To: freebsd-current@freebsd.org User-Agent: SquirrelMail/1.4.20 [SVN] MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Subject: Questions about witness reports on sparc64 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2010 01:47:11 -0000 Folks, I'm pretty new here, so feel free to use the clue stick :) Playing with current- on Sun Fire V210 I see more or less consistent lock reversal reports like the following: lock order reversal: 1st 0xc1c85fb8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559 2nd 0xfffff80002784600 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: _witness_debugger() at _witness_debugger+0x38 witness_checkorder() at witness_checkorder+0xcf8 _sx_xlock() at _sx_xlock+0x9c ufsdirhash_acquire() at ufsdirhash_acquire+0x30 ufsdirhash_add() at ufsdirhash_add+0x4 ufs_direnter() at ufs_direnter+0x75c ufs_makeinode() at ufs_makeinode+0x4dc ufs_create() at ufs_create+0x40 VOP_CREATE_APV() at VOP_CREATE_APV+0x108 vn_open_cred() at vn_open_cred+0x1fc vn_open() at vn_open+0x1c kern_openat() at kern_openat+0x10c kern_open() at kern_open+0x18 open() at open+0x14 syscall() at syscall+0x25c -- syscall (5, FreeBSD ELF64, open) %o7=0x412bfda8 -- userland() at 0x41344808 user trace: trap %o7=0x412bfda8 pc 0x41344808, sp 0x7fdffff99d1 pc 0x120008, sp 0x7fdffff9aa1 pc 0x11216c, sp 0x7fdffff9b61 pc 0x10d32c, sp 0x7fdffffdd11 pc 0x10d9e4, sp 0x7fdffffde01 pc 0x10cf2c, sp 0x7fdffffded1 pc 0x10eea4, sp 0x7fdffffdfa1 pc 0x15a0a0, sp 0x7fdffffe061 pc 0x13024c, sp 0x7fdffffe191 pc 0x103c10, sp 0x7fdffffe281 pc 0x4028ef34, sp 0x7fdffffe341 done That's just on startup, nothing fancy going on. My questions are - is it worth spamming the list with these? If so, what other info needed? Full dmesg bellow. Thanks, -- Nikolai GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-CURRENT #0: Sun Feb 28 23:19:48 EST 2010 root@moon.ipv6.fetissov.org:/usr/obj/usr/src/sys/GENERIC sparc64 WARNING: WITNESS option enabled, expect reduced performance. real memory = 4294967296 (4096 MB) avail memory = 4174233600 (3980 MB) cpu0: Sun Microsystems UltraSparc-IIIi Processor (1002.00 MHz CPU) cpu1: Sun Microsystems UltraSparc-IIIi Processor (1002.00 MHz CPU) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs ispfw: registered firmware ispfw: registered firmware ispfw: registered firmware ispfw: registered firmware ispfw: registered firmware ispfw: registered firmware ispfw: registered firmware ispfw: registered firmware ispfw: registered firmware ispfw: registered firmware ispfw: registered firmware ispfw: registered firmware ispfw: registered firmware ispfw: registered firmware ispfw: registered firmware kbd0 at kbdmux0 nexus0: nexus0: mem 0x40000000000-0x40000000007 type memory-controller (no driver attached) nexus0: mem 0x40000800000-0x40000800007 type memory-controller (no driver attached) pcib0: mem 0x4000ff00000-0x4000ff0afff,0x4000fc10000-0x4000fc1701f,0x7f600000000-0x7f6000000ff,0x4000ff80000-0x4000ff8ffff irq 2035,2032,2033,2036,2019 on nexus0 pcib0: Tomatillo, version 4, IGN 0x1f, bus B, 66MHz pcib0: DVMA map: 0xc0000000 to 0xdfffffff 65536 entries pcib0: [FILTER] pcib0: [FILTER] pcib0: [FILTER] pcib0: [FILTER] pci0: on pcib0 bge0: mem 0x200000-0x20ffff,0x110000-0x11ffff at device 2.0 on pci0 miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge0: Ethernet address: 00:03:ba:45:87:2d bge0: [ITHREAD] bge1: mem 0x400000-0x40ffff,0x120000-0x12ffff at device 2.1 on pci0 miibus1: on bge1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge1: Ethernet address: 00:03:ba:45:87:2e bge1: [ITHREAD] pcib1: mem 0x4000f600000-0x4000f60afff,0x4000f410000-0x4000f41701f,0x7fe00000000-0x7fe000000ff,0x4000f780000-0x4000f78ffff irq 1970,1968,1969,1972,1953 on nexus0 pcib1: Tomatillo, version 4, IGN 0x1e, bus A, 33MHz pcib1: DVMA map: 0xc0000000 to 0xdfffffff 65536 entries pcib1: [FILTER] pci1: on pcib1 isab0: at device 7.0 on pci1 isa0: on isab0 pci1: at device 6.0 (no driver attached) ohci0: mem 0x1000000-0x1000fff at device 10.0 on pci1 ohci0: [ITHREAD] usbus0: on ohci0 atapci0: port 0x900-0x907,0x918-0x91b,0x910-0x917,0x908-0x90b,0x920-0x92f at device 13.0 on pci1 atapci0: [ITHREAD] atapci0: using PIO transfers above 137GB as workaround for 48bit DMA access bug, expect reduced performance ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] pcib2: mem 0x4000e600000-0x4000e60afff,0x4000e410000-0x4000e41701f,0x7ce00000000-0x7ce000000ff,0x4000e780000-0x4000e78ffff irq 1842,1840,1841,1844,1827 on nexus0 pcib2: Tomatillo, version 4, IGN 0x1c, bus A, 66MHz pcib2: DVMA map: 0xc0000000 to 0xdfffffff 65536 entries pcib2: [FILTER] pcib2: [FILTER] pcib2: [FILTER] pcib2: [FILTER] pci2: on pcib2 sym0: <1010-66> port 0x900-0x9ff mem 0x100000-0x1003ff,0x102000-0x103fff at device 2.0 on pci2 sym0: No NVRAM, ID 7, Fast-80, LVD, parity checking sym0: [ITHREAD] sym1: <1010-66> port 0xa00-0xaff mem 0x104000-0x1043ff,0x106000-0x107fff at device 2.1 on pci2 sym1: No NVRAM, ID 7, Fast-80, LVD, parity checking sym1: [ITHREAD] pcib3: mem 0x4000ef00000-0x4000ef0afff,0x4000ec10000-0x4000ec1701f,0x7c600000000-0x7c6000000ff,0x4000ef80000-0x4000ef8ffff irq 1907,1904,1905,1908,1893 on nexus0 pcib3: Tomatillo, version 4, IGN 0x1d, bus B, 66MHz pcib3: DVMA map: 0xc0000000 to 0xdfffffff 65536 entries pcib3: [FILTER] pci3: on pcib3 bge2: mem 0x200000-0x20ffff,0x110000-0x11ffff at device 2.0 on pci3 miibus2: on bge2 brgphy2: PHY 1 on miibus2 brgphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge2: Ethernet address: 00:03:ba:45:87:2f bge2: [ITHREAD] bge3: mem 0x400000-0x40ffff,0x120000-0x12ffff at device 2.1 on pci3 miibus3: on bge3 brgphy3: PHY 1 on miibus3 brgphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge3: Ethernet address: 00:03:ba:45:87:30 bge3: [ITHREAD] nexus0: type unknown (no driver attached) rtc0: at port 0x70-0x71 on isa0 uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 44 on isa0 uart0: [FILTER] uart0: console (9600,n,8,1) uart1: <16550 or compatible> at port 0x2e8-0x2ef irq 44 on isa0 uart1: [FILTER] Timecounter "tick" frequency 1002000000 Hz quality 10 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 acd0: DVDROM at ata2-master UDMA33 uhub0: 2 ports with 2 removable, self powered SMP: AP CPU #1 Launched! da1 at sym0 bus 0 scbus0 target 1 lun 0WARNING: WITNESS option enabled, expect reduced performance. da1: Fixed Direct Access SCSI-3 device da1: 80.000MB/s transfers (40.000MHz, offset 31, 16bit) da1: Command Queueing enabled da1: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C) da0 at sym0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-3 device da0: 160.000MB/s transfers (80.000MHz DT, offset 62, 16bit) da0: Command Queueing enabled da0: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C) GEOM: da0: adding VTOC8 information. GEOM: da1: adding VTOC8 information. Trying to mount root from ufs:/dev/da0a lock order reversal: 1st 0xc1c85fb8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559 2nd 0xfffff80002784600 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: _witness_debugger() at _witness_debugger+0x38 witness_checkorder() at witness_checkorder+0xcf8 _sx_xlock() at _sx_xlock+0x9c ufsdirhash_acquire() at ufsdirhash_acquire+0x30 ufsdirhash_add() at ufsdirhash_add+0x4 ufs_direnter() at ufs_direnter+0x75c ufs_makeinode() at ufs_makeinode+0x4dc ufs_create() at ufs_create+0x40 VOP_CREATE_APV() at VOP_CREATE_APV+0x108 vn_open_cred() at vn_open_cred+0x1fc vn_open() at vn_open+0x1c kern_openat() at kern_openat+0x10c kern_open() at kern_open+0x18 open() at open+0x14 syscall() at syscall+0x25c -- syscall (5, FreeBSD ELF64, open) %o7=0x412bfda8 -- userland() at 0x41344808 user trace: trap %o7=0x412bfda8 pc 0x41344808, sp 0x7fdffff99d1 pc 0x120008, sp 0x7fdffff9aa1 pc 0x11216c, sp 0x7fdffff9b61 pc 0x10d32c, sp 0x7fdffffdd11 pc 0x10d9e4, sp 0x7fdffffde01 pc 0x10cf2c, sp 0x7fdffffded1 pc 0x10eea4, sp 0x7fdffffdfa1 pc 0x15a0a0, sp 0x7fdffffe061 pc 0x13024c, sp 0x7fdffffe191 pc 0x103c10, sp 0x7fdffffe281 pc 0x4028ef34, sp 0x7fdffffe341 done lock order reversal: 1st 0xfffff8000dbb49f8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2091 2nd 0xc16573f8 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_softdep.c:6193 3rd 0xfffff8000d679718 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2091 KDB: stack backtrace: _witness_debugger() at _witness_debugger+0x38 witness_checkorder() at witness_checkorder+0xcf8 __lockmgr_args() at __lockmgr_args+0x898 ffs_lock() at ffs_lock+0x88 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x110 _vn_lock() at _vn_lock+0x80 vget() at vget+0x104 vfs_hash_get() at vfs_hash_get+0xdc ffs_vgetf() at ffs_vgetf+0x20 softdep_sync_metadata() at softdep_sync_metadata+0x6a0 ffs_syncvnode() at ffs_syncvnode+0x424 ffs_truncate() at ffs_truncate+0x454 ufs_direnter() at ufs_direnter+0x920 ufs_mkdir() at ufs_mkdir+0x824 VOP_MKDIR_APV() at VOP_MKDIR_APV+0x108 kern_mkdirat() at kern_mkdirat+0x26c kern_mkdir() at kern_mkdir+0x14 mkdir() at mkdir+0x10 syscall() at syscall+0x25c -- syscall (136, FreeBSD ELF64, mkdir) %o7=0x100ce0 -- userland() at 0x40473188 user trace: trap %o7=0x100ce0 pc 0x40473188, sp 0x7fdffffdd71 pc 0x100950, sp 0x7fdffffdeb1 pc 0x40206f34, sp 0x7fdffffdf71 done