Date: Tue, 9 Mar 2010 15:26:25 +0100 From: =?ISO-8859-1?Q?Johan_Str=F6m?= <johan@stromnet.se> To: freebsd-fs@freebsd.org Subject: ZFS: zpool import hang on "zio->io_cv)" (with DDB output). Help needed! Message-ID: <8BA8475A-424E-489B-B643-8757F375320B@stromnet.se>
next in thread | raw e-mail | index | archive | help
Hi List!
I'm in the tedious process of upgrading my pool on my FreeBSD 8.0 box
(with a pool from 7.x). Yesterday I pulled a few disks from my
mirrored pool (one pool with multiple mirrors), in order to be able to
(free ports) plug in new ones and build my brand new pool.
After removing disks, the pool got into state DEGRADED ofcourse, since
some of the disks where gone, but data was still there. I did zpool
detach on the removed disks, and the pool was ONLINE again, all fine!
Then I rebooted to make another disk available through my old LSI
MegaRaid card (didnt take time to figure out cryptic syntax of the
megarc CLI util). On boot, I was met by something similar to this:
pool: tank
state: UNAVAIL
status: One or more devices could not be used because the label is
missing
or invalid. There are insufficient replicas for the pool to
continue
functioning.
action: Destroy and re-create the pool from a backup source.
see: http://www.sun.com/msg/ZFS-8000-5E
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank UNAVAIL 0 0 0 insufficient replicas
ad10s1d ONLINE 0 0 0
mirror DEGRADED 0 0 0
ad12 FAULTED 0 0 0 corrupted data
ad16 UNAVAIL 0 0 0 corrupted data
mirror DEGRADED 0 0 0
ad20 FAULTED 0 0 0 corrupted data
ad18 UNAVAIL 0 0 0 corrupted data
mirror UNAVAIL 0 0 0 insufficient replicas
ad6 UNAVAIL 0 0 0 corrupted data
ad4 FAULTED 0 0 0 corrupted data
mirror ONLINE 0 0 0
amrd0 ONLINE 0 0 0
amrd1 ONLINE 0 0 0
The reason the disks was still there was due to me haveing an old
zfs.cache file in the boot (it boots from anothre drive, but thats
another story..)
In this case, a export/import did the trick, after reimporting, the
pool was back online. All fine. I created a new pool and did some
bonnie++ testing on it, and suddenly the box paniced or something (I
didn't have dumpon enabled :/ and didn't see the screen until it
rebooted)..
Now my problems come. The box came up again, with the above output. I
tried zpool export again, fine. Zpool import however, hang. Waited an
hour, nothing. After rebuilding the kernel with DDB/witness and did
the import again, i've managed to get this output (im not really sure
what is usable here):
back-1 # zpool import
load: 0.29 cmd: zpool 3193 [zio->io_cv)] 2.54r 0.00u 0.01s 0% 2236k
In DDB:
> tr 3193
Tracing pid 3193 tid 100122 td 0xffffff00035ca390
sched_switch() at sched_switch+0xde
mi_switch() at mi_switch+0x170
sleepq_wait() at sleepq_wait+0x44
_cv_wait() at _cv_wait+0x13c
zio_wait() at zio_wait+0x61
arc_read_nolock() at arc_read_nolock+0x345
dmu_objset_open_impl() at dmu_objset_open_impl+0xd0
dsl_pool_open() at dsl_pool_open+0x5a
spa_load() at spa_load+0x31b
spa_tryimport() at spa_tryimport+0xa9
zfs_ioc_pool_tryimport() at zfs_ioc_pool_tryimport+0x3f
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xf6
ioctl() at ioctl+0xfd
syscall() at syscall+0x19e
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x8010eb86c, rsp =
0x7fffffff8e28, rbp = 0x801323300 ---
db> show thread 100122
Thread 100122 at 0xffffff00035ca390:
proc (pid 3193): 0xffffff000383c460
name: zpool
stack: 0xffffff805740a000-0xffffff805740dfff
flags: 0x44 pflags: 0x10000
state: INHIBITED: {SLEEPING}
wmesg: zio->io_cv) wchan: 0xffffff00506e5858
priority: 131
container lock: sleepq chain (0xffffffff80c61e68)
db>
db> show sleepchain 10012
db> show sleepchain 3193
thread 100122 (pid 3193, zpool) sleeping on 0xffffff00506e5858 "zio-
>io_cv)"
db> show lock 0xffffff00506e5858
class: spin mutex
name: zio->io_cv)
flags: {SPIN}
state: {OWNED}
The box is currenlty in this state, so if you reply to me now I can
continue to do debugging according to instructions.
Since I cannot get anywhere on my own with this, all and any help is
appreciated since I really need this pool to get online..
dmesg is posted below (including some LOR?)
Thanks!
Johan
DMESG:
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights
reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-RELEASE-p2 #10: Tue Mar 9 12:44:15 CET 2010
johan@back-1.stromnet.se:/usr/obj/usr/src/sys/BACK1
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz (2666.68-MHz K8-
class CPU)
Origin = "GenuineIntel" Id = 0x6fb Stepping = 11
Features
=
0xbfebfbff
<
FPU
,VME
,DE
,PSE
,TSC
,MSR
,PAE
,MCE
,CX8
,APIC
,SEP
,MTRR
,PGE
,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2
=0xe3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 2147483648 (2048 MB)
avail memory = 2040631296 (1946 MB)
ACPI APIC Table: <GBT GBTUACPI>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
acpi0: <GBT GBTUACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 7f4e0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff
on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xe000-0xe007 mem
0xe6300000-0xe637ffff,0xd0000000-0xdfffffff,0xe6000000-0xe60fffff irq
16 at device 2.0 on pci0
agp0: <Intel G33 SVGA controller> on vgapci0
agp0: detected 7164k stolen memory
agp0: aperture size is 256M
uhci0: <Intel 82801I (ICH9) USB controller> port 0xe100-0xe11f irq 16
at device 26.0 on pci0
uhci0: [ITHREAD]
uhci0: LegSup = 0x2f00
usbus0: <Intel 82801I (ICH9) USB controller> on uhci0
uhci1: <Intel 82801I (ICH9) USB controller> port 0xe200-0xe21f irq 21
at device 26.1 on pci0
uhci1: [ITHREAD]
uhci1: LegSup = 0x2f00
usbus1: <Intel 82801I (ICH9) USB controller> on uhci1
uhci2: <Intel 82801I (ICH9) USB controller> port 0xe600-0xe61f irq 18
at device 26.2 on pci0
uhci2: [ITHREAD]
uhci2: LegSup = 0x2f00
usbus2: <Intel 82801I (ICH9) USB controller> on uhci2
ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem
0xe6384000-0xe63843ff irq 18 at device 26.7 on pci0
ehci0: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <Intel 82801I (ICH9) USB 2.0 controller> on ehci0
hdac0: <Intel 82801I High Definition Audio Controller> mem
0xe6380000-0xe6383fff irq 22 at device 27.0 on pci0
hdac0: HDA Driver Revision: 20090624_0136
hdac0: [ITHREAD]
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0xa000-0xa01f
mem 0xe1020000-0xe103ffff,0xe1000000-0xe101ffff irq 18 at device 0.0
on pci2
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:1b:21:05:00:b4
pcib3: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci3: <ACPI PCI bus> on pcib3
atapci0: <JMicron JMB363 SATA300 controller> port
0xb000-0xb007,0xb100-0xb103,0xb200-0xb207,0xb300-0xb303,0xb400-0xb40f
mem 0xe6100000-0xe6101fff irq 19 at device 0.0 on pci3
atapci0: [ITHREAD]
atapci0: AHCI called from vendor specific driver
atapci0: AHCI v1.00 controller with 2 3Gbps ports, PM supported
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
ata4: <ATA channel 2> on atapci0
ata4: [ITHREAD]
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci4: <ACPI PCI bus> on pcib4
re0: <RealTek 8168/8168B/8168C/8168CP/8168D/8168DP/8111B/8111C/8111CP/
8111DP PCIe Gigabit Ethernet> port 0xc000-0xc0ff mem
0xe3000000-0xe3000fff irq 16 at device 0.0 on pci4
re0: Using 1 MSI messages
re0: Chip rev. 0x38000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
re0: Ethernet address: 00:1a:4d:5a:97:87
re0: [FILTER]
uhci3: <Intel 82801I (ICH9) USB controller> port 0xe300-0xe31f irq 23
at device 29.0 on pci0
uhci3: [ITHREAD]
uhci3: LegSup = 0x2f00
usbus4: <Intel 82801I (ICH9) USB controller> on uhci3
uhci4: <Intel 82801I (ICH9) USB controller> port 0xe400-0xe41f irq 19
at device 29.1 on pci0
uhci4: [ITHREAD]
uhci4: LegSup = 0x2f00
usbus5: <Intel 82801I (ICH9) USB controller> on uhci4
uhci5: <Intel 82801I (ICH9) USB controller> port 0xe500-0xe51f irq 18
at device 29.2 on pci0
uhci5: [ITHREAD]
uhci5: LegSup = 0x2f00
usbus6: <Intel 82801I (ICH9) USB controller> on uhci5
ehci1: <Intel 82801I (ICH9) USB 2.0 controller> mem
0xe6385000-0xe63853ff irq 23 at device 29.7 on pci0
ehci1: [ITHREAD]
usbus7: EHCI version 1.0
usbus7: <Intel 82801I (ICH9) USB 2.0 controller> on ehci1
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci5: <ACPI PCI bus> on pcib5
amr0: <LSILogic MegaRAID 1.53> mem 0xe6200000-0xe620ffff irq 20 at
device 0.0 on pci5
amr0: Using 64-bit DMA
amr0: [ITHREAD]
amr0: delete logical drives supported by controller
amr0: <LSILogic MegaRAID SATA 150-4D> Firmware 713S, BIOS G121, 64MB RAM
xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xd000-0xd07f mem
0xe5004000-0xe500407f irq 19 at device 1.0 on pci5
miibus1: <MII bus> on xl0
xlphy0: <3c905C 10/100 internal PHY> PHY 24 on miibus1
xlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
xl0: Ethernet address: 00:04:76:ef:c6:36
xl0: [ITHREAD]
skc0: <SysKonnect Gigabit Ethernet (V1.0)> port 0xd100-0xd1ff mem
0xe5000000-0xe5003fff irq 18 at device 2.0 on pci5
pci0:5:2:0: invalid VPD data, remain 0xfc
skc0: SysKonnect SK-NET Gigabit Ethernet Adapter SK-9843 SX rev. (0x0)
sk0: <XaQti Corp. XMAC II> on skc0
sk0: Ethernet address: 00:00:5a:98:43:68
miibus2: <MII bus> on sk0
xmphy0: <XaQti Corp. XMAC II gigabit interface> PHY 0 on miibus2
xmphy0: 1000baseSX, 1000baseSX-FDX, auto
skc0: [ITHREAD]
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <Intel ICH9 SATA300 controller> port
0xe700-0xe707,0xe800-0xe803,0xe900-0xe907,0xea00-0xea03,0xeb00-0xeb1f
mem 0xe6386000-0xe63867ff irq 19 at device 31.2 on pci0
atapci1: [ITHREAD]
atapci1: AHCI called from vendor specific driver
atapci1: AHCI v1.20 controller with 6 3Gbps ports, PM supported
ata5: <ATA channel 0> on atapci1
ata5: [ITHREAD]
ata6: <ATA channel 1> on atapci1
ata6: [ITHREAD]
ata7: <ATA channel 2> on atapci1
ata7: [ITHREAD]
ata8: <ATA channel 3> on atapci1
ata8: [ITHREAD]
ata9: <ATA channel 4> on atapci1
ata9: [ITHREAD]
ata10: <ATA channel 5> on atapci1
ata10: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0
fdc0: [FILTER]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
cpu0: <ACPI CPU> on acpi0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 82a082a0600082a
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 82a082a0600082a
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
orm0: <ISA Option ROMs> at iomem 0xcc000-0xcc7ff,0xcd000-0xcefff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
isa0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is
present;
to enable, add "vfs.zfs.prefetch_disable=0" to /boot/
loader.conf.
ZFS filesystem version 13
ZFS storage pool version 13
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 480Mbps High Speed USB v2.0
ad4: 476938MB <Seagate ST3500630NS 3.AEG> at ata2-master SATA300
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ugen7.1: <Intel> at usbus7
uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
ad6: 1907729MB <WDC WD20EARS-00S8B1 80.00A80> at ata3-master SATA300
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
ad10: 715404MB <WDC WD7500AAKS-00RBA0 30.04G30> at ata5-master SATA300
ad12: 305245MB <Seagate ST3320620AS 3.AAE> at ata6-master SATA150
ad14: 715404MB <WDC WD7500AAKS-00RBA0 30.04G30> at ata7-master SATA300
ad16: 1907729MB <WDC WD20EARS-00S8B1 80.00A80> at ata8-master SATA300
GEOM_MIRROR: Device mirror/gm1a launched (1/1).
GEOM_MIRROR: Device mirror/gm1b launched (1/1).
ad18: 1907729MB <WDC WD20EARS-00S8B1 80.00A80> at ata9-master SATA300
GEOM_MIRROR: Device mirror/swap launched (1/1).
ad20: 286187MB <Maxtor 7L300S0 BANC1G10> at ata10-master SATA150
hdac0: HDA Codec #2: Realtek ALC885
pcm0: <HDA Realtek ALC885 PCM #0 Analog> at cad 2 nid 1 on hdac0
pcm1: <HDA Realtek ALC885 PCM #1 Analog> at cad 2 nid 1 on hdac0
pcm2: <HDA Realtek ALC885 PCM #2 Analog> at cad 2 nid 1 on hdac0
pcm3: <HDA Realtek ALC885 PCM #3 Analog> at cad 2 nid 1 on hdac0
pcm4: <HDA Realtek ALC885 PCM #4 Digital> at cad 2 nid 1 on hdac0
pcm5: <HDA Realtek ALC885 PCM #5 Analog> at cad 2 nid 1 on hdac0
amr0: delete logical drives supported by controller
amrd0: <LSILogic MegaRAID logical drive> on amr0
amrd0: 476935MB (976762880 sectors) RAID 0 (optimal)
amrd1: <LSILogic MegaRAID logical drive> on amr0
amrd1: 476935MB (976762880 sectors) RAID 0 (optimal)
amrd2: <LSILogic MegaRAID logical drive> on amr0
amrd2: 1907724MB (3907018752 sectors) RAID 0 (optimal)
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Root mount waiting for: usbus7 usbus3
uhub3: 6 ports with 6 removable, self powered
uhub7: 6 ports with 6 removable, self powered
Trying to mount root from zfs:zroot
ugen0.2: <Telldus> at usbus0
ugen1.2: <American Power Conversion> at usbus1
uma_zalloc_arg: zone "256" with the following non-sleepable locks held:
exclusive rw ifnet_rw (ifnet_rw) r = 0 (0xffffffff80e01f60) locked @ /
usr/src/sys/net/if.c:402
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2c
witness_warn() at witness_warn+0x2c2
uma_zalloc_arg() at uma_zalloc_arg+0x29d
malloc() at malloc+0x5d
if_grow() at if_grow+0x2f
if_alloc() at if_alloc+0x2b3
gif_clone_create() at gif_clone_create+0x53
ifc_simple_create() at ifc_simple_create+0x89
if_clone_createif() at if_clone_createif+0x64
ifioctl() at ifioctl+0x6b5
kern_ioctl() at kern_ioctl+0xf6
ioctl() at ioctl+0xfd
syscall() at syscall+0x19e
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800b8286c, rsp =
0x7fffffffe4a8, rbp = 0x7fffffffef6e ---
lock order reversal:
1st 0xffffffff80c093e0 pf task mtx (pf task mtx) @ /usr/src/sys/
contrib/pf/net/pf_ioctl.c:1393
2nd 0xffffffff80e01f60 ifnet_rw (ifnet_rw) @ /usr/src/sys/net/if.c:
2034
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2c
witness_checkorder() at witness_checkorder+0x66f
_rw_rlock() at _rw_rlock+0x29
ifunit() at ifunit+0x22
pfioctl() at pfioctl+0x262a
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xf6
ioctl() at ioctl+0xfd
syscall() at syscall+0x19e
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x80099886c, rsp =
0x7fffffffdb68, rbp = 0x7fffffffdc20 ---
lock order reversal:
1st 0xffffff00500c3098 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1054
2nd 0xffffff005010f448 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:
2083
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2c
witness_checkorder() at witness_checkorder+0x66f
__lockmgr_args() at __lockmgr_args+0x475
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46
_vn_lock() at _vn_lock+0x47
vget() at vget+0x56
devfs_allocv() at devfs_allocv+0x103
devfs_root() at devfs_root+0x48
vfs_donmount() at vfs_donmount+0xf43
nmount() at nmount+0x63
syscall() at syscall+0x19e
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x8007b04dc, rsp =
0x7fffffffdd28, rbp = 0x800a04048 ---
n
r
tun0: link state changed to UP
lock order reversal:
1st 0xffffff00500c3098 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1200
2nd 0xffffff005010f270 syncer (syncer) @ /usr/src/sys/kern/
vfs_subr.c:2188
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2c
witness_checkorder() at witness_checkorder+0x66f
__lockmgr_args() at __lockmgr_args+0x475
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46
_vn_lock() at _vn_lock+0x47
vrele() at vrele+0xc3
dounmount() at dounmount+0x269
unmount() at unmount+0x27e
syscall() at syscall+0x19e
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (22, FreeBSD ELF64, unmount), rip = 0x8006a09bc, rsp =
0x7fffffffde18, rbp = 0 ---
KDB: enter: manual escape to debugger
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8BA8475A-424E-489B-B643-8757F375320B>