Date: Tue, 9 Mar 2010 20:07:43 +0100 From: =?ISO-8859-1?Q?Johan_Str=F6m?= <johan@stromnet.se> To: =?ISO-8859-1?Q?Johan_Str=F6m?= <johan@stromnet.se> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS: zpool import hang on "zio->io_cv)" (with DDB output). Help needed! Message-ID: <4231C45D-499B-4FC9-90C3-BCC34DF20965@stromnet.se> In-Reply-To: <A6C7D84B-18D8-4614-A98C-4D8DC64A952C@stromnet.se> References: <8BA8475A-424E-489B-B643-8757F375320B@stromnet.se> <A6C7D84B-18D8-4614-A98C-4D8DC64A952C@stromnet.se>
next in thread | previous in thread | raw e-mail | index | archive | help
Got it working again, by ripping out my WD20EARS bench drives. Some =20 more background: The pool I was testing on was created like this (for 3 disks, ad6, =20 ad16, ad18): gpart create -s gpt adN gpart add -t freebsd-zfs -b 40 adN ...for all 3 drives.. zpool create bench raidz1 ad6 ad16 ad18 The reason for this was to test performance difference between running =20= straight on disk, and running on a 4k aligned partition (I was =20 benching and doing experiments on my new WD20EARS 4k sector size SATA =20= disks). Now, it was during bonnie++'ing these drives that the system paniced. =20= If that has anything at all to do with the panic, I do not know since =20= I was unable to obtain any dump/output. However, after booting again, it failed to import as described earlier. I now powered the machine down, and disconnected all the drives in the =20= above pool (ad6, ad16, ad18 and another amrd2 which was not in use at =20= the time). After power on, the import went fine! So, where was the problem? I don't really know. Could have been a =20 bunch of things I guess? Some thoughts: - ZFS failed to import from the GPT partitions for some reason - The ZFS labels was borked due to me having booted with invalid /=20 boot/zfs/zpool.cache (booting from old drive, new/correct loaders & =20 kernels though) - Borked ZFS labels due to panic. will try to plug the other disks =20 back in tomorrow and try to zdb out the labels. As a side note, from what I could get out of the testing, zfs on 4k =20 aligned partitions was not faster. However, for anyone using these =20 drives and UFS, aligning does make for a pretty nice improvement! Some =20= details can be found here: http://www.stromnet.se/~johan/bonnie-wd20ears-align-test.html (for =20 UFS, compare first two tables, unaligned vs aligned, on for example =20 Sequential output; block K/Sec and the latency). Disclaimer: this was not very scientific and I'm not sure if I'm =20 interpreting the results correctly, but it would seem that the average =20= values are 77-80MiB/s vs avg 55MiB/s, and latency sub 1000ms vs avg =20 2500ms. Exactly what that means I'm not sure but it IS better. :) =46rom what I've understood regarding zfs and 4k drivs, since ZFS =20 internally works with 512b blocks, it doesnt matter if I try to align =20= anyway, until the underlying system can report 4k sectors properly =20 (including getting these frikkin drives to do that, which they =20 currently don't). Anyhow, besides the big side note, if anyone cares to try to =20 reproduce.. Feel free :) I'll get back with zdb dumps on the other =20 drives later on. Thanks for ZFS and freebsd :) Johan On Mar 9, 2010, at 17:22 , Johan Str=F6m wrote: > Some followup, pjd tried to help me on IRC, without getting much =20 > further on the actual problem. > Some more information: > > Output from DDB PS: http://www.stromnet.se/~johan/back-1-ddb-ps.txt > Output from alltrace: = http://www.stromnet.se/~johan/back-1-ddb-alltrace.txt > > db> show alllocks > Process 2738 (sshd) thread 0xffffff0050418000 (100178) > Process 12 (intr) thread 0xffffff0002407ab0 (100018) > db> show lockedvnods > Locked vnodes > db > > > > Also tried accessing the disks (to make sure interrupts etc where =20 > alive) ,using dd if=3D/dev/adXX of=3D/dev/null count=3D1 on every disk = on =20 > the system, works fine. > Rebooted the system and tried with zfs debugging enabled too, and =20 > doing zpool import tank instead of just zpool import. no difference. =20= > Hanging there now. > > Johan > > > On Mar 9, 2010, at 15:26 , Johan Str=F6m wrote: > >> Hi List! >> >> I'm in the tedious process of upgrading my pool on my FreeBSD 8.0 =20 >> box (with a pool from 7.x). Yesterday I pulled a few disks from my =20= >> mirrored pool (one pool with multiple mirrors), in order to be able =20= >> to (free ports) plug in new ones and build my brand new pool. >> After removing disks, the pool got into state DEGRADED ofcourse, =20 >> since some of the disks where gone, but data was still there. I did =20= >> zpool detach on the removed disks, and the pool was ONLINE again, =20 >> all fine! >> Then I rebooted to make another disk available through my old LSI =20 >> MegaRaid card (didnt take time to figure out cryptic syntax of the =20= >> megarc CLI util). On boot, I was met by something similar to this: >> >> pool: tank >> state: UNAVAIL >> status: One or more devices could not be used because the label is =20= >> missing >> or invalid. There are insufficient replicas for the pool to =20= >> continue >> functioning. >> action: Destroy and re-create the pool from a backup source. >> see: http://www.sun.com/msg/ZFS-8000-5E >> scrub: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> tank UNAVAIL 0 0 0 insufficient replicas >> ad10s1d ONLINE 0 0 0 >> mirror DEGRADED 0 0 0 >> ad12 FAULTED 0 0 0 corrupted data >> ad16 UNAVAIL 0 0 0 corrupted data >> mirror DEGRADED 0 0 0 >> ad20 FAULTED 0 0 0 corrupted data >> ad18 UNAVAIL 0 0 0 corrupted data >> mirror UNAVAIL 0 0 0 insufficient replicas >> ad6 UNAVAIL 0 0 0 corrupted data >> ad4 FAULTED 0 0 0 corrupted data >> mirror ONLINE 0 0 0 >> amrd0 ONLINE 0 0 0 >> amrd1 ONLINE 0 0 0 >> >> The reason the disks was still there was due to me haveing an old =20 >> zfs.cache file in the boot (it boots from anothre drive, but thats =20= >> another story..) >> >> In this case, a export/import did the trick, after reimporting, the =20= >> pool was back online. All fine. I created a new pool and did some =20 >> bonnie++ testing on it, and suddenly the box paniced or something =20 >> (I didn't have dumpon enabled :/ and didn't see the screen until it =20= >> rebooted).. >> >> Now my problems come. The box came up again, with the above output. =20= >> I tried zpool export again, fine. Zpool import however, hang. =20 >> Waited an hour, nothing. After rebuilding the kernel with DDB/=20 >> witness and did the import again, i've managed to get this output =20 >> (im not really sure what is usable here): >> >> back-1 # zpool import >> load: 0.29 cmd: zpool 3193 [zio->io_cv)] 2.54r 0.00u 0.01s 0% 2236k >> >> >> >> In DDB: >> > tr 3193 >> >> Tracing pid 3193 tid 100122 td 0xffffff00035ca390 >> sched_switch() at sched_switch+0xde >> mi_switch() at mi_switch+0x170 >> sleepq_wait() at sleepq_wait+0x44 >> _cv_wait() at _cv_wait+0x13c >> zio_wait() at zio_wait+0x61 >> arc_read_nolock() at arc_read_nolock+0x345 >> dmu_objset_open_impl() at dmu_objset_open_impl+0xd0 >> dsl_pool_open() at dsl_pool_open+0x5a >> spa_load() at spa_load+0x31b >> spa_tryimport() at spa_tryimport+0xa9 >> zfs_ioc_pool_tryimport() at zfs_ioc_pool_tryimport+0x3f >> zfsdev_ioctl() at zfsdev_ioctl+0x8d >> devfs_ioctl_f() at devfs_ioctl_f+0x76 >> kern_ioctl() at kern_ioctl+0xf6 >> ioctl() at ioctl+0xfd >> syscall() at syscall+0x19e >> Xfast_syscall() at Xfast_syscall+0xe1 >> --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x8010eb86c, rsp =3D =20= >> 0x7fffffff8e28, rbp =3D 0x801323300 --- >> >> >> db> show thread 100122 >> >> Thread 100122 at 0xffffff00035ca390: >> proc (pid 3193): 0xffffff000383c460 >> name: zpool >> stack: 0xffffff805740a000-0xffffff805740dfff >> flags: 0x44 pflags: 0x10000 >> state: INHIBITED: {SLEEPING} >> wmesg: zio->io_cv) wchan: 0xffffff00506e5858 >> priority: 131 >> container lock: sleepq chain (0xffffffff80c61e68) >> db> >> >> db> show sleepchain 10012 >> >> db> show sleepchain 3193 >> >> thread 100122 (pid 3193, zpool) sleeping on 0xffffff00506e5858 "zio-=20= >> >io_cv)" >> db> show lock 0xffffff00506e5858 >> >> class: spin mutex >> name: zio->io_cv) >> flags: {SPIN} >> state: {OWNED} >> >> >> >> >> The box is currenlty in this state, so if you reply to me now I can =20= >> continue to do debugging according to instructions. >> Since I cannot get anywhere on my own with this, all and any help =20 >> is appreciated since I really need this pool to get online.. >> >> dmesg is posted below (including some LOR?) >> >> Thanks! >> Johan >> >> >> >> DMESG: >> >> Copyright (c) 1992-2009 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, =20= >> 1994 >> The Regents of the University of California. All rights =20 >> reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 8.0-RELEASE-p2 #10: Tue Mar 9 12:44:15 CET 2010 >> johan@back-1.stromnet.se:/usr/obj/usr/src/sys/BACK1 >> WARNING: WITNESS option enabled, expect reduced performance. >> Timecounter "i8254" frequency 1193182 Hz quality 0 >> CPU: Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz (2666.68-MHz =20 >> K8-class CPU) >> Origin =3D "GenuineIntel" Id =3D 0x6fb Stepping =3D 11 >> Features=20 >> =3D=20 >> 0xbfebfbff=20 >> <=20 >> FPU=20 >> ,VME=20 >> ,DE=20 >> ,PSE=20 >> ,TSC=20 >> ,MSR=20 >> ,PAE=20 >> ,MCE=20 >> ,CX8=20 >> ,APIC=20 >> ,SEP=20 >> ,MTRR=20 >> ,PGE=20 >> ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> >> Features2=20 >> =3D0xe3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM> >> AMD Features=3D0x20100800<SYSCALL,NX,LM> >> AMD Features2=3D0x1<LAHF> >> TSC: P-state invariant >> real memory =3D 2147483648 (2048 MB) >> avail memory =3D 2040631296 (1946 MB) >> ACPI APIC Table: <GBT GBTUACPI> >> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >> FreeBSD/SMP: 1 package(s) x 2 core(s) >> cpu0 (BSP): APIC ID: 0 >> cpu1 (AP): APIC ID: 1 >> ioapic0: Changing APIC ID to 2 >> ioapic0 <Version 2.0> irqs 0-23 on motherboard >> kbd1 at kbdmux0 >> cryptosoft0: <software crypto> on motherboard >> acpi0: <GBT GBTUACPI> on motherboard >> acpi0: [ITHREAD] >> acpi0: Power Button (fixed) >> acpi0: reservation of 0, a0000 (3) failed >> acpi0: reservation of 100000, 7f4e0000 (3) failed >> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 >> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >> acpi_hpet0: <High Precision Event Timer> iomem =20 >> 0xfed00000-0xfed003ff on acpi0 >> Timecounter "HPET" frequency 14318180 Hz quality 900 >> acpi_button0: <Power Button> on acpi0 >> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 >> pci0: <ACPI PCI bus> on pcib0 >> vgapci0: <VGA-compatible display> port 0xe000-0xe007 mem =20 >> 0xe6300000-0xe637ffff,0xd0000000-0xdfffffff,0xe6000000-0xe60fffff =20 >> irq 16 at device 2.0 on pci0 >> agp0: <Intel G33 SVGA controller> on vgapci0 >> agp0: detected 7164k stolen memory >> agp0: aperture size is 256M >> uhci0: <Intel 82801I (ICH9) USB controller> port 0xe100-0xe11f irq =20= >> 16 at device 26.0 on pci0 >> uhci0: [ITHREAD] >> uhci0: LegSup =3D 0x2f00 >> usbus0: <Intel 82801I (ICH9) USB controller> on uhci0 >> uhci1: <Intel 82801I (ICH9) USB controller> port 0xe200-0xe21f irq =20= >> 21 at device 26.1 on pci0 >> uhci1: [ITHREAD] >> uhci1: LegSup =3D 0x2f00 >> usbus1: <Intel 82801I (ICH9) USB controller> on uhci1 >> uhci2: <Intel 82801I (ICH9) USB controller> port 0xe600-0xe61f irq =20= >> 18 at device 26.2 on pci0 >> uhci2: [ITHREAD] >> uhci2: LegSup =3D 0x2f00 >> usbus2: <Intel 82801I (ICH9) USB controller> on uhci2 >> ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem =20 >> 0xe6384000-0xe63843ff irq 18 at device 26.7 on pci0 >> ehci0: [ITHREAD] >> usbus3: EHCI version 1.0 >> usbus3: <Intel 82801I (ICH9) USB 2.0 controller> on ehci0 >> hdac0: <Intel 82801I High Definition Audio Controller> mem =20 >> 0xe6380000-0xe6383fff irq 22 at device 27.0 on pci0 >> hdac0: HDA Driver Revision: 20090624_0136 >> hdac0: [ITHREAD] >> pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 >> pci1: <ACPI PCI bus> on pcib1 >> pcib2: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0 >> pci2: <ACPI PCI bus> on pcib2 >> em0: <Intel(R) PRO/1000 Network Connection 6.9.14> port =20 >> 0xa000-0xa01f mem 0xe1020000-0xe103ffff,0xe1000000-0xe101ffff irq =20 >> 18 at device 0.0 on pci2 >> em0: Using MSI interrupt >> em0: [FILTER] >> em0: Ethernet address: 00:1b:21:05:00:b4 >> pcib3: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0 >> pci3: <ACPI PCI bus> on pcib3 >> atapci0: <JMicron JMB363 SATA300 controller> port =20 >> 0xb000=20 >> -0xb007,0xb100-0xb103,0xb200-0xb207,0xb300-0xb303,0xb400-0xb40f mem =20= >> 0xe6100000-0xe6101fff irq 19 at device 0.0 on pci3 >> atapci0: [ITHREAD] >> atapci0: AHCI called from vendor specific driver >> atapci0: AHCI v1.00 controller with 2 3Gbps ports, PM supported >> ata2: <ATA channel 0> on atapci0 >> ata2: [ITHREAD] >> ata3: <ATA channel 1> on atapci0 >> ata3: [ITHREAD] >> ata4: <ATA channel 2> on atapci0 >> ata4: [ITHREAD] >> pcib4: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0 >> pci4: <ACPI PCI bus> on pcib4 >> re0: <RealTek 8168/8168B/8168C/8168CP/8168D/8168DP/8111B/8111C/=20 >> 8111CP/8111DP PCIe Gigabit Ethernet> port 0xc000-0xc0ff mem =20 >> 0xe3000000-0xe3000fff irq 16 at device 0.0 on pci4 >> re0: Using 1 MSI messages >> re0: Chip rev. 0x38000000 >> re0: MAC rev. 0x00000000 >> miibus0: <MII bus> on re0 >> rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0 >> rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, =20 >> 1000baseT, 1000baseT-FDX, auto >> re0: Ethernet address: 00:1a:4d:5a:97:87 >> re0: [FILTER] >> uhci3: <Intel 82801I (ICH9) USB controller> port 0xe300-0xe31f irq =20= >> 23 at device 29.0 on pci0 >> uhci3: [ITHREAD] >> uhci3: LegSup =3D 0x2f00 >> usbus4: <Intel 82801I (ICH9) USB controller> on uhci3 >> uhci4: <Intel 82801I (ICH9) USB controller> port 0xe400-0xe41f irq =20= >> 19 at device 29.1 on pci0 >> uhci4: [ITHREAD] >> uhci4: LegSup =3D 0x2f00 >> usbus5: <Intel 82801I (ICH9) USB controller> on uhci4 >> uhci5: <Intel 82801I (ICH9) USB controller> port 0xe500-0xe51f irq =20= >> 18 at device 29.2 on pci0 >> uhci5: [ITHREAD] >> uhci5: LegSup =3D 0x2f00 >> usbus6: <Intel 82801I (ICH9) USB controller> on uhci5 >> ehci1: <Intel 82801I (ICH9) USB 2.0 controller> mem =20 >> 0xe6385000-0xe63853ff irq 23 at device 29.7 on pci0 >> ehci1: [ITHREAD] >> usbus7: EHCI version 1.0 >> usbus7: <Intel 82801I (ICH9) USB 2.0 controller> on ehci1 >> pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0 >> pci5: <ACPI PCI bus> on pcib5 >> amr0: <LSILogic MegaRAID 1.53> mem 0xe6200000-0xe620ffff irq 20 at =20= >> device 0.0 on pci5 >> amr0: Using 64-bit DMA >> amr0: [ITHREAD] >> amr0: delete logical drives supported by controller >> amr0: <LSILogic MegaRAID SATA 150-4D> Firmware 713S, BIOS G121, =20 >> 64MB RAM >> xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xd000-0xd07f mem =20 >> 0xe5004000-0xe500407f irq 19 at device 1.0 on pci5 >> miibus1: <MII bus> on xl0 >> xlphy0: <3c905C 10/100 internal PHY> PHY 24 on miibus1 >> xlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto >> xl0: Ethernet address: 00:04:76:ef:c6:36 >> xl0: [ITHREAD] >> skc0: <SysKonnect Gigabit Ethernet (V1.0)> port 0xd100-0xd1ff mem =20 >> 0xe5000000-0xe5003fff irq 18 at device 2.0 on pci5 >> pci0:5:2:0: invalid VPD data, remain 0xfc >> skc0: SysKonnect SK-NET Gigabit Ethernet Adapter SK-9843 SX rev. =20 >> (0x0) >> sk0: <XaQti Corp. XMAC II> on skc0 >> sk0: Ethernet address: 00:00:5a:98:43:68 >> miibus2: <MII bus> on sk0 >> xmphy0: <XaQti Corp. XMAC II gigabit interface> PHY 0 on miibus2 >> xmphy0: 1000baseSX, 1000baseSX-FDX, auto >> skc0: [ITHREAD] >> isab0: <PCI-ISA bridge> at device 31.0 on pci0 >> isa0: <ISA bus> on isab0 >> atapci1: <Intel ICH9 SATA300 controller> port =20 >> 0xe700=20 >> -0xe707,0xe800-0xe803,0xe900-0xe907,0xea00-0xea03,0xeb00-0xeb1f mem =20= >> 0xe6386000-0xe63867ff irq 19 at device 31.2 on pci0 >> atapci1: [ITHREAD] >> atapci1: AHCI called from vendor specific driver >> atapci1: AHCI v1.20 controller with 6 3Gbps ports, PM supported >> ata5: <ATA channel 0> on atapci1 >> ata5: [ITHREAD] >> ata6: <ATA channel 1> on atapci1 >> ata6: [ITHREAD] >> ata7: <ATA channel 2> on atapci1 >> ata7: [ITHREAD] >> ata8: <ATA channel 3> on atapci1 >> ata8: [ITHREAD] >> ata9: <ATA channel 4> on atapci1 >> ata9: [ITHREAD] >> ata10: <ATA channel 5> on atapci1 >> ata10: [ITHREAD] >> pci0: <serial bus, SMBus> at device 31.3 (no driver attached) >> atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0 >> fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 =20= >> on acpi0 >> fdc0: [FILTER] >> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on =20 >> acpi0 >> uart0: [FILTER] >> ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0 >> ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode >> ppc0: [ITHREAD] >> ppbus0: <Parallel port bus> on ppc0 >> plip0: <PLIP network interface> on ppbus0 >> plip0: [ITHREAD] >> lpt0: <Printer> on ppbus0 >> lpt0: [ITHREAD] >> lpt0: Interrupt-driven port >> ppi0: <Parallel I/O> on ppbus0 >> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 >> atkbd0: <AT Keyboard> irq 1 on atkbdc0 >> kbd0 at atkbd0 >> atkbd0: [GIANT-LOCKED] >> atkbd0: [ITHREAD] >> cpu0: <ACPI CPU> on acpi0 >> est0: <Enhanced SpeedStep Frequency Control> on cpu0 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 82a082a0600082a >> device_attach: est0 attach returned 6 >> p4tcc0: <CPU Frequency Thermal Control> on cpu0 >> cpu1: <ACPI CPU> on acpi0 >> est1: <Enhanced SpeedStep Frequency Control> on cpu1 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 82a082a0600082a >> device_attach: est1 attach returned 6 >> p4tcc1: <CPU Frequency Thermal Control> on cpu1 >> orm0: <ISA Option ROMs> at iomem 0xcc000-0xcc7ff,0xcd000-0xcefff on =20= >> isa0 >> sc0: <System console> at flags 0x100 on isa0 >> sc0: VGA <16 virtual consoles, flags=3D0x300> >> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff =20 >> on isa0 >> ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM =20= >> is present; >> to enable, add "vfs.zfs.prefetch_disable=3D0" to /boot/=20 >> loader.conf. >> ZFS filesystem version 13 >> ZFS storage pool version 13 >> Timecounters tick every 1.000 msec >> IPsec: Initialized Security Association Processing. >> usbus0: 12Mbps Full Speed USB v1.0 >> usbus1: 12Mbps Full Speed USB v1.0 >> usbus2: 12Mbps Full Speed USB v1.0 >> usbus3: 480Mbps High Speed USB v2.0 >> usbus4: 12Mbps Full Speed USB v1.0 >> usbus5: 12Mbps Full Speed USB v1.0 >> usbus6: 12Mbps Full Speed USB v1.0 >> usbus7: 480Mbps High Speed USB v2.0 >> ad4: 476938MB <Seagate ST3500630NS 3.AEG> at ata2-master SATA300 >> ugen0.1: <Intel> at usbus0 >> uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on =20 >> usbus0 >> ugen1.1: <Intel> at usbus1 >> uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on =20 >> usbus1 >> ugen2.1: <Intel> at usbus2 >> uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on =20 >> usbus2 >> ugen3.1: <Intel> at usbus3 >> uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on =20 >> usbus3 >> ugen4.1: <Intel> at usbus4 >> uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on =20 >> usbus4 >> ugen5.1: <Intel> at usbus5 >> uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on =20 >> usbus5 >> ugen6.1: <Intel> at usbus6 >> uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on =20 >> usbus6 >> ugen7.1: <Intel> at usbus7 >> uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on =20 >> usbus7 >> ad6: 1907729MB <WDC WD20EARS-00S8B1 80.00A80> at ata3-master SATA300 >> uhub0: 2 ports with 2 removable, self powered >> uhub1: 2 ports with 2 removable, self powered >> uhub2: 2 ports with 2 removable, self powered >> uhub4: 2 ports with 2 removable, self powered >> uhub5: 2 ports with 2 removable, self powered >> uhub6: 2 ports with 2 removable, self powered >> ad10: 715404MB <WDC WD7500AAKS-00RBA0 30.04G30> at ata5-master =20 >> SATA300 >> ad12: 305245MB <Seagate ST3320620AS 3.AAE> at ata6-master SATA150 >> ad14: 715404MB <WDC WD7500AAKS-00RBA0 30.04G30> at ata7-master =20 >> SATA300 >> ad16: 1907729MB <WDC WD20EARS-00S8B1 80.00A80> at ata8-master SATA300 >> GEOM_MIRROR: Device mirror/gm1a launched (1/1). >> GEOM_MIRROR: Device mirror/gm1b launched (1/1). >> ad18: 1907729MB <WDC WD20EARS-00S8B1 80.00A80> at ata9-master SATA300 >> GEOM_MIRROR: Device mirror/swap launched (1/1). >> ad20: 286187MB <Maxtor 7L300S0 BANC1G10> at ata10-master SATA150 >> hdac0: HDA Codec #2: Realtek ALC885 >> pcm0: <HDA Realtek ALC885 PCM #0 Analog> at cad 2 nid 1 on hdac0 >> pcm1: <HDA Realtek ALC885 PCM #1 Analog> at cad 2 nid 1 on hdac0 >> pcm2: <HDA Realtek ALC885 PCM #2 Analog> at cad 2 nid 1 on hdac0 >> pcm3: <HDA Realtek ALC885 PCM #3 Analog> at cad 2 nid 1 on hdac0 >> pcm4: <HDA Realtek ALC885 PCM #4 Digital> at cad 2 nid 1 on hdac0 >> pcm5: <HDA Realtek ALC885 PCM #5 Analog> at cad 2 nid 1 on hdac0 >> amr0: delete logical drives supported by controller >> amrd0: <LSILogic MegaRAID logical drive> on amr0 >> amrd0: 476935MB (976762880 sectors) RAID 0 (optimal) >> amrd1: <LSILogic MegaRAID logical drive> on amr0 >> amrd1: 476935MB (976762880 sectors) RAID 0 (optimal) >> amrd2: <LSILogic MegaRAID logical drive> on amr0 >> amrd2: 1907724MB (3907018752 sectors) RAID 0 (optimal) >> SMP: AP CPU #1 Launched! >> WARNING: WITNESS option enabled, expect reduced performance. >> Root mount waiting for: usbus7 usbus3 >> uhub3: 6 ports with 6 removable, self powered >> uhub7: 6 ports with 6 removable, self powered >> Trying to mount root from zfs:zroot >> ugen0.2: <Telldus> at usbus0 >> ugen1.2: <American Power Conversion> at usbus1 >> uma_zalloc_arg: zone "256" with the following non-sleepable locks =20 >> held: >> exclusive rw ifnet_rw (ifnet_rw) r =3D 0 (0xffffffff80e01f60) locked =20= >> @ /usr/src/sys/net/if.c:402 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> _witness_debugger() at _witness_debugger+0x2c >> witness_warn() at witness_warn+0x2c2 >> uma_zalloc_arg() at uma_zalloc_arg+0x29d >> malloc() at malloc+0x5d >> if_grow() at if_grow+0x2f >> if_alloc() at if_alloc+0x2b3 >> gif_clone_create() at gif_clone_create+0x53 >> ifc_simple_create() at ifc_simple_create+0x89 >> if_clone_createif() at if_clone_createif+0x64 >> ifioctl() at ifioctl+0x6b5 >> kern_ioctl() at kern_ioctl+0xf6 >> ioctl() at ioctl+0xfd >> syscall() at syscall+0x19e >> Xfast_syscall() at Xfast_syscall+0xe1 >> --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800b8286c, rsp =3D =20= >> 0x7fffffffe4a8, rbp =3D 0x7fffffffef6e --- >> lock order reversal: >> 1st 0xffffffff80c093e0 pf task mtx (pf task mtx) @ /usr/src/sys/=20 >> contrib/pf/net/pf_ioctl.c:1393 >> 2nd 0xffffffff80e01f60 ifnet_rw (ifnet_rw) @ /usr/src/sys/net/if.c:=20= >> 2034 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> _witness_debugger() at _witness_debugger+0x2c >> witness_checkorder() at witness_checkorder+0x66f >> _rw_rlock() at _rw_rlock+0x29 >> ifunit() at ifunit+0x22 >> pfioctl() at pfioctl+0x262a >> devfs_ioctl_f() at devfs_ioctl_f+0x76 >> kern_ioctl() at kern_ioctl+0xf6 >> ioctl() at ioctl+0xfd >> syscall() at syscall+0x19e >> Xfast_syscall() at Xfast_syscall+0xe1 >> --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x80099886c, rsp =3D =20= >> 0x7fffffffdb68, rbp =3D 0x7fffffffdc20 --- >> lock order reversal: >> 1st 0xffffff00500c3098 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1054 >> 2nd 0xffffff005010f448 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:=20= >> 2083 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> _witness_debugger() at _witness_debugger+0x2c >> witness_checkorder() at witness_checkorder+0x66f >> __lockmgr_args() at __lockmgr_args+0x475 >> vop_stdlock() at vop_stdlock+0x39 >> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 >> _vn_lock() at _vn_lock+0x47 >> vget() at vget+0x56 >> devfs_allocv() at devfs_allocv+0x103 >> devfs_root() at devfs_root+0x48 >> vfs_donmount() at vfs_donmount+0xf43 >> nmount() at nmount+0x63 >> syscall() at syscall+0x19e >> Xfast_syscall() at Xfast_syscall+0xe1 >> --- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x8007b04dc, rsp =3D = =20 >> 0x7fffffffdd28, rbp =3D 0x800a04048 --- >> n >> >> r >> tun0: link state changed to UP >> lock order reversal: >> 1st 0xffffff00500c3098 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1200 >> 2nd 0xffffff005010f270 syncer (syncer) @ /usr/src/sys/kern/=20 >> vfs_subr.c:2188 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> _witness_debugger() at _witness_debugger+0x2c >> witness_checkorder() at witness_checkorder+0x66f >> __lockmgr_args() at __lockmgr_args+0x475 >> vop_stdlock() at vop_stdlock+0x39 >> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 >> _vn_lock() at _vn_lock+0x47 >> vrele() at vrele+0xc3 >> dounmount() at dounmount+0x269 >> unmount() at unmount+0x27e >> syscall() at syscall+0x19e >> Xfast_syscall() at Xfast_syscall+0xe1 >> --- syscall (22, FreeBSD ELF64, unmount), rip =3D 0x8006a09bc, rsp =3D = =20 >> 0x7fffffffde18, rbp =3D 0 --- >> KDB: enter: manual escape to debugger >> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4231C45D-499B-4FC9-90C3-BCC34DF20965>