From owner-freebsd-stable@FreeBSD.ORG Sun Jul 18 21:08:11 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E6EA3106564A for ; Sun, 18 Jul 2010 21:08:10 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by mx1.freebsd.org (Postfix) with ESMTP id 8AB478FC08 for ; Sun, 18 Jul 2010 21:08:10 +0000 (UTC) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.14.4/8.14.3) with ESMTP id o6IL88eG043887 for ; Sun, 18 Jul 2010 17:08:08 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <201007182108.o6IL88eG043887@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Sun, 18 Jul 2010 17:08:09 -0400 To: freebsd-stable@freebsd.org From: Mike Tancsa Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: deadlock or bad disk ? RELENG_8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Jul 2010 21:08:11 -0000 On the serial console I see swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 and on a session I had open from before # killall -9 watchdogd just hangs, I guess because its having trouble reading from the disk. If I hit CTRL+t, I see load: 0.00 cmd: csh 73167 [vnread] 22.32r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 22.65r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 22.96r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 23.20r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 23.40r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 23.61r 0.00u 0.00s 0% 3232k Its RELENG_8 amd64 from July 13th and the swap is on an ARECA drive and I dont see any errors on any of the raidset members. I also have a large zfs spool and a small mount point on a 3ware controller but unfortunately, nothing in the logs post reboot and nothing from smartctl cat /var/run/dmesg.boot Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-PRERELEASE #0: Tue Jul 13 09:55:48 EDT 2010 mdtancsa@backup3.sentex.ca:/usr/obj/usr/src/sys/backup amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (2400.10-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 Features=0xbfebfbff Features2=0xe3bd AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 8267673600 (7884 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of fed08000, 1000 (3) failed acpi0: reservation of fed1c000, 4000 (3) failed acpi0: reservation of fed20000, 20000 (3) failed acpi0: reservation of fed50000, 40000 (3) failed acpi0: reservation of ffc00000, 300000 (3) failed acpi0: reservation of fec00000, 1000 (3) failed acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of e0000000, 10000000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, dff00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 ACPI Warning: Incorrect checksum in table [OEMB] - 0xD1, should be 0xD0 (20100331/tbutils-354) cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 pcib2: at device 0.0 on pci1 pci3: on pcib2 arcmsr0: mem 0xfc9ff000-0xfc9fffff irq 18 at device 14.0 on pci3 ARECA RAID ADAPTER0: Driver Version 1.20.00.16 2009-10-10 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.44 2008-2-1 arcmsr0: [ITHREAD] pcib3: at device 0.2 on pci1 pci2: on pcib3 uhci0: port 0x7800-0x781f irq 16 at device 26.0 on pci0 uhci0: [ITHREAD] usbus0: on uhci0 uhci1: port 0x7880-0x789f irq 21 at device 26.1 on pci0 uhci1: [ITHREAD] usbus1: on uhci1 uhci2: port 0x7c00-0x7c1f irq 18 at device 26.2 on pci0 uhci2: [ITHREAD] usbus2: on uhci2 ehci0: mem 0xfc8ffc00-0xfc8fffff irq 18 at device 26.7 on pci0 ehci0: [ITHREAD] usbus3: EHCI version 1.0 usbus3: on ehci0 pci0: at device 27.0 (no driver attached) pcib4: irq 17 at device 28.0 on pci0 pci9: on pcib4 em0: port 0xdc00-0xdc1f mem 0xfcfe0000-0xfcffffff,0xfcf00000-0xfcf7ffff,0xfcfdc000-0xfcfdffff irq 16 at device 0.0 on pci9 em0: Using MSI interrupt em0: [FILTER] em0: Ethernet address: 00:1b:21:3f:62:72 pcib5: irq 16 at device 28.1 on pci0 pci8: on pcib5 siis0: port 0xcc00-0xcc7f mem 0xfceffc00-0xfceffc7f,0xfcef8000-0xfcefbfff irq 17 at device 0.0 on pci8 siis0: [ITHREAD] siisch0: at channel 0 on siis0 siisch0: [ITHREAD] siisch1: at channel 1 on siis0 siisch1: [ITHREAD] pcib6: irq 18 at device 28.2 on pci0 pci7: on pcib6 3ware device driver for 9000 series storage controllers, version: 3.80.06.002 twa0: <3ware 9000 series Storage Controller> port 0xb800-0xb8ff mem 0xfa000000-0xfbffffff,0xfcdff000-0xfcdfffff irq 18 at device 0.0 on pci7 twa0: [ITHREAD] twa0: WARNING: (0x04: 0x0008): Unclean shutdown detected: unit=0 twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-2LP, 2 ports, Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004 pcib7: irq 19 at device 28.3 on pci0 pci6: on pcib7 fwohci0: <1394 Open Host Controller Interface> port 0xa800-0xa8ff mem 0xfccff800-0xfccfffff irq 19 at device 0.0 on pci6 fwohci0: [ITHREAD] fwohci0: OHCI version 1.10 (ROM=1) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 00:1e:8c:00:00:c4:10:80 fwohci0: Phy 1394a available S400, 2 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: on fwohci0 dcons_crom0: on firewire0 dcons_crom0: bus_addr 0x8eacc0 fwe0: on firewire0 if_fwe0: Fake Ethernet address: 02:1e:8c:c4:10:80 fwe0: Ethernet address: 02:1e:8c:c4:10:80 fwip0: on firewire0 fwip0: Firewire address: 00:1e:8c:00:00:c4:10:80 @ 0xfffe00000000, S400, maxrec 2048 fwohci0: Initiate bus reset fwohci0: fwohci_intr_core: BUS reset fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode pcib8: irq 17 at device 28.4 on pci0 pci5: on pcib8 ahci0: mem 0xfcbfa000-0xfcbfbfff irq 16 at device 0.0 on pci5 ahci0: [ITHREAD] ahci0: AHCI v1.00 with 2 3Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 ahcich0: [ITHREAD] ahcich1: at channel 1 on ahci0 ahcich1: [ITHREAD] atapci0: port 0x9c00-0x9c07,0x9880-0x9883,0x9800-0x9807,0x9480-0x9483,0x9400-0x940f irq 17 at device 0.1 on pci5 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] pcib9: irq 16 at device 28.5 on pci0 pci4: on pcib9 ale0: port 0x8c00-0x8c7f mem 0xfcac0000-0xfcafffff irq 17 at device 0.0 on pci4 ale0: 960 Tx FIFO, 1024 Rx FIFO ale0: Using 1 MSI messages. miibus0: on ale0 atphy0: PHY 0 on miibus0 atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, auto ale0: Ethernet address: e0:cb:4e:42:4b:37 ale0: [FILTER] uhci3: port 0x7080-0x709f irq 23 at device 29.0 on pci0 uhci3: [ITHREAD] usbus4: on uhci3 uhci4: port 0x7400-0x741f irq 19 at device 29.1 on pci0 uhci4: [ITHREAD] usbus5: on uhci4 uhci5: port 0x7480-0x749f irq 18 at device 29.2 on pci0 uhci5: [ITHREAD] usbus6: on uhci5 ehci1: mem 0xfc8ff800-0xfc8ffbff irq 23 at device 29.7 on pci0 ehci1: [ITHREAD] usbus7: EHCI version 1.0 usbus7: on ehci1 pcib10: at device 30.0 on pci0 pci10: on pcib10 vgapci0: port 0xe000-0xe0ff mem 0xfd000000-0xfdffffff,0xfebff000-0xfebfffff irq 16 at device 0.0 on pci10 isab0: at device 31.0 on pci0 isa0: on isab0 ahci1: port 0x6c00-0x6c07,0x6880-0x6883,0x6800-0x6807,0x6480-0x6483,0x6400-0x641f mem 0xfc8fe800-0xfc8fefff irq 19 at device 31.2 on pci0 ahci1: [ITHREAD] ahci1: AHCI v1.20 with 6 3Gbps ports, Port Multiplier supported ahcich2: at channel 0 on ahci1 ahcich2: [ITHREAD] ahcich3: at channel 1 on ahci1 ahcich3: [ITHREAD] ahcich4: at channel 2 on ahci1 ahcich4: [ITHREAD] ahcich5: at channel 3 on ahci1 ahcich5: [ITHREAD] ahcich6: at channel 4 on ahci1 ahcich6: [ITHREAD] ahcich7: at channel 5 on ahci1 ahcich7: [ITHREAD] pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart0: console (9600,n,8,1) atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] orm0: at iomem 0xc0000-0xc97ff,0xc9800-0xca7ff,0xca800-0xcc7ff,0xd4800-0xd77ff,0xd7800-0xd87ff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 est0: on cpu0 p4tcc0: on cpu0 est1: on cpu1 p4tcc1: on cpu1 est2: on cpu2 p4tcc2: on cpu2 est3: on cpu3 p4tcc3: on cpu3 Timecounters tick every 1.000 msec firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) firewire0: bus manager 0 usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 12Mbps Full Speed USB v1.0 usbus7: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 ugen7.1: at usbus7 uhub7: on usbus7 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub6: 2 ports with 2 removable, self powered (probe16:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step da0 at arcmsr0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 166.666MB/s transfers (83.333MHz, offset 32, 16bit) da0: Command Queueing enabled da0: 76293MB (156249600 512 byte sectors: 255H 63S/T 9726C) da1 at arcmsr0 bus 0 scbus0 target 0 lun 1 da1: Fixed Direct Access SCSI-5 device da1: 166.666MB/s transfers (83.333MHz, offset 32, 16bit) da1: Command Queueing enabled da1: 2784728MB (5703123456 512 byte sectors: 255H 63S/T 355003C) ada0 at ahcich2 bus 0 scbus6 target 0 lun 0da2 at twa0 bus 0 scbus3 target 0 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 100.000MB/s transfers da2: 66747MB (136697856 512 byte sectors: 255H 63S/T 8509C) ada0: ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada1 at ahcich3 bus 0 scbus7 target 0 lun 0 ada1: ATA-8 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada2 at ahcich4 bus 0 scbus8 target 0 lun 0 ada2: ATA-8 SATA 2.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada3 at ahcich5 bus 0 scbus9 target 0 lun 0 ada3: ATA-8 SATA 2.x device ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) pass2 at arcmsr0 bus 0 scbus0 target 16 lun 0 pass2: Fixed Processor SCSI-0 device SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! uhub3: 6 ports with 6 removable, self powered uhub7: 6 ports with 6 removable, self powered Root mount waiting for: usbus7 Trying to mount root from ufs:/dev/da2s1a WARNING: / was not properly dismounted ZFS filesystem version 3 ZFS storage pool version 14 ugen5.2: at usbus5 twa0: INFO: (0x04: 0x000C): Initialize started: unit=0 em0: link state changed to UP ale0: link state changed to UP ---Mike -------------------------------------------------------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet since 1994 www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike