From owner-freebsd-stable@FreeBSD.ORG Tue Dec 22 12:16:47 2009 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB5A21065697 for ; Tue, 22 Dec 2009 12:16:47 +0000 (UTC) (envelope-from petefrench@ticketswitch.com) Received: from constantine.ticketswitch.com (constantine.ticketswitch.com [IPv6:2002:57e0:1d4e:1::3]) by mx1.freebsd.org (Postfix) with ESMTP id 7DD558FC2D for ; Tue, 22 Dec 2009 12:16:45 +0000 (UTC) Received: from dilbert.rattatosk ([10.64.50.6] helo=dilbert.ticketswitch.com) by constantine.ticketswitch.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1NN3fP-000Ckx-Vb for freebsd-stable@FreeBSD.org; Tue, 22 Dec 2009 12:16:44 +0000 Received: from petefrench by dilbert.ticketswitch.com with local (Exim 4.70 (FreeBSD)) (envelope-from ) id 1NN3fP-00048P-Tb for freebsd-stable@FreeBSD.org; Tue, 22 Dec 2009 12:16:43 +0000 Date: Tue, 22 Dec 2009 12:16:43 +0000 Message-Id: To: freebsd-stable@FreeBSD.org From: Pete French Cc: Subject: Disc lock up on 8.0-STABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Dec 2009 12:16:47 -0000 I've been gradually testing 8.0 on several machines propr to deploying it live, but I currently have a machine which appears to lock-up at 3am every day. The symptoms are that the machine is still pingable, but doing anything which requires access to the disc just freezes (so you cant login for example). I've seen simiilar behaviour behore on machines when the disc syste has locked up for some reason, so am ttentatively guessing that this is the cause. The machine is an HP DL360 G5 with a ciss0 controller for the drives. I have upgraded to the latest STABLE but the freeze still happens. Am including a dmesg below, and will compile it with KDB, DDB to see what happens. The machine is booting from a UFS partition, but is using ZFS for everything else. The fcat it deadlocks at 3am makes me thing this is something to do with scheduled jobs maybe ? Then again, I have an almost identical DL360 which is running 8.0 and is rock solid. -pete. Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-STABLE #0: Mon Dec 21 15:42:31 GMT 2009 webadmin@florentine.rattatosk:/usr/obj/usr/src/sys/GENERIC amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz (2333.43-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6fb Stepping = 11 Features=0xbfebfbff Features2=0x4e3bd AMD Features=0x20000800 AMD Features2=0x1 TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 4104138752 (3914 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 20090521 tbfadt-707 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 pcib0: on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 ACPI Warning: \\_SB_.PCI0.PT02._PRT: Return Package has no elements (empty) 20090521 nspredef-545 pci9: on pcib1 pcib2: at device 0.0 on pci9 pci10: on pcib2 pcib3: at device 0.0 on pci10 pci11: on pcib3 pcib4: at device 1.0 on pci10 pci14: on pcib4 pcib5: at device 2.0 on pci10 pci15: on pcib5 pcib6: at device 0.3 on pci9 pci16: on pcib6 pcib7: at device 3.0 on pci0 pci6: on pcib7 pcib8: at device 0.0 on pci6 pci7: on pcib8 pcib9: at device 4.0 on pci7 pci8: on pcib9 ciss0: port 0x4000-0x40ff mem 0xfde80000-0xfdefffff,0xfde70000-0xfde77fff irq 16 at device 8.0 on pci7 ciss0: PERFORMANT Transport ciss0: got 2 MSI messages] ciss0: [ITHREAD] pcib10: at device 4.0 on pci0 pci19: on pcib10 pcib11: at device 5.0 on pci0 pci22: on pcib11 pcib12: at device 6.0 on pci0 pci2: on pcib12 pcib13: at device 0.0 on pci2 pci3: on pcib13 bce0: mem 0xf8000000-0xf9ffffff irq 18 at device 0.0 on pci3 miibus0: on bce0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bce0: Ethernet address: 00:1e:0b:5f:1f:76 bce0: [ITHREAD] bce0: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (1.9.6); Flags (MSI|MFW); MFW () pcib14: at device 7.0 on pci0 pci4: on pcib14 pcib15: at device 0.0 on pci4 pci5: on pcib15 bce1: mem 0xfa000000-0xfbffffff irq 19 at device 0.0 on pci5 miibus1: on bce1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bce1: Ethernet address: 00:1e:0b:5f:fd:d8 bce1: [ITHREAD] bce1: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (1.9.6); Flags (MSI|MFW); MFW () uhci0: port 0x1000-0x101f irq 16 at device 29.0 on pci0 uhci0: [ITHREAD] usbus0: on uhci0 uhci1: port 0x1020-0x103f irq 17 at device 29.1 on pci0 uhci1: [ITHREAD] usbus1: on uhci1 uhci2: port 0x1040-0x105f irq 18 at device 29.2 on pci0 uhci2: [ITHREAD] usbus2: on uhci2 uhci3: port 0x1060-0x107f irq 19 at device 29.3 on pci0 uhci3: [ITHREAD] usbus3: on uhci3 ehci0: mem 0xf7df0000-0xf7df03ff irq 16 at device 29.7 on pci0 ehci0: [ITHREAD] usbus4: EHCI version 1.0 usbus4: on ehci0 pcib16: at device 30.0 on pci0 pci1: on pcib16 vgapci0: port 0x3000-0x30ff mem 0xd8000000-0xdfffffff,0xf7ff0000-0xf7ffffff irq 23 at device 3.0 on pci1 pci1: at device 4.0 (no driver attached) pci1: at device 4.2 (no driver attached) uhci4: port 0x3800-0x381f irq 22 at device 4.4 on pci1 uhci4: [ITHREAD] usbus5: on uhci4 pci1: at device 4.6 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x500-0x50f irq 17 at device 31.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] acpi_tz0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] cpu0: on acpi0 est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 725072506000725 device_attach: est0 attach returned 6 p4tcc0: on cpu0 cpu1: on acpi0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 725072506000725 device_attach: est1 attach returned 6 p4tcc1: on cpu1 cpu2: on acpi0 est2: on cpu2 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 725072506000725 device_attach: est2 attach returned 6 p4tcc2: on cpu2 cpu3: on acpi0 est3: on cpu3 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 725072506000725 device_attach: est3 attach returned 6 p4tcc3: on cpu3 orm0: at iomem 0xc0000-0xcafff,0xe6000-0xe7fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atrtc0: at port 0x70 irq 8 on isa0 ppc0: cannot reserve I/O port range uart1: at port 0x2f8-0x2ff irq 3 on isa0 uart1: [FILTER] ZFS filesystem version 13 ZFS storage pool version 13 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 480Mbps High Speed USB v2.0 usbus5: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: <(0x103c)> at usbus5 uhub5: <(0x103c) UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered ugen5.2: at usbus5 ukbd0: on usbus5 kbd2 at ukbd0 ums0: on usbus5 ugen5.3: at usbus5 uhub6: on usbus5 uhub4: 8 ports with 8 removable, self powered uhub6: 7 ports with 7 removable, self powered da0 at ciss0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 135.168MB/s transfers da0: Command Queueing enabled da0: 69970MB (143299800 512 byte sectors: 255H 63S/T 8920C) SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! Trying to mount root from ufs:/dev/da0s1a WARNING: / was not properly dismounted Setting hostuuid: 33393935-3234-5553-4538-30344e37364e. Setting hostid: 0xa1c8b883. Entropy harvesting: interrupts ethernet point_to_point kickstart . Starting file system checks: /dev/da0s1a: UNREF FILE I=308432 OWNER=root MODE=100644 /dev/da0s1a: SIZE=0 MTIME=Dec 21 18:07 2009 (CLEARED) /dev/da0s1a: UNREF FILE I=308546 OWNER=root MODE=140666 /dev/da0s1a: SIZE=0 MTIME=Dec 21 18:06 2009 (CLEARED) /dev/da0s1a: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) /dev/da0s1a: SUMMARY INFORMATION BAD (SALVAGED) /dev/da0s1a: BLK(S) MISSING IN BIT MAPS (SALVAGED) /dev/da0s1a: 24964 files, 642961 used, 1386070 free (32630 frags, 169180 blocks, 1.6% fragmentation) Mounting local file systems: . Setting hostname: florentine.rattatosk . Starting Network: lo0 bce0 bce1 lagg0. lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 nd6 options=3 bce0: flags=8843 metric 0 mtu 1500 options=1bb ether 00:1e:0b:5f:1f:76 media: Ethernet autoselect (none) status: no carrier bce1: flags=8843 metric 0 mtu 1500 options=1bb ether 00:1e:0b:5f:1f:76 media: Ethernet autoselect (none) status: no carrier lagg0: flags=8843 metric 0 mtu 1500 options=1bb ether 00:1e:0b:5f:1f:76 inet 10.48.19.0 netmask 0xffff0000 broadcast 10.48.255.255 inet 10.48.19.229 netmask 0xffff0000 broadcast 10.48.255.255 inet 10.48.19.223 netmask 0xffff0000 broadcast 10.48.255.255 inet 10.48.19.226 netmask 0xffff0000 broadcast 10.48.255.255 inet 10.48.19.224 netmask 0xffff0000 broadcast 10.48.255.255 inet 10.48.19.228 netmask 0xffff0000 broadcast 10.48.255.255 inet 10.48.19.227 netmask 0xffff0000 broadcast 10.48.255.255 inet 10.48.19.239 netmask 0xffff0000 broadcast 10.48.255.255 inet 10.48.19.230 netmask 0xffff0000 broadcast 10.48.255.255 inet 10.48.19.232 netmask 0xffff0000 broadcast 10.48.255.255 inet 10.48.19.235 netmask 0xffff0000 broadcast 10.48.255.255 inet 10.48.19.245 netmask 0xffff0000 broadcast 10.48.255.255 media: Ethernet autoselect status: no carrier laggproto lacp laggport: bce1 flags=0<> laggport: bce0 flags=0<> add net default: gateway 10.48.0.9 Starting devd. Starting ums0 moused . Creating and/or trimming log files . Starting syslogd. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat /usr/local/lib/mysql 32-bit compatibility ldconfig path: /usr/lib32 /usr/local/lib32/compat Starting tomcat60. Starting named. Dec 22 12:06:32 florentine named[827]: the working directory is not writable Starting rpcbind. NFS access cache time=60 Clearing /tmp (X related). bce0: link state changed to UP lagg0: link state changed to UP Starting mountd. Starting nfsd. Updating motd: . Starting ntpd. bce1: link state changed to UP Starting dhcpd. Starting pdns. Dec 22 12:06:33 Reading random entropy from '/dev/urandom' Dec 22 12:06:33 florentine pdns[1211]: UDP server bound to 0.0.0.0:53 Dec 22 12:06:33 florentine pdns[1211]: UDPv6 server bound to [::]:53 Dec 22 12:06:33 florentine pdns[1211]: TCP server bound to 0.0.0.0:53 Dec 22 12:06:33 florentine pdns[1211]: TCPv6 server bound to [::]:53 Dec 22 12:06:33 florentine pdns[1211]: DNS Proxy launched, local port 26807, remote 127.0.0.1:5300 Dec 22 12:06:33 florentine pdns[1211]: Creating backend connection for TCP Starting mysql. Starting exim. Dec 22 12:06:33 florentine pdns[1211]: Backend launched with banner: OK TicketSwitch Mode 1 Performing sanity check on apache22 configuration: Syntax OK Starting apache22. Configuring syscons: blanktime . Starting sshd. Starting cron. Starting inetd. Starting background file system checks in 60 seconds.