From owner-freebsd-bugs@FreeBSD.ORG Tue Oct 29 00:00:01 2013 Return-Path: Delivered-To: freebsd-bugs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 98641E2A for ; Tue, 29 Oct 2013 00:00:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 76FC229C5 for ; Tue, 29 Oct 2013 00:00:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r9T001qd035038 for ; Tue, 29 Oct 2013 00:00:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r9T001cf035014; Tue, 29 Oct 2013 00:00:01 GMT (envelope-from gnats) Resent-Date: Tue, 29 Oct 2013 00:00:01 GMT Resent-Message-Id: <201310290000.r9T001cf035014@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Shawn Wallbridge Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 3BE01DF2 for ; Mon, 28 Oct 2013 23:57:51 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from oldred.freebsd.org (oldred.freebsd.org [8.8.178.121]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 26A9729AB for ; Mon, 28 Oct 2013 23:57:51 +0000 (UTC) Received: from oldred.freebsd.org ([127.0.1.6]) by oldred.freebsd.org (8.14.5/8.14.7) with ESMTP id r9SNvoVN071929 for ; Mon, 28 Oct 2013 23:57:50 GMT (envelope-from nobody@oldred.freebsd.org) Received: (from nobody@localhost) by oldred.freebsd.org (8.14.5/8.14.5/Submit) id r9SNvo7P071928; Mon, 28 Oct 2013 23:57:50 GMT (envelope-from nobody) Message-Id: <201310282357.r9SNvo7P071928@oldred.freebsd.org> Date: Mon, 28 Oct 2013 23:57:50 GMT From: Shawn Wallbridge To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Subject: kern/183424: Kernel Panic, dump says "handle_workitem_remove: bad file delta" X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Oct 2013 00:00:01 -0000 >Number: 183424 >Category: kern >Synopsis: Kernel Panic, dump says "handle_workitem_remove: bad file delta" >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Oct 29 00:00:00 UTC 2013 >Closed-Date: >Last-Modified: >Originator: Shawn Wallbridge >Release: FreeBSD 9.2-RELEASE #0 r255898 >Organization: Imaginary Forces >Environment: FreeBSD mercury 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013 root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 >Description: Machine has kernel panic'ed around 6 or 7 times now. Each time with the same Panic. Panic String: handle_workitem_remove: bad file delta I have 6 core dumps to work with. It's a pretty plain file server, NFS and Samba (mostly NFS). No fancy config, just 72TB off of a 3Ware controller. dmesg: root@mercury:/var/crash # dmesg Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013 root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 gcc version 4.2.1 20070831 patched [FreeBSD] CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2400.13-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206c2 Family = 0x6 Model = 0x2c Stepping = 2 Features=0xbfebfbff Features2=0x9ee3fd AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 34363932672 (32772 MB) avail memory = 33145389056 (31609 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <123011 APIC1930> FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 18 cpu5 (AP): APIC ID: 19 cpu6 (AP): APIC ID: 20 cpu7 (AP): APIC ID: 21 cpu8 (AP): APIC ID: 32 cpu9 (AP): APIC ID: 33 cpu10 (AP): APIC ID: 34 cpu11 (AP): APIC ID: 35 cpu12 (AP): APIC ID: 50 cpu13 (AP): APIC ID: 51 cpu14 (AP): APIC ID: 52 cpu15 (AP): APIC ID: 53 ioapic0: Changing APIC ID to 6 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) acpi0: reservation of 400, 100 (3) failed cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 450 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 Event timer "HPET3" frequency 14318180 Hz quality 440 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff iomem 0xfed40000-0xfed44fff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 igb0: port 0xcc00-0xcc1f mem 0xfbbe0000-0xfbbfffff,0xfbbc0000-0xfbbdffff,0xfbb9c000-0xfbb9ffff irq 16 at device 0.0 on pci1 igb0: Using MSIX interrupts with 9 vectors igb0: Ethernet address: 00:25:90:92:a4:26 igb0: Bound queue 0 to cpu 0 igb0: Bound queue 1 to cpu 1 igb0: Bound queue 2 to cpu 2 igb0: Bound queue 3 to cpu 3 igb0: Bound queue 4 to cpu 4 igb0: Bound queue 5 to cpu 5 igb0: Bound queue 6 to cpu 6 igb0: Bound queue 7 to cpu 7 igb1: port 0xc880-0xc89f mem 0xfbb60000-0xfbb7ffff,0xfbb40000-0xfbb5ffff,0xfbb1c000-0xfbb1ffff irq 17 at device 0.1 on pci1 igb1: Using MSIX interrupts with 9 vectors igb1: Ethernet address: 00:25:90:92:a4:27 igb1: Bound queue 0 to cpu 8 igb1: Bound queue 1 to cpu 9 igb1: Bound queue 2 to cpu 10 igb1: Bound queue 3 to cpu 11 igb1: Bound queue 4 to cpu 12 igb1: Bound queue 5 to cpu 13 igb1: Bound queue 6 to cpu 14 igb1: Bound queue 7 to cpu 15 pcib2: at device 3.0 on pci0 pci2: on pcib2 pcib3: at device 7.0 on pci0 pci3: on pcib3 pcib4: at device 9.0 on pci0 pci4: on pcib4 LSI 3ware device driver for SAS/SATA storage controllers, version: 10.80.00.005 tws0: port 0xd800-0xd8ff mem 0xfbc9c000-0xfbc9ffff,0xfbcc0000-0xfbcfffff irq 16 at device 0.0 on pci4 tws0: Using legacy INTx tws0: Controller details: Model 9750-24i4e, 28 Phys, Firmware FH9X 5.12.00.007, BIOS BE9X 5.11.00.006 pci0: at device 20.0 (no driver attached) pci0: at device 22.0 (no driver attached) pci0: at device 22.1 (no driver attached) pci0: at device 22.2 (no driver attached) pci0: at device 22.3 (no driver attached) pci0: at device 22.4 (no driver attached) pci0: at device 22.5 (no driver attached) pci0: at device 22.6 (no driver attached) pci0: at device 22.7 (no driver attached) uhci0: port 0xac00-0xac1f irq 16 at device 26.0 on pci0 uhci0: LegSup = 0x2f00 usbus0 on uhci0 uhci1: port 0xa880-0xa89f irq 21 at device 26.1 on pci0 uhci1: LegSup = 0x2f00 usbus1 on uhci1 uhci2: port 0xa800-0xa81f irq 19 at device 26.2 on pci0 uhci2: LegSup = 0x2f00 usbus2 on uhci2 ehci0: mem 0xfbeda000-0xfbeda3ff irq 18 at device 26.7 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci0 hdac0: mem 0xfbed4000-0xfbed7fff irq 22 at device 27.0 on pci0 pcib5: irq 17 at device 28.0 on pci0 pci5: on pcib5 atapci0: port 0xe400-0xe407,0xec00-0xec03,0xe880-0xe887,0xe800-0xe803,0xe480-0xe48f irq 16 at device 0.0 on pci5 ata2: at channel 0 on atapci0 uhci3: port 0xa480-0xa49f irq 23 at device 29.0 on pci0 uhci3: LegSup = 0x2f00 usbus4 on uhci3 uhci4: port 0xa400-0xa41f irq 19 at device 29.1 on pci0 uhci4: LegSup = 0x2f00 usbus5 on uhci4 uhci5: port 0xa080-0xa09f irq 18 at device 29.2 on pci0 uhci5: LegSup = 0x2f00 usbus6 on uhci5 ehci1: mem 0xfbed8000-0xfbed83ff irq 23 at device 29.7 on pci0 usbus7: EHCI version 1.0 usbus7 on ehci1 pcib6: at device 30.0 on pci0 pci6: on pcib6 vgapci0: mem 0xf9000000-0xf9ffffff,0xfaffc000-0xfaffffff,0xfb000000-0xfb7fffff irq 16 at device 4.0 on pci6 pci6: at device 5.0 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 ahci0: port 0x9480-0x9487,0xa000-0xa003,0x9c00-0x9c07,0x9880-0x9883,0x9800-0x981f mem 0xfbed2000-0xfbed27ff irq 19 at device 31.2 on pci0 ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahcich4: at channel 4 on ahci0 ahcich5: at channel 5 on ahci0 pci0: at device 31.3 (no driver attached) pcib7: on acpi0 pci128: on pcib7 pcib8: at device 0.0 on pci128 pci129: on pcib8 pcib9: at device 1.0 on pci128 pci130: on pcib9 pcib10: at device 3.0 on pci128 pci131: on pcib10 pcib11: at device 5.0 on pci128 pci132: on pcib11 pcib12: at device 7.0 on pci128 pci133: on pcib12 pcib13: at device 0.0 on pci133 pci134: on pcib13 pcib14: at device 2.0 on pci134 pci135: on pcib14 mxge0: mem 0xf6000000-0xf6ffffff,0xf8d00000-0xf8dfffff irq 18 at device 0.0 on pci135 mxge0: Ethernet address: 00:60:dd:44:aa:da pcib15: at device 4.0 on pci134 pci136: on pcib15 mxge1: mem 0xf7000000-0xf7ffffff,0xf8f00000-0xf8ffffff irq 16 at device 0.0 on pci136 mxge1: Ethernet address: 00:60:dd:44:aa:db pci128: at device 20.0 (no driver attached) pci128: at device 20.1 (no driver attached) pci128: at device 20.2 (no driver attached) pci128: at device 20.3 (no driver attached) pci128: at device 22.0 (no driver attached) pci128: at device 22.1 (no driver attached) pci128: at device 22.2 (no driver attached) pci128: at device 22.3 (no driver attached) pci128: at device 22.4 (no driver attached) pci128: at device 22.5 (no driver attached) pci128: at device 22.6 (no driver attached) pci128: at device 22.7 (no driver attached) acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart2: <16550 or compatible> port 0x3e8-0x3ef irq 5 on acpi0 qpi0: on motherboard pcib16: pcibus 255 on qpi0 pci255: on pcib16 pcib17: pcibus 254 on qpi0 pci254: on pcib17 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcafff,0xd8800-0xda7ff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: cannot reserve I/O port range est0: on cpu0 p4tcc0: on cpu0 est1: on cpu1 p4tcc1: on cpu1 est2: on cpu2 p4tcc2: on cpu2 est3: on cpu3 p4tcc3: on cpu3 est4: on cpu4 p4tcc4: on cpu4 est5: on cpu5 p4tcc5: on cpu5 est6: on cpu6 p4tcc6: on cpu6 est7: on cpu7 p4tcc7: on cpu7 est8: on cpu8 p4tcc8: on cpu8 est9: on cpu9 p4tcc9: on cpu9 est10: on cpu10 p4tcc10: on cpu10 est11: on cpu11 p4tcc11: on cpu11 est12: on cpu12 p4tcc12: on cpu12 est13: on cpu13 p4tcc13: on cpu13 est14: on cpu14 p4tcc14: on cpu14 est15: on cpu15 p4tcc15: on cpu15 Timecounters tick every 1.000 msec hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 20,22,21,23 and 24,26 on hdaa0 pcm1: at nid 27 and 25 on hdaa0 usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 12Mbps Full Speed USB v1.0 usbus7: 480Mbps High Speed USB v2.0 (probe65:tws0:0:65:0): INQUIRY. CDB: 12 00 00 00 24 00 (probe65:tws0:0:65:0): CAM status: Invalid Target ID (probe65:tws0:0:65:0): Error 22, Unretryable error ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 ugen7.1: at usbus7 uhub7: on usbus7 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub6: 2 ports with 2 removable, self powered da0 at tws0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 6000.000MB/s transfers da0: 76293740MB (156249579520 512 byte sectors: 255H 63S/T 9726086C) ada0 at ahcich0 bus 0 scbus2 target 0 lun 0 ada0: ATA-8 SATA 2.x device cd0 at ahcich2 bus 0 scbus4 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 150.000MB/s transfers (SATA 1.x, UDMA6, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray open ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) ada0: quirks=0x1<4K> ada0: Previously was known as ad6 ada1 at ahcich1 bus 0 scbus3 target 0 lun 0 ada1: ATA-8 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) ada1: quirks=0x1<4K> ada1: Previously was known as ad8 SMP: AP CPU #1 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #12 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #15 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #13 Launched! SMP: AP CPU #11 Launched! Timecounter "TSC-low" frequency 1200067010 Hz quality 1000 GEOM_RAID: Intel-8a1cfe11: Array Intel-8a1cfe11 created. GEOM_RAID: Intel-8a1cfe11: Disk ada0 state changed from NONE to ACTIVE. GEOM_RAID: Intel-8a1cfe11: Subdisk Volume0:0-ada0 state changed from NONE to ACTIVE. GEOM_RAID: Intel-8a1cfe11: Disk ada1 state changed from NONE to ACTIVE. GEOM_RAID: Intel-8a1cfe11: Subdisk Volume0:1-ada1 state changed from NONE to ACTIVE. GEOM_RAID: Intel-8a1cfe11: Array started. GEOM_RAID: Intel-8a1cfe11: Volume Volume0 state changed from STARTING to OPTIMAL. GEOM_RAID: Intel-8a1cfe11: Provider raid/r0 for volume Volume0 created. Root mount waiting for: usbus7 usbus3 uhub3: 6 ports with 6 removable, self powered uhub7: 6 ports with 6 removable, self powered Root mount waiting for: usbus7 usbus3 Trying to mount root from ufs:/dev/raid/r0p2 [rw]... ugen5.2: at usbus5 ukbd0: on usbus5 kbd2 at ukbd0 ugen2.2: at usbus2 ukbd2: on usbus2 kbd3 at ukbd2 ums0: on usbus5 ums1: on usbus2 ums0: 5 buttons and [XYZ] coordinates ID=1 >How-To-Repeat: Just happens, no way to induce the issue. I haven't seen any increase in load or traffic that would cause the issue. >Fix: >Release-Note: >Audit-Trail: >Unformatted: