From owner-freebsd-bugs@FreeBSD.ORG Mon Jan 19 02:50:03 2009 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9FCDF1065675 for ; Mon, 19 Jan 2009 02:50:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7D3888FC1C for ; Mon, 19 Jan 2009 02:50:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n0J2o3Ij047214 for ; Mon, 19 Jan 2009 02:50:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n0J2o3Pq047206; Mon, 19 Jan 2009 02:50:03 GMT (envelope-from gnats) Resent-Date: Mon, 19 Jan 2009 02:50:03 GMT Resent-Message-Id: <200901190250.n0J2o3Pq047206@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Dylan Simon Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 066161065673 for ; Mon, 19 Jan 2009 02:40:37 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id E93E98FC20 for ; Mon, 19 Jan 2009 02:40:36 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n0J2eagl040024 for ; Mon, 19 Jan 2009 02:40:36 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id n0J2eaBO040023; Mon, 19 Jan 2009 02:40:36 GMT (envelope-from nobody) Message-Id: <200901190240.n0J2eaBO040023@www.freebsd.org> Date: Mon, 19 Jan 2009 02:40:36 GMT From: Dylan Simon To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/130726: DMA errors accessing multiple SATA channels X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jan 2009 02:50:04 -0000 >Number: 130726 >Category: kern >Synopsis: DMA errors accessing multiple SATA channels >Confidential: no >Severity: serious >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Jan 19 02:50:03 UTC 2009 >Closed-Date: >Last-Modified: >Originator: Dylan Simon >Release: 8.0-CURRENT 20090114 >Organization: NYU >Environment: FreeBSD lust.cns.nyu.edu 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Wed Jan 14 19:58:58 EST 2009 dylan@lust.cns.nyu.edu:/usr/obj/usr/src/sys/SIN amd64 >Description: kernel: ad8: FAILURE - load data kernel: ad8: setting up DMA failed kernel: g_vfs_done():ad8s1e[WRITE(offset=1881014272, length=131072)]error = 5 kernel: ad6: FAILURE - load data kernel: ad6: setting up DMA failed kernel: g_vfs_done():ad6s1e[READ(offset=4117364736, length=32768)]error = 5 kernel: vnode_pager_getpages: I/O read error Disk errors of this form occur after a few minutes of disk load. It seems to occur only when operations are attempting to write to SATA disks on different channels at roughly the same time. Eventually results in panics or hangs. Occurs with GENERIC 200812 snapshot kernel too. Occurs regardless of legacy/enhanced BIOS setting. Seen with gmirror, ufs, and zfs under both nfs and local access. Does not occur at all when accessed disks are on the same channel (e.g., ad6+ad7 or ad8 alone in this case). Does not occur on same hardware with linux under similar conditions. Hardware: Supermicro C2SEA with 6 SATA ports on ICH10, four identical disks under various configurations. Symptoms matching this have also been seen on an ICH7 with two disks on different channels on hardware that works fine with 7.1, triggered by daily periodic scripts. atacontrol list: ATA channel 2: Master: acd0 ATA/ATAPI revision 7 Slave: no device present ATA channel 3: Master: ad6 Serial ATA II Slave: ad7 Serial ATA II ATA channel 4: Master: ad8 Serial ATA II Slave: ad9 Serial ATA II ATA channel 5: Master: no device present Slave: no device present ATA channel 6: Master: no device present Slave: no device present pciconf -lv (partial): pcib3@pci0:0:30:0: class=0x060401 card=0xb88015d9 chip=0x244e8086 rev=0x90 hdr=0x01 vendor = 'Intel Corporation' device = '82801 Family (ICH2/3/4/4/5/5/6/7/8/9,63xxESB) Hub Interface to PCI Bridge' class = bridge subclass = PCI-PCI isab0@pci0:0:31:0: class=0x060100 card=0xb88015d9 chip=0x3a188086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = PCI-ISA atapci1@pci0:0:31:2: class=0x01018f card=0xb88015d9 chip=0x3a208086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' class = mass storage subclass = ATA atapci2@pci0:0:31:5: class=0x010185 card=0xb88015d9 chip=0x3a268086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' class = mass storage subclass = ATA verbose dmesg (partial): lust kernel: atapci0: port 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f irq 22 at device 4.0 on pci3 lust kernel: atapci0: Reserved 0x10 bytes for rid 0x20 type 4 at 0xe400 lust kernel: ioapic0: routing intpin 22 (PCI IRQ 22) to vector 53 lust kernel: atapci0: [MPSAFE] lust kernel: atapci0: [ITHREAD] lust kernel: ata2: on atapci0 lust kernel: atapci0: Reserved 0x8 bytes for rid 0x10 type 4 at 0xec00 lust kernel: atapci0: Reserved 0x4 bytes for rid 0x14 type 4 at 0xe880 lust kernel: ata2: reset tp1 mask=03 ostat0=50 ostat1=00 lust kernel: ata2: stat0=0x00 err=0x01 lsb=0x14 msb=0xeb lust kernel: ata2: stat1=0x00 err=0x00 lsb=0x00 msb=0x00 lust kernel: ata2: reset tp2 stat0=00 stat1=00 devices=0x10000 lust kernel: ata2: [MPSAFE] lust kernel: ata2: [ITHREAD] lust kernel: pci3: at device 8.0 (no driver attached) lust kernel: isab0: at device 31.0 on pci0 lust kernel: isa0: on isab0 lust kernel: atapci1: port 0xc400-0xc407,0xc080-0xc083,0xc000-0xc007,0xbc00-0xbc03,0xb880-0xb88f,0xb800-0xb80f irq 19 at device 31.2 on pci0 lust kernel: atapci1: Reserved 0x10 bytes for rid 0x20 type 4 at 0xb880 lust kernel: atapci1: [MPSAFE] lust kernel: atapci1: [ITHREAD] lust kernel: atapci1: Reserved 0x10 bytes for rid 0x24 type 4 at 0xb800 lust kernel: ata3: on atapci1 lust kernel: atapci1: Reserved 0x8 bytes for rid 0x10 type 4 at 0xc400 lust kernel: atapci1: Reserved 0x4 bytes for rid 0x14 type 4 at 0xc080 lust kernel: ata3: reset tp1 mask=03 ostat0=50 ostat1=50 lust kernel: ata3: stat0=0x50 err=0x01 lsb=0x00 msb=0x00 lust kernel: ata3: stat1=0x50 err=0x01 lsb=0x00 msb=0x00 lust kernel: ata3: reset tp2 stat0=50 stat1=50 devices=0x3 lust kernel: ata3: [MPSAFE] lust kernel: ata3: [ITHREAD] lust kernel: ata4: on atapci1 lust kernel: atapci1: Reserved 0x8 bytes for rid 0x18 type 4 at 0xc000 lust kernel: atapci1: Reserved 0x4 bytes for rid 0x1c type 4 at 0xbc00 lust kernel: ata4: reset tp1 mask=03 ostat0=50 ostat1=50 lust kernel: ata4: stat0=0x50 err=0x01 lsb=0x00 msb=0x00 lust kernel: ata4: stat1=0x50 err=0x01 lsb=0x00 msb=0x00 lust kernel: ata4: reset tp2 stat0=50 stat1=50 devices=0x3 lust kernel: ata4: [MPSAFE] lust kernel: ata4: [ITHREAD] lust kernel: pci0: at device 31.3 (no driver attached) lust kernel: atapci2: port 0xb400-0xb407,0xb080-0xb083,0xb000-0xb007,0xac00-0xac03,0xa880-0xa88f,0xa800-0xa80f irq 19 at device 31.5 on pci0 lust kernel: atapci2: Reserved 0x10 bytes for rid 0x20 type 4 at 0xa880 lust kernel: atapci2: [MPSAFE] lust kernel: atapci2: [ITHREAD] lust kernel: atapci2: Reserved 0x10 bytes for rid 0x24 type 4 at 0xa800 lust kernel: ata5: on atapci2 lust kernel: atapci2: Reserved 0x8 bytes for rid 0x10 type 4 at 0xb400 lust kernel: atapci2: Reserved 0x4 bytes for rid 0x14 type 4 at 0xb080 lust kernel: ata5: reset tp1 mask=03 ostat0=7f ostat1=7f lust kernel: ata5: stat0=0x7f err=0xff lsb=0xff msb=0xff lust kernel: ata5: stat1=0x7f err=0xff lsb=0xff msb=0xff lust kernel: ata5: reset tp2 stat0=ff stat1=ff devices=0x0 lust kernel: ata5: [MPSAFE] lust kernel: ata5: [ITHREAD] lust kernel: ata6: on atapci2 lust kernel: atapci2: Reserved 0x8 bytes for rid 0x18 type 4 at 0xb000 lust kernel: atapci2: Reserved 0x4 bytes for rid 0x1c type 4 at 0xac00 lust kernel: ata6: reset tp1 mask=03 ostat0=7f ostat1=7f lust kernel: ata6: stat0=0x7f err=0xff lsb=0xff msb=0xff lust kernel: ata6: stat1=0x7f err=0xff lsb=0xff msb=0xff lust kernel: ata6: reset tp2 stat0=ff stat1=ff devices=0x0 lust kernel: ata6: [MPSAFE] lust kernel: ata6: [ITHREAD] lust kernel: ata2: identify ch->devices=00010000 lust kernel: ata2-master: pio=PIO4 wdma=WDMA2 udma=UDMA33 cable=40 wire lust kernel: acd0: setting PIO4 on IT8213F chip lust kernel: acd0: setting UDMA33 on IT8213F chip lust kernel: acd0: DVDROM drive at ata2 as master lust kernel: acd0: read 8268KB/s (8268KB/s), 198KB buffer, UDMA33 lust kernel: acd0: Reads: CDR, CDRW, CDDA stream, DVDROM, DVDR, DVDRAM, packet lust kernel: acd0: Writes: lust kernel: acd0: Audio: play, 256 volume levels lust kernel: acd0: Mechanism: ejectable tray, unlocked lust kernel: acd0: Medium: no/blank disc lust kernel: ata3: identify ch->devices=00000003 lust kernel: ata3-master: pio=PIO4 wdma=WDMA2 udma=UDMA133 cable=40 wire lust kernel: ata3-slave: pio=PIO4 wdma=WDMA2 udma=UDMA133 cable=40 wire lust kernel: ad6: 953869MB at ata3-master SATA300 lust kernel: ad6: 1953525168 sectors [1938021C/16H/63S] 16 sectors/interrupt 1 depth queue lust kernel: GEOM: new disk ad6 lust kernel: ad7: 953869MB at ata3-slave SATA300 lust kernel: ad7: 1953525168 sectors [1938021C/16H/63S] 16 sectors/interrupt 1 depth queue lust kernel: ata4: identify ch->devices=00000003 lust kernel: ata4-master: pio=PIO4 wdma=WDMA2 udma=UDMA133 cable=40 wire lust kernel: GEOM: new disk ad7 lust kernel: ata4-slave: pio=PIO4 wdma=WDMA2 udma=UDMA133 cable=40 wire lust kernel: ad8: 953869MB at ata4-master SATA300 lust kernel: ad8: 1953525168 sectors [1938021C/16H/63S] 16 sectors/interrupt 1 depth queue lust kernel: GEOM: new disk ad8 lust kernel: ad9: 953869MB at ata4-slave SATA300 lust kernel: ad9: 1953525168 sectors [1938021C/16H/63S] 16 sectors/interrupt 1 depth queue lust kernel: ata5: identify ch->devices=00000000 lust kernel: ata6: identify ch->devices=00000000 lust kernel: ioapic0: Assigning ISA IRQ 1 to local APIC 0 lust kernel: ioapic0: Assigning ISA IRQ 9 to local APIC 1 lust kernel: ioapic0: Assigning PCI IRQ 17 to local APIC 0 lust kernel: ioapic0: Assigning PCI IRQ 18 to local APIC 1 lust kernel: ioapic0: Assigning PCI IRQ 19 to local APIC 0 lust kernel: ioapic0: Assigning PCI IRQ 22 to local APIC 1 lust kernel: ioapic0: Assigning PCI IRQ 23 to local APIC 0 lust kernel: GEOM: new disk ad9 >How-To-Repeat: Perform access involving writes to SATA disks on different channels of an ICH controller. >Fix: >Release-Note: >Audit-Trail: >Unformatted: