From owner-freebsd-questions@FreeBSD.ORG Mon May 10 08:52:22 2004 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BC12816A4CE for ; Mon, 10 May 2004 08:52:22 -0700 (PDT) Received: from atwns1.omniresources.com (atwns1.omniresources.com [69.48.112.132]) by mx1.FreeBSD.org (Postfix) with ESMTP id DD4FC43D55 for ; Mon, 10 May 2004 08:52:20 -0700 (PDT) (envelope-from dpoland@omniresources.com) Received: from 5x2822.omniresources.com (5x2822.omniresources.com [192.168.254.149])i4AFqKQc094807 for ; Mon, 10 May 2004 10:52:20 -0500 (CDT) (envelope-from dpoland@atwdev.omniresources.com) Received: from 5x2822.omniresources.com (localhost [127.0.0.1]) i4AFqJ3n037521 for ; Mon, 10 May 2004 10:52:19 -0500 (CDT) (envelope-from dpoland@5x2822.omniresources.com) Received: (from dpoland@localhost) by 5x2822.omniresources.com (8.12.10/8.12.10/Submit) id i4AFqJCD037520 for questions@freebsd.org; Mon, 10 May 2004 10:52:19 -0500 (CDT) (envelope-from dpoland) Date: Mon, 10 May 2004 10:52:19 -0500 From: Doug Poland To: questions@freebsd.org Message-ID: <20040510155158.GA37371@omniresources.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.1i Subject: Need help diagnosing hardware failure X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 May 2004 15:52:22 -0000 Hello, Upon returning from a weeks vacation, I was dismayed to find my home file server (running 4.8-STABLE) had crashed. The box in question has an Adaptec Host adapter ahc0: port 0xf800-0xf8ff mem 0xfedfe000-0xfedfefff irq 10 at device 13.0 on pci0 aic7860: Ultra Single Channel A, SCSI Id=7, 3/253 SCBs and seven identical SCSI drives judeah# dmesg | grep IBMRAID da0: Fixed Direct Access SCSI-2 device da1: Fixed Direct Access SCSI-2 device da2: Fixed Direct Access SCSI-2 device da3: Fixed Direct Access SCSI-2 device da4: Fixed Direct Access SCSI-2 device da6: Fixed Direct Access SCSI-2 device da5: Fixed Direct Access SCSI-2 device in a vinum stipped volume... judeah# more /etc/vinum.conf drive a device /dev/da0e drive b device /dev/da1e drive c device /dev/da2e drive d device /dev/da3e drive e device /dev/da4e drive f device /dev/da5e drive g device /dev/da6e volume dataraid plex org striped 256k sd length 1920m drive a sd length 1920m drive b sd length 1920m drive c sd length 1920m drive d sd length 1920m drive e sd length 1920m drive f sd length 1920m drive g Perusal of /var/log/messages show... May 3 11:17:31 judeah /kernel: (da1:ahc0:0:1:0): SCB 0x5a - timed out May 3 11:17:31 judeah /kernel: >>>>>>>>>>>>>>>>>> Dump Card State Begins <<<<<<<<<<<<<<<<< May 3 11:17:31 judeah /kernel: ahc0: Dumping Card State while idle, at SEQADDR 0x7 May 3 11:17:31 judeah /kernel: Card was paused May 3 11:17:31 judeah /kernel: ACCUM = 0x97, SINDEX = 0x52, DINDEX = 0x8c, ARG_2 = 0x0 May 3 11:17:31 judeah /kernel: HCNT = 0x0 SCBPTR = 0x1 May 3 11:17:31 judeah /kernel: SCSISIGI[0x0] ERROR[0x40] SCSIBUSL[0x0] LASTPHASE[0x1] May 3 11:17:31 judeah /kernel: SCSISEQ[0x12] SBLKCTL[0x0] SCSIRATE[0x0] SEQCTL[0x10] May 3 11:17:31 judeah /kernel: SEQ_FLAGS[0xc0] SSTAT0[0x5] SSTAT1[0xa] SSTAT2[0x0] May 3 11:17:31 judeah /kernel: SSTAT3[0x0] SIMODE0[0x0] SIMODE1[0xa4] SXFRCTL0[0x80] May 3 11:17:31 judeah /kernel: DFCNTRL[0x0] DFSTATUS[0x29] May 3 11:17:31 judeah /kernel: STACK: 0x0 0x166 0x109 0x3 May 3 11:17:31 judeah /kernel: SCB count = 130 May 3 11:17:31 judeah /kernel: Kernel NEXTQSCB = 30 May 3 11:17:31 judeah /kernel: Card NEXTQSCB = 30 May 3 11:17:31 judeah /kernel: QINFIFO entries: May 3 11:17:31 judeah /kernel: Waiting Queue entries: May 3 11:17:31 judeah /kernel: Disconnected Queue entries: 2:90 May 3 11:17:31 judeah /kernel: QOUTFIFO entries: May 3 11:17:31 judeah /kernel: Sequencer Free SCB List: 1 0 May 3 11:17:31 judeah /kernel: Sequencer SCB Info: May 3 11:17:31 judeah /kernel: 0 SCB_CONTROL[0xe2] SCB_SCSIID[0x67] SCB_LUN[0x0] SCB_TAG[0xff] May 3 11:17:31 judeah /kernel: 1 SCB_CONTROL[0xe2] SCB_SCSIID[0x67] SCB_LUN[0x0] SCB_TAG[0xff] May 3 11:17:31 judeah /kernel: 2 SCB_CONTROL[0x66] SCB_SCSIID[0x17] SCB_LUN[0x0] SCB_TAG[0x5a] May 3 11:17:31 judeah /kernel: Pending list: May 3 11:17:31 judeah /kernel: 90 SCB_CONTROL[0x62] SCB_SCSIID[0x17] SCB_LUN[0x0] May 3 11:17:31 judeah /kernel: Kernel Free SCB list: 82 88 14 115 12 83 120 92 45 8 16 5 59 124 31 29 38 18 73 42 93 64 19 7 74 100 113 75 24 3 86 71 20 108 6 67 68 125 105 97 110 34 54 87 106 25 61 109 123 47 44 66 53 94 84 76 65 77 72 9 69 32 17 55 119 1 22 91 4 112 56 27 102 62 13 15 128 50 33 51 81 37 57 28 99 117 85 36 41 11 121 49 0 80 35 39 40 95 26 96 10 58 118 122 127 111 2 126 70 98 89 21 60 46 48 78 43 101 23 79 52 63 129 103 104 107 116 114 May 3 11:17:31 judeah /kernel: May 3 11:17:31 judeah /kernel: <<<<<<<<<<<<<<<< Dump Card State Ends >>>>>>>>>>>>>>>>>> The box rebooted and failed to come up to it's normal state because the the vinum volume that was running off this SCSI disk system failed to load. May 3 11:22:01 judeah /kernel: sg[0] - Addr 0x1ddd000 : Length 4096 May 3 11:22:01 judeah /kernel: sg[1] - Addr 0x7be000 : Length 4096 May 3 11:22:01 judeah /kernel: (da1:ahc0:0:1:0): no longer in timeout, status = 34b May 3 11:22:01 judeah /kernel: ahc0: Issued Channel A Bus Reset. 1 SCBs aborted May 3 11:22:01 judeah /kernel: vinum: dataraid.p0.s1 is stale by force May 3 11:22:01 judeah /kernel: vinum: dataraid.p0 is corrupt May 3 11:22:01 judeah /kernel: fatal :dataraid.p0.s1 write error, block 1905465 for 8192 bytes May 3 11:22:01 judeah /kernel: dataraid.p0.s1: user buffer block 13336624 for 8192 bytes It looks like SCSI disk da1 was timing out but recovered. This is speculation on my part. Upon rebooting today, da1 seems to be OK? May 10 07:03:00 judeah /kernel: da1 at ahc0 bus 0 target 1 lun 0 May 10 07:03:00 judeah /kernel: da1: Fixed Direct Access SCSI-2 device May 10 07:03:00 judeah /kernel: da1: 10.000MB/s transfers (10.000MHz, offset 15), Tagged Queueing Enabled May 10 07:03:00 judeah /kernel: da1: 1920MB (3933040 512 byte sectors: 255H 63S/T 244C) So, the question, do I have a hardware failure? If so, is it the Adaptec 2940/UW controller or the SCSI disk? When I get this resolved, I'll obviously have to figure out how to fix my corrupt vinum volume :( -- Regards, Doug