From owner-freebsd-hardware@FreeBSD.ORG Thu Jan 21 07:22:00 2010 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23FE11065679; Thu, 21 Jan 2010 07:22:00 +0000 (UTC) (envelope-from stephane.lapie@darkbsd.org) Received: from quasar.darkbsd.org (shinigami.darkbsd.org [82.227.96.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8432D8FC17; Thu, 21 Jan 2010 07:21:59 +0000 (UTC) Received: from quasar.darkbsd.org (localhost [127.0.0.1]) by quasar.darkbsd.org (Postfix) with ESMTP id D0ADA1D02; Thu, 21 Jan 2010 08:21:52 +0100 (CET) Received: from quasar.darkbsd.org ([127.0.0.1]) by quasar.darkbsd.org (quasar.darkbsd.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SXwsPjhsMKPt; Thu, 21 Jan 2010 08:21:50 +0100 (CET) Received: from [192.168.166.29] (unknown [210.188.173.245]) (Authenticated sender: darksoul) by quasar.darkbsd.org (Postfix) with ESMTPSA id 00AC71CFB; Thu, 21 Jan 2010 08:21:48 +0100 (CET) Message-ID: <4B58008C.4050207@darkbsd.org> Date: Thu, 21 Jan 2010 16:21:48 +0900 From: Stephane LAPIE User-Agent: Thunderbird 2.0.0.23 (X11/20090817) MIME-Version: 1.0 To: John Baldwin References: <4B56CD4C.80503@darkbsd.org> <201001200848.16874.jhb@freebsd.org> <4B571CB7.3020303@darkbsd.org> <201001201105.26367.jhb@freebsd.org> In-Reply-To: <201001201105.26367.jhb@freebsd.org> X-Enigmail-Version: 0.96.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig13C762675B37AB0B44CDB19F" Cc: freebsd-hardware@freebsd.org Subject: Re: DELL SAS5/E Controller bug X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2010 07:22:00 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig13C762675B37AB0B44CDB19F Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: quoted-printable John Baldwin wrote: > On Wednesday 20 January 2010 10:09:43 am Stephane LAPIE wrote: >> John Baldwin wrote: >>> On Wednesday 20 January 2010 4:30:52 am Stephane LAPIE wrote: >>>> Hello list, >>>> >>>> Basically I'm experiencing the same problem as described here : >>>> https://forums.freebsd.org/showthread.php?t=3D9407 (linking for refe= rence) >>>> >>>> Drives disconnections are not recognized instantly, and instead I ge= t >>>> the following dmesg entries : >>>> mpt0: mpt_cam_event: 0x16 >>>> mpt0: mpt_cam_event: 0x16 >>>> >>>> (Sometimes I also get "mpt0: mpt_cam_event: 0x12" events) >>>> >>>> This is really crippling as this litterally paralyzes the ZFS pool u= ntil >>>> the controller finally comes to its senses (...or until a disk gets >>>> replugged in, which provokes a flush of all the buffered failed SCSI= >>>> requests). >>>> >>>> Hardware is recognized as : >>>> mpt0@pci0:6:8:0: class=3D0x010000 card=3D0x1f041028 chip=3D0x0054100= 0 rev=3D0x01 >>>> hdr=3D0x00 >>>> vendor =3D 'LSI Logic (Was: Symbios Logic, NCR)' >>>> device =3D 'SAS 3000 series, 8-port with 1068 -StorPort' >>>> class =3D mass storage >>>> subclass =3D SCSI >>>> >>>> Did anyone else experience this, or find a proper work-around ? >>> Invoke 'camcontrol rescan' after removing a drive. mptutil(8) does t= he=20 >>> equivalent when adding and removing volumes to make up for the driver= not=20 >>> automatically rescanning. >> I already tried reset/rescan via camcontrol, but after removing a driv= e,=20 >> the process freezes (process status "D", Ctrl+T in terminal shows it's= =20 >> in a "cbwait" state, it can't be bg'ed). I did not wait for a hardware= =20 >> timeout, I tried replugging the drive, which released the ZFS and=20 >> camcontrol locks. >> >> >> Also, I tried poking around with mptutil and could obtain the followin= g=20 >> information, if it can be of any help : >> >> freebsd-r610# mptutil -u 0 show adapter >> mpt0 Adapter: >> Board Name: SAS5e >> Board Assembly: >> Chip Name: C1068 >> Chip Revision: UNUSED >> RAID Levels: none >> mptutil: Reading config page header failed: Invalid configuration page= >> >> (The above error message should be normal since this is not a RAID=20 >> controller, though a bit jarring) >=20 > This patch should fix that: >=20 > Index: mpt_show.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- mpt_show.c (revision 202640) > +++ mpt_show.c (working copy) > @@ -78,6 +78,7 @@ > CONFIG_PAGE_MANUFACTURING_0 *man0; > CONFIG_PAGE_IOC_2 *ioc2; > CONFIG_PAGE_IOC_6 *ioc6; > + U16 IOCStatus; > int fd, comma; > =20 > if (ac !=3D 1) { > @@ -108,7 +109,7 @@ > =20 > free(man0); > =20 > - ioc2 =3D mpt_read_ioc_page(fd, 2, NULL); > + ioc2 =3D mpt_read_ioc_page(fd, 2, &IOCStatus); > if (ioc2 !=3D NULL) { > printf(" RAID Levels:"); > comma =3D 0; > @@ -151,9 +152,10 @@ > printf(" none"); > printf("\n"); > free(ioc2); > - } > + } else if (IOCStatus !=3D MPI_IOCSTATUS_CONFIG_INVALID_PAGE) > + warnx("mpt_read_ioc_page(2): %s", mpt_ioc_status(IOCStatus)); > =20 > - ioc6 =3D mpt_read_ioc_page(fd, 6, NULL); > + ioc6 =3D mpt_read_ioc_page(fd, 6, &IOCStatus); > if (ioc6 !=3D NULL) { > display_stripe_map(" RAID0 Stripes", > ioc6->SupportedStripeSizeMapIS); > @@ -172,7 +174,8 @@ > printf("-%u", ioc6->MaxDrivesIME); > printf("\n"); > free(ioc6); > - } > + } else if (IOCStatus !=3D MPI_IOCSTATUS_CONFIG_INVALID_PAGE) > + warnx("mpt_read_ioc_page(2): %s", mpt_ioc_status(IOCStatus)); > =20 > /* TODO: Add an ioctl to fetch IOC_FACTS and print firmware version. = */ > =20 >=20 >> However, the following is a bit disturbing : >> >> freebsd-r610# mptutil -u 0 show drives >> mpt0 Physical Drives: >> da0 ( 932G) ONLINE SAS bus 0 id 0 >> da1 ( 932G) ONLINE SAS bus 0 id 1 >> da2 ( 932G) ONLINE SAS bus 0 id 2 >> da3 ( 932G) ONLINE SAS bus 0 id 3 >> da4 ( 932G) ONLINE SAS bus 0 id 4 >> da5 ( 932G) ONLINE SAS bus 0 id 5 >> da6 ( 932G) ONLINE SAS bus 0 id 6 >> da7 ( 932G) ONLINE SAS bus 0 id 7 >> da8 ( 932G) ONLINE SAS bus 0 id 8 >> da9 ( 932G) ONLINE SAS bus 0 id 9 >> da10 ( 932G) ONLINE SAS bus 0 id 10 >> da11 ( 932G) ONLINE SAS bus 0 id 11 >> da12 ( 932G) ONLINE SAS bus 0 id 12 >> da13 ( 932G) ONLINE SAS bus 0 id 13 >> da14 ( 932G) ONLINE SAS bus 0 id 14 >> da15 ( 136G) ONLINE SAS bus 0 id 0 >> >> The above listing seems weird, as da15 should belong to mpt1. >=20 > Agreed. I specifically ask that CAM only return results for devices on= bus 0 > of mptX. Before when I debugged this I used gdb and set a breakpoint i= n > mpt_fetch_disks() so I could examine the structures that CAM returned. = This > is the code that identifies mptX vs mpt: >=20 > /* Match mptX bus 0. */ > ccb.cdm.patterns[0].type =3D DEV_MATCH_BUS; > b =3D &ccb.cdm.patterns[0].pattern.bus_pattern; > snprintf(b->dev_name, sizeof(b->dev_name), "mpt"); > b->unit_number =3D mpt_unit; > b->bus_id =3D 0; > b->flags =3D BUS_MATCH_NAME | BUS_MATCH_UNIT | BUS_MATCH_BUS_ID; >=20 > 'mpt_unit' is a global variable that is set to the value of the 'u' > parameter. >=20 >> freebsd-r610# mptutil -u 1 show drives >> mptutil: mpt_fetch_disks got wrong CAM matches >> mpt1 Physical Drives: >> 0 ( 137G) ONLINE SAS bus 0 id 1 >> 1 ( 137G) ONLINE SAS bus 0 id 9 >=20 > Similarly I would use gdb to exmaine the reply from CAM here to see why= > it got 'wrong CAM matches'. The code expects the first match to match > the bus and the next N matches should be 'daX' devices. >=20 I just applied your patch to mptutil source, which now returns : freebsd-r610# mptutil show adapter mpt0 Adapter: Board Name: SAS5e Board Assembly: Chip Name: C1068 Chip Revision: UNUSED RAID Levels: none mptutil: mpt_read_ioc_page(2): Invalid configuration page I will give a try on the gdb thing once I get a chance of installing the source tree on this test machine. Also, I pasted the dmesg trace of trying to remove da0 and da6 and trying to have the system register the removal via a "camcontrol rescan 0= " : -> Unplugging "da0" and "da6" : mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x16 -> Then running "camcontrol rescan 0" (which leaves "cbwait" state and finishes at 187s real time) mpt0: request 0xffffff80005bcea0:5936 timed out for ccb 0xffffff00032d4000 (req->ccb 0xffffff00032d4000) mpt0: attempting to abort req 0xffffff80005bcea0:5936 function 0 mpt0: mpt_wait_req(1) timed out mpt0: mpt_recover_commands: abort timed-out. Resetting controller mpt0: mpt_cam_event: 0x0 mpt0: completing timedout/aborted req 0xffffff80005bcea0:5936 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x16 (da0:mpt0:0:0:0): lost device (da0:mpt0:0:0:0): Synchronize cache failed, status =3D=3D 0x4a, scsi stat= us =3D=3D 0x0 (da0:mpt0:0:0:0): removing device entry (da6:mpt0:0:6:0): lost device (da6:mpt0:0:6:0): Synchronize cache failed, status =3D=3D 0x4a, scsi stat= us =3D=3D 0x0 (da6:mpt0:0:6:0): removing device entry -> Then replugging the drive "da0" : mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x16 -> Then running "camcontrol rescan 0" (which responds in a few seconds time, between 7~10s) : da0 at mpt0 bus 0 target 6 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 300.000MB/s transfers da0: Command Queueing enabled da0: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) -> Then replugging the drive "da6" : mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x16 -> Then running "camcontrol rescan 0" (which responds in a few seconds time, between 7~10s) : da6 at mpt0 bus 0 target 6 lun 0 da6: Fixed Direct Access SCSI-5 device da6: 300.000MB/s transfers da6: Command Queueing enabled da6: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C) Is there any documentation or hint as to what those mpt_cam_event are ? I could whip myself a quick patch to at least change the display so one would figure what these are. It feels like the 0x12 and 0x16 have to be handled to invalidate the device that has been unplugged so the next request won't timeout but fail directly. This is actually not as crippling as I initially thought for use with ZFS, but waiting 190s (3x60s timeouts + normal execution of camcontrol ?) sounds a bit overboard. On a separate note (and this is not a real problem), my test case of plugging out two drives from a 2xraidz1 pool made me notice a little quirk that has the system reusing the first free daX slot upon replugging a drive and thus getting another device name. I guess that could probably be worked around with either of the two methods : - Binding bus:target:lun to a fixed daX number - zpool export/importing the pool upon disk failure (though this is a bit constraining) Thanks again for your time, --=20 Stephane LAPIE, EPITA SRS, Promo 2005 "Even when they have digital readouts, I can't understand them." --MegaTokyo --------------enig13C762675B37AB0B44CDB19F Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktYAIwACgkQ24Ql8u6TF2NhwgCgm3LGsRSL47RB+2hNSPc0Y1nd ZKMAn3VTjkqyQFuB0WJ5AHnaPiHVwhl5 =Pr/d -----END PGP SIGNATURE----- --------------enig13C762675B37AB0B44CDB19F--