Date: Thu, 30 May 2013 19:47:30 GMT From: Ryan Steinmetz <zi@FreeBSD.org> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/179118: [mfi] COMMAND 0x.. TIMEOUT AFTER ## SECONDS (Dell H710 Mini (blades)) Message-ID: <201305301947.r4UJlUlN043143@oldred.FreeBSD.org> Resent-Message-ID: <201305301950.r4UJo0GK041121@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 179118
>Category: kern
>Synopsis: [mfi] COMMAND 0x.. TIMEOUT AFTER ## SECONDS (Dell H710 Mini (blades))
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Thu May 30 19:50:00 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator: Ryan Steinmetz
>Release: 9.1-RELEASE
>Organization:
>Environment:
9.1-RELEASE
>Description:
I've had 9.1-R running on a few Dell M620 blades (with H710 controllers) in them for a bit now and have had command timeout errors showing up from time to time. The system appears to still be responsive, although, I have noticed a couple of times where disk I/O will seem to pause for a few seconds. No panics, nothing forcing me to restart.
sbruno@ reported that he was running R620s with H710P cards in them (not blades) with A02 (21.1.0-0007) firmware and was not running into this issue. I downgraded one of my systems to
the same firmware, but still ran into timeouts. Note H710 versus H710P.
Workload is elastic search, which is going to yield bursts of read/write. Sustained write or read at certain points. There will be periods of no activity as well.
Interestingly enough, I ran bonnie in a loop for a number of hours and did not receive any timeouts.
--
# mfiutil show adapter
mfi0 Adapter:
Product Name: PERC H710 Mini
Serial Number: 31A00ZD
Firmware: 21.2.0-0007
RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID6, RAID10, RAID50
Battery Backup: present
NVRAM: 32K
Onboard Memory: 512M
Minimum Stripe: 64k
Maximum Stripe: 1M
# uname -rm
9.1-RELEASE-p3 amd64
# pciconf -lv
mfi0@pci0:2:0:0: class=0x010400 card=0x1f371028 chip=0x005b1000 rev=0x05 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'MegaRAID SAS 2208 [Thunderbolt]'
class = mass storage
subclass = RAID
# dmesg | grep mfi0
mfi0: 1428 (422722487s/0x0020/info) - Shutdown command received from host
mfi0: 1429 (boot + 4s/0x0020/info) - Firmware initialization started (PCI ID 005b/1000/1f37/1028)
mfi0: 1430 (boot + 4s/0x0020/info) - Firmware version 3.130.05-2086
mfi0: 1431 (boot + 5s/0x0008/info) - Battery Present
mfi0: 1432 (boot + 5s/0x0020/info) - Package version 21.2.0-0007
mfi0: 1433 (boot + 5s/0x0020/info) - Board Revision A00
mfi0: 1434 (boot + 6s/0x0008/info) - Battery temperature is normal
mfi0: 1435 (boot + 6s/0x0008/info) - Current capacity of the battery is above threshold
mfi0: 1436 (boot + 20s/0x0004/info) - Enclosure PD 20(c None/p1) communication restored
mfi0: 1437 (boot + 20s/0x0002/info) - Inserted: Encl PD 20
mfi0: 1438 (boot + 20s/0x0002/info) - Inserted: PD 20(c None/p1) Info: enclPd=20, scsiType=d, portMap=00, sasAddr=5948f090ebf23500,0000000000000000
mfi0: 1439 (boot + 20s/0x0002/info) - Inserted: PD 00(e0x20/s0)
mfi0: 1440 (boot + 20s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=50000c0f02c1bab6,0000000000000000
mfi0: 1441 (boot + 20s/0x0002/info) - Inserted: PD 01(e0x20/s1)
mfi0: 1442 (boot + 20s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info: enclPd=20, scsiType=0, portMap=01, sasAddr=50000c0f026bb0d2,0000000000000000
mfi0: 1443 (422722572s/0x0020/info) - Time established as 05/24/13 14:56:12; (45 seconds since power on)
mfi0: 1444 (422722598s/0x0008/info) - Battery started charging
mfi0: 1445 (422722793s/0x0008/info) - Battery charge complete
mfi0: 1446 (422722801s/0x0020/info) - Host driver is loaded and operational
mfid0 on mfi0
mfid0: 857856MB (1756889088 sectors) RAID volume (no label) is optimal
Trying to mount root from ufs:/dev/mfid0p3 [rw]...
mfi0: 1447 (422723530s/0x0002/WARN) - PD 00(e0x20/s0) Path 50000c0f02c1bab6 reset (Type 03)
mfi0: 1448 (422723530s/0x0002/WARN) - PD 01(e0x20/s1) Path 50000c0f026bb0d2 reset (Type 03)
mfi0: 1449 (422723530s/0x0002/info) - Unexpected sense: PD 01(e0x20/s1) Path 50000c0f026bb0d2, CDB: 2a 00 23 44 ea 00 00 00 80 00, Sense: 6/29/02
mfi0: 1450 (422723530s/0x0002/info) - Unexpected sense: PD 00(e0x20/s0) Path 50000c0f02c1bab6, CDB: 2a 00 23 44 ea 00 00 00 80 00, Sense: 6/29/02
mfi0: COMMAND 0xffffff8002b915b8 TIMEOUT AFTER 56 SECONDS
mfi0: COMMAND 0xffffff8002b906d8 TIMEOUT AFTER 56 SECONDS
mfi0: COMMAND 0xffffff8002b916c8 TIMEOUT AFTER 56 SECONDS
mfi0: COMMAND 0xffffff8002b92740 TIMEOUT AFTER 56 SECONDS
mfi0: COMMAND 0xffffff8002b92388 TIMEOUT AFTER 56 SECONDS
mfi0: COMMAND 0xffffff8002b928d8 TIMEOUT AFTER 46 SECONDS
mfi0: COMMAND 0xffffff8002b91750 TIMEOUT AFTER 45 SECONDS
mfi0: COMMAND 0xffffff8002b8fd48 TIMEOUT AFTER 45 SECONDS
mfi0: COMMAND 0xffffff8002b91f48 TIMEOUT AFTER 58 SECONDS
mfi0: COMMAND 0xffffff8002b92c08 TIMEOUT AFTER 58 SECONDS
mfi0: COMMAND 0xffffff8002b915b8 TIMEOUT AFTER 36 SECONDS
mfi0: 899 (422741297s/0x0002/WARN) - Encl PD 20 Path 5948f090ec1e9c00 reset (Type 03)
mfi0: 900 (422766000s/0x0020/info) - Patrol Read started
mfi0: 901 (422775083s/0x0020/info) - Patrol Read complete
mfi0: 902 (422790275s/0x0002/WARN) - Encl PD 20 Path 5948f090ec1e9c00 reset (Type 03)
mfi0: COMMAND 0xffffff8002b90870 TIMEOUT AFTER 42 SECONDS
mfi0: COMMAND 0xffffff8002b8f5d8 TIMEOUT AFTER 34 SECONDS
mfi0: 903 (422852812s/0x0002/WARN) - PD 01(e0x20/s1) Path 50000c0f020c5e26 reset (Type 03)
mfi0: 904 (422852812s/0x0002/WARN) - Encl PD 20 Path 5948f090ec1e9c00 reset (Type 03)
mfi0: 905 (422852818s/0x0002/info) - Unexpected sense: PD 01(e0x20/s1) Path 50000c0f020c5e26, CDB: 2a 00 00 00 01 22 00 00 08 00, Sense: 6/29/02
mfi0: COMMAND 0xffffff8002b905c8 TIMEOUT AFTER 31 SECONDS
mfi0: COMMAND 0xffffff8002b8f330 TIMEOUT AFTER 31 SECONDS
mfi0: COMMAND 0xffffff8002b929e8 TIMEOUT AFTER 43 SECONDS
mfi0: COMMAND 0xffffff8002b91310 TIMEOUT AFTER 43 SECONDS
mfi0: COMMAND 0xffffff8002b90540 TIMEOUT AFTER 38 SECONDS
mfi0: COMMAND 0xffffff8002b8f660 TIMEOUT AFTER 38 SECONDS
mfi0: COMMAND 0xffffff8002b92fc0 TIMEOUT AFTER 38 SECONDS
mfi0: COMMAND 0xffffff8002b8f110 TIMEOUT AFTER 38 SECONDS
mfi0: COMMAND 0xffffff8002b910f0 TIMEOUT AFTER 38 SECONDS
mfi0: COMMAND 0xffffff8002b91a80 TIMEOUT AFTER 38 SECONDS
mfi0: COMMAND 0xffffff8002b90e48 TIMEOUT AFTER 34 SECONDS
mfi0: COMMAND 0xffffff8002b90a90 TIMEOUT AFTER 34 SECONDS
mfi0: COMMAND 0xffffff8002b91640 TIMEOUT AFTER 34 SECONDS
mfi0: COMMAND 0xffffff8002b92960 TIMEOUT AFTER 40 SECONDS
mfi0: COMMAND 0xffffff8002b92960 TIMEOUT AFTER 70 SECONDS
mfi0: COMMAND 0xffffff8002b92740 TIMEOUT AFTER 32 SECONDS
mfi0: COMMAND 0xffffff8002b92740 TIMEOUT AFTER 32 SECONDS
mfi0: COMMAND 0xffffff8002b928d8 TIMEOUT AFTER 32 SECONDS
mfi0: COMMAND 0xffffff8002b8fa18 TIMEOUT AFTER 31 SECONDS
mfi0: COMMAND 0xffffff8002b93268 TIMEOUT AFTER 51 SECONDS
mfi0: COMMAND 0xffffff8002b8f5d8 TIMEOUT AFTER 51 SECONDS
mfi0: COMMAND 0xffffff8002b8f198 TIMEOUT AFTER 51 SECONDS
mfi0: COMMAND 0xffffff8002b92630 TIMEOUT AFTER 51 SECONDS
mfi0: COMMAND 0xffffff8002b8f3b8 TIMEOUT AFTER 51 SECONDS
mfi0: COMMAND 0xffffff8002b8f000 TIMEOUT AFTER 51 SECONDS
mfi0: COMMAND 0xffffff8002b92740 TIMEOUT AFTER 51 SECONDS
mfi0: COMMAND 0xffffff8002b928d8 TIMEOUT AFTER 51 SECONDS
mfi0: COMMAND 0xffffff8002b8fa18 TIMEOUT AFTER 42 SECONDS
mfi0: COMMAND 0xffffff8002b8fd48 TIMEOUT AFTER 42 SECONDS
mfi0: 906 (422962690s/0x0002/WARN) - Encl PD 20 Path 5948f090ec1e9c00 reset (Type 03)
mfi0: 907 (423017849s/0x0002/WARN) - Encl PD 20 Path 5948f090ec1e9c00 reset (Type 03)
mfi0: 908 (423040626s/0x0002/WARN) - Encl PD 20 Path 5948f090ec1e9c00 reset (Type 03)
mfi0: COMMAND 0xffffff8002b904b8 TIMEOUT AFTER 31 SECONDS
>How-To-Repeat:
Install FreeBSD 9.1-RELEASE on a Dell M620 blade with a H710 RAID controller.
Wait.
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201305301947.r4UJlUlN043143>