Date: Fri, 07 Dec 2012 14:07:07 +0100 From: Johan Hendriks <joh.hendriks@gmail.com> To: freebsd-fs@freebsd.org Subject: Re: ZFS hang Message-ID: <50C1E9FB.4020108@gmail.com> In-Reply-To: <50C1DDE8.9030503@icritical.com> References: <50C1CB34.3000308@icritical.com> <50C1DDE8.9030503@icritical.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Matt Burke schreef: > After rebooting the box, I've just seen this on the console (after 'Setting > hostid'): > > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 21 53 8 0 1 0 0 > (da8:isci0:0:0:0): CAM status: SCSI Status Error > (da8:isci0:0:0:0): SCSI status: Check Condition > (da8:isci0:0:0:0): SCSI sense: MEDIUM ERROR asc:ffffffff,ffffffff (Reserved > ASC/ASCQ pair) > (da8:isci0:0:0:0): Info: 0x4215378 > (da8:isci0:0:0:0): Retrying command (per sense data) > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 21 53 8 0 1 0 0 > (da8:isci0:0:0:0): CAM status: SCSI Status Error > (da8:isci0:0:0:0): SCSI status: Check Condition > (da8:isci0:0:0:0): SCSI sense: MEDIUM ERROR asc:ffffffff,ffffffff (Reserved > ASC/ASCQ pair) > (da8:isci0:0:0:0): Info: 0x4215378 > (da8:isci0:0:0:0): Retrying command (per sense data) > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 21 53 8 0 1 0 0 > (da8:isci0:0:0:0): CAM status: SCSI Status Error > (da8:isci0:0:0:0): SCSI status: Check Condition > (da8:isci0:0:0:0): SCSI sense: MEDIUM ERROR asc:ffffffff,ffffffff (Reserved > ASC/ASCQ pair) > (da8:isci0:0:0:0): Info: 0x4215378 > (da8:isci0:0:0:0): Retrying command (per sense data) > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 21 53 8 0 1 0 0 > (da8:isci0:0:0:0): CAM status: SCSI Status Error > (da8:isci0:0:0:0): SCSI status: Check Condition > (da8:isci0:0:0:0): SCSI sense: MEDIUM ERROR asc:ffffffff,ffffffff (Reserved > ASC/ASCQ pair) > (da8:isci0:0:0:0): Info: 0x4215378 > (da8:isci0:0:0:0): Retrying command (per sense data) > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 21 53 8 0 1 0 0 > (da8:isci0:0:0:0): CAM status: SCSI Status Error > (da8:isci0:0:0:0): SCSI status: Check Condition > (da8:isci0:0:0:0): SCSI sense: MEDIUM ERROR asc:ffffffff,ffffffff (Reserved > ASC/ASCQ pair) > (da8:isci0:0:0:0): Info: 0x4215378 > (da8:isci0:0:0:0): Error 5, Retries exhausted > > and then again for the following: > > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 21 65 8 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 21 75 8 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 21 76 8 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 82 58 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 83 58 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 84 58 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 94 58 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 95 58 0 1 0 0 (only 2 retries) > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 9b 58 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 a4 58 0 1 0 0 (only 1 retry) > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 a5 58 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 a6 58 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 b4 58 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 b5 58 0 1 0 0 (2 retries) > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 b6 58 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 bc 58 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 c7 58 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 d7 58 0 1 0 0 (1 retry) > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 d8 58 0 1 0 0 > (da8:isci0:0:0:0): READ(10). CDB: 28 0 4 1 e8 58 0 1 0 0 > > Obviously, the cause of my problems would seem to be a hosed disk. However > the kernel msgbuf shows no complaints from the drive before reboot. > > da8 is a 60GB OCZ Agility 3 SSD (purchased prior to realising just how > unreliable they are). According to the SMART data, it's had just 146GB of > reads and 278GB writes over 3 power cycles with only 3 months power on > time, similar to the others that have failed (~60% failure rate for ours) > > I can understand the drive failing, I just can't understand how it hung the > system. I have had a similar thing happen on one of these machines before > (with GENERIC and no dumpdev, so no debugging) with one of these disks on > an Areca HBA. > > I've also had these drives fail on the onboard SATA controller, along with > SAS drives on the SAS controllers, with no undesirable effects (other than > having to swap it out). > > Could there be a problem with ATA devices on SCSI controllers which is > causing failures to be silently dropped? Is ZFS lacking a timeout on IO calls? > > I'm going to move all these SSDs onto the SATA controller, and see if I can > replicate the problem, but I'm not holding my breath over a conclusive result. > > > I had somthing simular. This was on 9.0 with a LSI 9211-8i controller and a supermicro backplane. One disk was failing, with simular kernel messages. It was a seagate 300 GB SAS disk But zfs got stuck on it. It totally hangs the system. Which was no fun at all because we use ZFS as the storage for our VMware hypervisors. After replacing that drive all was fine again. But it looks like ZFS or something else can not handle it properly. I also had a disk that was known to have bad sectors, this disk was ejected from the pool like it should with a message that too many errors occured on the disk. Gr Johan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50C1E9FB.4020108>