From owner-freebsd-bugs@FreeBSD.ORG Fri May 10 07:20:00 2013 Return-Path: Delivered-To: freebsd-bugs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C8853135 for ; Fri, 10 May 2013 07:20:00 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id AA9F13A4 for ; Fri, 10 May 2013 07:20:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r4A7K0RL058930 for ; Fri, 10 May 2013 07:20:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r4A7K0rr058928; Fri, 10 May 2013 07:20:00 GMT (envelope-from gnats) Resent-Date: Fri, 10 May 2013 07:20:00 GMT Resent-Message-Id: <201305100720.r4A7K0rr058928@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Garrett Wollman Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E555AE5B for ; Fri, 10 May 2013 07:12:27 +0000 (UTC) (envelope-from wollman@xyz.csail.mit.edu) Received: from khavrinen.csail.mit.edu (khavrinen.csail.mit.edu [IPv6:2001:470:8b2d:1e1c:21b:21ff:feb8:d7b0]) by mx1.freebsd.org (Postfix) with ESMTP id 6A49E336 for ; Fri, 10 May 2013 07:12:27 +0000 (UTC) Received: from xyz.csail.mit.edu (xyz.csail.mit.edu [128.31.0.28]) by khavrinen.csail.mit.edu (8.14.5/8.14.5) with ESMTP id r4A7CPPG021781 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO CN= issuer=) for ; Fri, 10 May 2013 03:12:25 -0400 (EDT) (envelope-from wollman@xyz.csail.mit.edu) Received: (from wollman@localhost) by xyz.csail.mit.edu (8.14.5/8.14.5/Submit) id r4A7CPrC076478; Fri, 10 May 2013 03:12:25 -0400 (EDT) (envelope-from wollman) Message-Id: <201305100712.r4A7CPrC076478@xyz.csail.mit.edu> Date: Fri, 10 May 2013 03:12:25 -0400 (EDT) From: Garrett Wollman To: FreeBSD-gnats-submit@freebsd.org X-Send-Pr-Version: 3.113 Subject: kern/178473: gmultipath needs to be smarter about errors X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 May 2013 07:20:00 -0000 >Number: 178473 >Category: kern >Synopsis: gmultipath needs to be smarter about errors >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: change-request >Submitter-Id: current-users >Arrival-Date: Fri May 10 07:20:00 UTC 2013 >Closed-Date: >Last-Modified: >Originator: Garrett Wollman >Release: FreeBSD 9.1-RELEASE-p2 amd64 >Organization: MIT Computer Science & Artificial Intelligence Laboratory >Environment: System: FreeBSD xyz.csail.mit.edu 9.1-RELEASE-p2 FreeBSD 9.1-RELEASE-p2 #27 r250065M: Mon Apr 29 17:11:15 EDT 2013 wollman@xyz.csail.mit.edu:/usr/obj/usr/src/sys/CSAIL amd64 >Description: Currently, hard medium errors reported by the underlying provider will cause geom_multipath to cycle infinitely, turning what should be a failure reported to the consumer (ZFS in my case, which can do something useful with it) into a write that simply never returns at all, untill the hardware is physically offlined. It looks like this: (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x2103f4f7 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x2103f4f7 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040687 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040687 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040f4f (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040f4f (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 43 fe 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x210443ff (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 43 fe 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x210443ff (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 ac 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x2103f4f7 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 ac 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x2103f4f7 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040687 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040687 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040f4f (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040f4f (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 17 f3 0 0 2a 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21041817 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 67 eb 0 0 2a 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x210467ec (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 67 eb 0 0 2a 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x210467ec (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 d9 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x2103f4f7 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 d9 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x2103f4f7 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040687 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040687 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040f4f (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040f4f (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x2103f4f7 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x2103f4f7 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040687 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040687 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 e 7d 0 1 0 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040f4f (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 e 7d 0 1 0 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040f4f (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 pid 62004 (ntpd), uid 0: exited on signal 11 (core dumped) pid 62021 (ntpd), uid 0: exited on signal 11 (core dumped) (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x2103f4f7 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x2103f4f7 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040687 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040687 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040f4f (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040f4f (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040687 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040687 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040f4f (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040f4f (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 >How-To-Repeat: Build a big multipathed zpool, and use it. Eventually, a disk will fail. Wonder why ZFS doesn't notice it. (Apparently the SMART implementation on the drive doesn't consider a single hard read error to be "bad enough" to raise the failure-imminent bit.) Finally get the answer in the nightly root mail. >Fix: Don't know. geom_multipath should have some way of figuring out that a failure is "really" a failure on the underlying device, and notify the consumer appropriately. Perhaps keep some state in the bio that remembers which providers have failed the request already, and if every available provider fails the request without any intervening configuration changes, then return the last failure to the consumer. >Release-Note: >Audit-Trail: >Unformatted: