Date: Fri, 10 May 2013 03:12:25 -0400 (EDT) From: Garrett Wollman <wollman@csail.mit.edu> To: FreeBSD-gnats-submit@freebsd.org Subject: kern/178473: gmultipath needs to be smarter about errors Message-ID: <201305100712.r4A7CPrC076478@xyz.csail.mit.edu> Resent-Message-ID: <201305100720.r4A7K0rr058928@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 178473 >Category: kern >Synopsis: gmultipath needs to be smarter about errors >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: change-request >Submitter-Id: current-users >Arrival-Date: Fri May 10 07:20:00 UTC 2013 >Closed-Date: >Last-Modified: >Originator: Garrett Wollman >Release: FreeBSD 9.1-RELEASE-p2 amd64 >Organization: MIT Computer Science & Artificial Intelligence Laboratory >Environment: System: FreeBSD xyz.csail.mit.edu 9.1-RELEASE-p2 FreeBSD 9.1-RELEASE-p2 #27 r250065M: Mon Apr 29 17:11:15 EDT 2013 wollman@xyz.csail.mit.edu:/usr/obj/usr/src/sys/CSAIL amd64 >Description: Currently, hard medium errors reported by the underlying provider will cause geom_multipath to cycle infinitely, turning what should be a failure reported to the consumer (ZFS in my case, which can do something useful with it) into a write that simply never returns at all, untill the hardware is physically offlined. It looks like this: (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x2103f4f7 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x2103f4f7 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040687 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040687 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040f4f (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040f4f (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 43 fe 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x210443ff (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 43 fe 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x210443ff (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 ac 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x2103f4f7 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 ac 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x2103f4f7 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040687 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040687 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040f4f (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040f4f (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 17 f3 0 0 2a 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21041817 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 67 eb 0 0 2a 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x210467ec (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 67 eb 0 0 2a 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x210467ec (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 d9 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x2103f4f7 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 d9 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x2103f4f7 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040687 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040687 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040f4f (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040f4f (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x2103f4f7 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x2103f4f7 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040687 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040687 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 e 7d 0 1 0 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040f4f (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 e 7d 0 1 0 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040f4f (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 pid 62004 (ntpd), uid 0: exited on signal 11 (core dumped) pid 62021 (ntpd), uid 0: exited on signal 11 (core dumped) (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x2103f4f7 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x2103f4f7 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040687 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040687 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040f4f (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040f4f (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040687 (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040687 (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 (da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da158:mps2:0:77:0): CAM status: SCSI Status Error (da158:mps2:0:77:0): SCSI status: Check Condition (da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da158:mps2:0:77:0): Info: 0x21040f4f (da158:mps2:0:77:0): Actual Retry Count: 63 (da158:mps2:0:77:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85 GEOM_MULTIPATH: da85 is now active path in s25d12 (da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 (da85:mps1:0:102:0): CAM status: SCSI Status Error (da85:mps1:0:102:0): SCSI status: Check Condition (da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted) (da85:mps1:0:102:0): Info: 0x21040f4f (da85:mps1:0:102:0): Actual Retry Count: 63 (da85:mps1:0:102:0): Error 5, Unretryable error GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158 GEOM_MULTIPATH: da158 is now active path in s25d12 >How-To-Repeat: Build a big multipathed zpool, and use it. Eventually, a disk will fail. Wonder why ZFS doesn't notice it. (Apparently the SMART implementation on the drive doesn't consider a single hard read error to be "bad enough" to raise the failure-imminent bit.) Finally get the answer in the nightly root mail. >Fix: Don't know. geom_multipath should have some way of figuring out that a failure is "really" a failure on the underlying device, and notify the consumer appropriately. Perhaps keep some state in the bio that remembers which providers have failed the request already, and if every available provider fails the request without any intervening configuration changes, then return the last failure to the consumer. >Release-Note: >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201305100712.r4A7CPrC076478>