From owner-freebsd-fs@FreeBSD.ORG Sat Dec 6 01:40:09 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 78BA5715 for ; Sat, 6 Dec 2014 01:40:09 +0000 (UTC) Received: from hades.sorbs.net (hades.sorbs.net [67.231.146.201]) by mx1.freebsd.org (Postfix) with ESMTP id 63680858 for ; Sat, 6 Dec 2014 01:40:09 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from isux.com (firewall.isux.com [213.165.190.213]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0NG500B5V0U71E00@hades.sorbs.net> for freebsd-fs@freebsd.org; Fri, 05 Dec 2014 17:44:32 -0800 (PST) Message-id: <54825E70.20900@sorbs.net> Date: Sat, 06 Dec 2014 02:40:00 +0100 From: Michelle Sullivan User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.24) Gecko/20100301 SeaMonkey/1.1.19 To: freebsd-fs@freebsd.org Subject: ZFS weird issue... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Dec 2014 01:40:09 -0000 Here's what happened: LSI9260-16i, 16x3T SATA drives. 15 LDs each configured as single disk RAID0 last drive a hot spare. All LDs configured as a ZFS RaidZ2 pool ('sorbs') Bay/Drive 8 failed, hot spare kicked in. ZFS resilvered. (mfid15 became spare-8) On reboot mfid9-15 were re-named automatically to mfid8-14 ... ZFS didn't seems to care... Days later new drive to replace the dead drive arrived and was inserted. System refused to re-add as there was data in the cache, so rebooted and cleared the cache (as per many on web faq's) Reconfigured it to match the others. Can't do a zpool replace mfid8 because that's already in the pool... (was mfid9) can't use mfid15 because zpool reports it's not part of the config... can't use the uniq-id it received (can't find vdev) ... HELP!! :) This is what I can see currently ... Thanks, in advance. root@colossus:~ # zpool status -v pool: VirtualDisks state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM VirtualDisks ONLINE 0 0 0 zvol/sorbs/VirtualDisks ONLINE 0 0 0 errors: No known data errors pool: sorbs state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://illumos.org/msg/ZFS-8000-2Q scan: scrub in progress since Fri Dec 5 17:11:29 2014 2.51T scanned out of 29.9T at 89.4M/s, 89h7m to go 0 repaired, 8.40% done config: NAME STATE READ WRITE CKSUM sorbs DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 mfid0 ONLINE 0 0 0 mfid1 ONLINE 0 0 0 mfid2 ONLINE 0 0 0 mfid3 ONLINE 0 0 0 mfid4 ONLINE 0 0 0 mfid5 ONLINE 0 0 0 mfid6 ONLINE 0 0 0 mfid7 ONLINE 0 0 0 spare-8 DEGRADED 0 0 0 1702922605 UNAVAIL 0 0 0 was /dev/mfid8 mfid14 ONLINE 0 0 0 mfid8 ONLINE 0 0 0 mfid9 ONLINE 0 0 0 mfid10 ONLINE 0 0 0 mfid11 ONLINE 0 0 0 mfid12 ONLINE 0 0 0 mfid13 ONLINE 0 0 0 spares 933862663 INUSE was /dev/mfid14 errors: No known data errors root@colossus:~ # uname -a FreeBSD colossus.sorbs.net 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013 root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 root@colossus:~ # sh ./lsi.sh drives Slot Number: 0 - Online, Spun Up Slot Number: 1 - Online, Spun Up Slot Number: 2 - Online, Spun Up Slot Number: 3 - Online, Spun Up Slot Number: 4 - Online, Spun Up Slot Number: 5 - Online, Spun Up Slot Number: 6 - Online, Spun Up Slot Number: 7 - Online, Spun Up Slot Number: 8 - Online, Spun Up Slot Number: 9 - Online, Spun Up Slot Number: 10 - Online, Spun Up Slot Number: 11 - Online, Spun Up Slot Number: 12 - Online, Spun Up Slot Number: 13 - Online, Spun Up Slot Number: 14 - Online, Spun Up Slot Number: 15 - Online, Spun Up root@colossus:~ # sh ./lsi.sh status Adapter 0 -- Virtual Drive Information: Virtual Drive: 0 (Target Id: 0) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 1 (Target Id: 1) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 2 (Target Id: 2) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 3 (Target Id: 3) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 4 (Target Id: 4) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 5 (Target Id: 5) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 6 (Target Id: 6) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 7 (Target Id: 7) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 8 (Target Id: 8) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 512 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Disk's Default Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 9 (Target Id: 9) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 10 (Target Id: 10) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 11 (Target Id: 11) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 12 (Target Id: 12) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 13 (Target Id: 13) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 14 (Target Id: 14) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 15 (Target Id: 15) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 2.728 TB Sector Size : 512 Is VD emulated : Yes Parity Size : 0 State : Optimal Strip Size : 256 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Exit Code: 0x00 ############################################### Adapter 0: Patrol Read Information: Patrol Read Mode: Auto Patrol Read Execution Delay: 672 hours Number of iterations completed: 2 Next start time: 12/13/2014, 03:00:00 Current State: Stopped Patrol Read on SSD Devices: Disabled Exit Code: 0x00 ############################################### Check Consistency on VD #0 is not in progress. Check Consistency on VD #1 is not in progress. Check Consistency on VD #2 is not in progress. Check Consistency on VD #3 is not in progress. Check Consistency on VD #4 is not in progress. Check Consistency on VD #5 is not in progress. Check Consistency on VD #6 is not in progress. Check Consistency on VD #7 is not in progress. Check Consistency on VD #8 is not in progress. Check Consistency on VD #9 is not in progress. Check Consistency on VD #10 is not in progress. Check Consistency on VD #11 is not in progress. Check Consistency on VD #12 is not in progress. Check Consistency on VD #13 is not in progress. Check Consistency on VD #14 is not in progress. Check Consistency on VD #15 is not in progress. Exit Code: 0x00 root@colossus:~ # ls -l /dev/mfi* crw-r----- 1 root operator 0x22 Dec 5 17:18 /dev/mfi0 crw-r----- 1 root operator 0x68 Dec 5 17:18 /dev/mfid0 crw-r----- 1 root operator 0x69 Dec 5 17:18 /dev/mfid1 crw-r----- 1 root operator 0x78 Dec 5 17:18 /dev/mfid10 crw-r----- 1 root operator 0x79 Dec 5 17:18 /dev/mfid11 crw-r----- 1 root operator 0x7a Dec 5 17:18 /dev/mfid12 crw-r----- 1 root operator 0x82 Dec 5 17:18 /dev/mfid13 crw-r----- 1 root operator 0x83 Dec 5 17:18 /dev/mfid14 crw-r----- 1 root operator 0x84 Dec 5 17:18 /dev/mfid15 crw-r----- 1 root operator 0x6a Dec 5 17:18 /dev/mfid2 crw-r----- 1 root operator 0x6b Dec 5 17:18 /dev/mfid3 crw-r----- 1 root operator 0x6c Dec 5 17:18 /dev/mfid4 crw-r----- 1 root operator 0x6d Dec 5 17:18 /dev/mfid5 crw-r----- 1 root operator 0x6e Dec 5 17:18 /dev/mfid6 crw-r----- 1 root operator 0x75 Dec 5 17:18 /dev/mfid7 crw-r----- 1 root operator 0x76 Dec 5 17:18 /dev/mfid8 crw-r----- 1 root operator 0x77 Dec 5 17:18 /dev/mfid9 root@colossus:~ # -- Michelle Sullivan http://www.mhix.org/