Date: Sat, 06 Dec 2014 01:44:59 +0000 From: Steven Hartland <killing@multiplay.co.uk> To: freebsd-fs@freebsd.org Subject: Re: ZFS weird issue... Message-ID: <54825F9B.6000009@multiplay.co.uk> In-Reply-To: <54825E70.20900@sorbs.net> References: <54825E70.20900@sorbs.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Formatting for you zpool list looks odd, making it not clear if its your
spare which is unavailable or one of your pool drives.
If its the spare, try zpool remove.
On 06/12/2014 01:40, Michelle Sullivan wrote:
> Here's what happened:
>
> LSI9260-16i, 16x3T SATA drives. 15 LDs each configured as single disk
> RAID0 last drive a hot spare.
>
> All LDs configured as a ZFS RaidZ2 pool ('sorbs')
>
> Bay/Drive 8 failed, hot spare kicked in. ZFS resilvered. (mfid15 became
> spare-8)
>
> On reboot mfid9-15 were re-named automatically to mfid8-14 ... ZFS
> didn't seems to care...
>
> Days later new drive to replace the dead drive arrived and was
> inserted. System refused to re-add as there was data in the cache, so
> rebooted and cleared the cache (as per many on web faq's) Reconfigured
> it to match the others. Can't do a zpool replace mfid8 because that's
> already in the pool... (was mfid9) can't use mfid15 because zpool
> reports it's not part of the config... can't use the uniq-id it received
> (can't find vdev) ... HELP!! :)
>
> This is what I can see currently ...
>
> Thanks, in advance.
>
> root@colossus:~ # zpool status -v
> pool: VirtualDisks
> state: ONLINE
> scan: none requested
> config:
>
> NAME STATE READ WRITE CKSUM
> VirtualDisks ONLINE 0 0 0
> zvol/sorbs/VirtualDisks ONLINE 0 0 0
>
> errors: No known data errors
>
> pool: sorbs
> state: DEGRADED
> status: One or more devices could not be opened. Sufficient replicas
> exist for
> the pool to continue functioning in a degraded state.
> action: Attach the missing device and online it using 'zpool online'.
> see: http://illumos.org/msg/ZFS-8000-2Q
> scan: scrub in progress since Fri Dec 5 17:11:29 2014
> 2.51T scanned out of 29.9T at 89.4M/s, 89h7m to go
> 0 repaired, 8.40% done
> config:
>
> NAME STATE READ WRITE CKSUM
> sorbs DEGRADED 0 0 0
> raidz2-0 DEGRADED 0 0 0
> mfid0 ONLINE 0 0 0
> mfid1 ONLINE 0 0 0
> mfid2 ONLINE 0 0 0
> mfid3 ONLINE 0 0 0
> mfid4 ONLINE 0 0 0
> mfid5 ONLINE 0 0 0
> mfid6 ONLINE 0 0 0
> mfid7 ONLINE 0 0 0
> spare-8 DEGRADED 0 0 0
> 1702922605 UNAVAIL 0 0 0 was /dev/mfid8
> mfid14 ONLINE 0 0 0
> mfid8 ONLINE 0 0 0
> mfid9 ONLINE 0 0 0
> mfid10 ONLINE 0 0 0
> mfid11 ONLINE 0 0 0
> mfid12 ONLINE 0 0 0
> mfid13 ONLINE 0 0 0
> spares
> 933862663 INUSE was /dev/mfid14
>
> errors: No known data errors
> root@colossus:~ # uname -a
> FreeBSD colossus.sorbs.net 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898:
> Thu Sep 26 22:50:31 UTC 2013
> root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
> root@colossus:~ # sh ./lsi.sh drives
> Slot Number: 0 - Online, Spun Up
> Slot Number: 1 - Online, Spun Up
> Slot Number: 2 - Online, Spun Up
> Slot Number: 3 - Online, Spun Up
> Slot Number: 4 - Online, Spun Up
> Slot Number: 5 - Online, Spun Up
> Slot Number: 6 - Online, Spun Up
> Slot Number: 7 - Online, Spun Up
> Slot Number: 8 - Online, Spun Up
> Slot Number: 9 - Online, Spun Up
> Slot Number: 10 - Online, Spun Up
> Slot Number: 11 - Online, Spun Up
> Slot Number: 12 - Online, Spun Up
> Slot Number: 13 - Online, Spun Up
> Slot Number: 14 - Online, Spun Up
> Slot Number: 15 - Online, Spun Up
> root@colossus:~ # sh ./lsi.sh status
>
>
> Adapter 0 -- Virtual Drive Information:
> Virtual Drive: 0 (Target Id: 0)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 1 (Target Id: 1)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 2 (Target Id: 2)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 3 (Target Id: 3)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 4 (Target Id: 4)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 5 (Target Id: 5)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 6 (Target Id: 6)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 7 (Target Id: 7)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 8 (Target Id: 8)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 512 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Disk's Default
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 9 (Target Id: 9)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 10 (Target Id: 10)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 11 (Target Id: 11)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 12 (Target Id: 12)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 13 (Target Id: 13)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 14 (Target Id: 14)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
> Virtual Drive: 15 (Target Id: 15)
> Name :
> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
> Size : 2.728 TB
> Sector Size : 512
> Is VD emulated : Yes
> Parity Size : 0
> State : Optimal
> Strip Size : 256 KB
> Number Of Drives : 1
> Span Depth : 1
> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
> Bad BBU
> Default Access Policy: Read/Write
> Current Access Policy: Read/Write
> Disk Cache Policy : Enabled
> Encryption Type : None
> Bad Blocks Exist: No
> Is VD Cached: No
>
>
>
> Exit Code: 0x00
> ###############################################
>
> Adapter 0: Patrol Read Information:
>
> Patrol Read Mode: Auto
> Patrol Read Execution Delay: 672 hours
> Number of iterations completed: 2
> Next start time: 12/13/2014, 03:00:00
> Current State: Stopped
> Patrol Read on SSD Devices: Disabled
>
> Exit Code: 0x00
> ###############################################
>
> Check Consistency on VD #0 is not in progress.
> Check Consistency on VD #1 is not in progress.
> Check Consistency on VD #2 is not in progress.
> Check Consistency on VD #3 is not in progress.
> Check Consistency on VD #4 is not in progress.
> Check Consistency on VD #5 is not in progress.
> Check Consistency on VD #6 is not in progress.
> Check Consistency on VD #7 is not in progress.
> Check Consistency on VD #8 is not in progress.
> Check Consistency on VD #9 is not in progress.
> Check Consistency on VD #10 is not in progress.
> Check Consistency on VD #11 is not in progress.
> Check Consistency on VD #12 is not in progress.
> Check Consistency on VD #13 is not in progress.
> Check Consistency on VD #14 is not in progress.
> Check Consistency on VD #15 is not in progress.
>
> Exit Code: 0x00
> root@colossus:~ # ls -l /dev/mfi*
> crw-r----- 1 root operator 0x22 Dec 5 17:18 /dev/mfi0
> crw-r----- 1 root operator 0x68 Dec 5 17:18 /dev/mfid0
> crw-r----- 1 root operator 0x69 Dec 5 17:18 /dev/mfid1
> crw-r----- 1 root operator 0x78 Dec 5 17:18 /dev/mfid10
> crw-r----- 1 root operator 0x79 Dec 5 17:18 /dev/mfid11
> crw-r----- 1 root operator 0x7a Dec 5 17:18 /dev/mfid12
> crw-r----- 1 root operator 0x82 Dec 5 17:18 /dev/mfid13
> crw-r----- 1 root operator 0x83 Dec 5 17:18 /dev/mfid14
> crw-r----- 1 root operator 0x84 Dec 5 17:18 /dev/mfid15
> crw-r----- 1 root operator 0x6a Dec 5 17:18 /dev/mfid2
> crw-r----- 1 root operator 0x6b Dec 5 17:18 /dev/mfid3
> crw-r----- 1 root operator 0x6c Dec 5 17:18 /dev/mfid4
> crw-r----- 1 root operator 0x6d Dec 5 17:18 /dev/mfid5
> crw-r----- 1 root operator 0x6e Dec 5 17:18 /dev/mfid6
> crw-r----- 1 root operator 0x75 Dec 5 17:18 /dev/mfid7
> crw-r----- 1 root operator 0x76 Dec 5 17:18 /dev/mfid8
> crw-r----- 1 root operator 0x77 Dec 5 17:18 /dev/mfid9
> root@colossus:~ #
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54825F9B.6000009>