Date: Sat, 06 Dec 2014 03:21:30 +0100 From: Michelle Sullivan <michelle@sorbs.net> To: Steven Hartland <killing@multiplay.co.uk> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS weird issue... Message-ID: <5482682A.7090107@sorbs.net> In-Reply-To: <54825F9B.6000009@multiplay.co.uk> References: <54825E70.20900@sorbs.net> <54825F9B.6000009@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Steven Hartland wrote:
> Formatting for you zpool list looks odd, making it not clear if its
> your spare which is unavailable or one of your pool drives.
>
> If its the spare, try zpool remove.
root@colossus:~ # zpool remove sorbs mfid14
cannot remove mfid14: only inactive hot spares, cache, top-level, or log
devices can be removed
The 'UNAVAIL' is the new drive.
Michelle
>
> On 06/12/2014 01:40, Michelle Sullivan wrote:
>> Here's what happened:
>>
>> LSI9260-16i, 16x3T SATA drives. 15 LDs each configured as single disk
>> RAID0 last drive a hot spare.
>>
>> All LDs configured as a ZFS RaidZ2 pool ('sorbs')
>>
>> Bay/Drive 8 failed, hot spare kicked in. ZFS resilvered. (mfid15 became
>> spare-8)
>>
>> On reboot mfid9-15 were re-named automatically to mfid8-14 ... ZFS
>> didn't seems to care...
>>
>> Days later new drive to replace the dead drive arrived and was
>> inserted. System refused to re-add as there was data in the cache, so
>> rebooted and cleared the cache (as per many on web faq's) Reconfigured
>> it to match the others. Can't do a zpool replace mfid8 because that's
>> already in the pool... (was mfid9) can't use mfid15 because zpool
>> reports it's not part of the config... can't use the uniq-id it received
>> (can't find vdev) ... HELP!! :)
>>
>> This is what I can see currently ...
>>
>> Thanks, in advance.
>>
>> root@colossus:~ # zpool status -v
>> pool: VirtualDisks
>> state: ONLINE
>> scan: none requested
>> config:
>>
>> NAME STATE READ WRITE CKSUM
>> VirtualDisks ONLINE 0 0 0
>> zvol/sorbs/VirtualDisks ONLINE 0 0 0
>>
>> errors: No known data errors
>>
>> pool: sorbs
>> state: DEGRADED
>> status: One or more devices could not be opened. Sufficient replicas
>> exist for
>> the pool to continue functioning in a degraded state.
>> action: Attach the missing device and online it using 'zpool online'.
>> see: http://illumos.org/msg/ZFS-8000-2Q
>> scan: scrub in progress since Fri Dec 5 17:11:29 2014
>> 2.51T scanned out of 29.9T at 89.4M/s, 89h7m to go
>> 0 repaired, 8.40% done
>> config:
>>
>> NAME STATE READ WRITE CKSUM
>> sorbs DEGRADED 0 0 0
>> raidz2-0 DEGRADED 0 0 0
>> mfid0 ONLINE 0 0 0
>> mfid1 ONLINE 0 0 0
>> mfid2 ONLINE 0 0 0
>> mfid3 ONLINE 0 0 0
>> mfid4 ONLINE 0 0 0
>> mfid5 ONLINE 0 0 0
>> mfid6 ONLINE 0 0 0
>> mfid7 ONLINE 0 0 0
>> spare-8 DEGRADED 0 0 0
>> 1702922605 UNAVAIL 0 0 0 was /dev/mfid8
>> mfid14 ONLINE 0 0 0
>> mfid8 ONLINE 0 0 0
>> mfid9 ONLINE 0 0 0
>> mfid10 ONLINE 0 0 0
>> mfid11 ONLINE 0 0 0
>> mfid12 ONLINE 0 0 0
>> mfid13 ONLINE 0 0 0
>> spares
>> 933862663 INUSE was /dev/mfid14
>>
>> errors: No known data errors
>> root@colossus:~ # uname -a
>> FreeBSD colossus.sorbs.net 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898:
>> Thu Sep 26 22:50:31 UTC 2013
>> root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
>> root@colossus:~ # sh ./lsi.sh drives
>> Slot Number: 0 - Online, Spun Up
>> Slot Number: 1 - Online, Spun Up
>> Slot Number: 2 - Online, Spun Up
>> Slot Number: 3 - Online, Spun Up
>> Slot Number: 4 - Online, Spun Up
>> Slot Number: 5 - Online, Spun Up
>> Slot Number: 6 - Online, Spun Up
>> Slot Number: 7 - Online, Spun Up
>> Slot Number: 8 - Online, Spun Up
>> Slot Number: 9 - Online, Spun Up
>> Slot Number: 10 - Online, Spun Up
>> Slot Number: 11 - Online, Spun Up
>> Slot Number: 12 - Online, Spun Up
>> Slot Number: 13 - Online, Spun Up
>> Slot Number: 14 - Online, Spun Up
>> Slot Number: 15 - Online, Spun Up
>> root@colossus:~ # sh ./lsi.sh status
>>
>> Adapter 0 -- Virtual Drive Information:
>> Virtual Drive: 0 (Target Id: 0)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 64 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 1 (Target Id: 1)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 64 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 2 (Target Id: 2)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 64 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 3 (Target Id: 3)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 64 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 4 (Target Id: 4)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 64 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 5 (Target Id: 5)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 64 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 6 (Target Id: 6)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 64 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 7 (Target Id: 7)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 64 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 8 (Target Id: 8)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 512 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Disk's Default
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 9 (Target Id: 9)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 64 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 10 (Target Id: 10)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 64 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 11 (Target Id: 11)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 64 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 12 (Target Id: 12)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 64 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 13 (Target Id: 13)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 64 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 14 (Target Id: 14)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 64 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 15 (Target Id: 15)
>> Name :
>> RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size : 2.728 TB
>> Sector Size : 512
>> Is VD emulated : Yes
>> Parity Size : 0
>> State : Optimal
>> Strip Size : 256 KB
>> Number Of Drives : 1
>> Span Depth : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy : Enabled
>> Encryption Type : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>>
>> Exit Code: 0x00
>> ###############################################
>> Adapter 0: Patrol Read Information:
>>
>> Patrol Read Mode: Auto
>> Patrol Read Execution Delay: 672 hours
>> Number of iterations completed: 2
>> Next start time: 12/13/2014, 03:00:00
>> Current State: Stopped
>> Patrol Read on SSD Devices: Disabled
>>
>> Exit Code: 0x00
>> ###############################################
>> Check Consistency on VD #0 is
>> not in progress.
>> Check Consistency on VD #1 is not in progress.
>> Check Consistency on VD #2 is not in progress.
>> Check Consistency on VD #3 is not in progress.
>> Check Consistency on VD #4 is not in progress.
>> Check Consistency on VD #5 is not in progress.
>> Check Consistency on VD #6 is not in progress.
>> Check Consistency on VD #7 is not in progress.
>> Check Consistency on VD #8 is not in progress.
>> Check Consistency on VD #9 is not in progress.
>> Check Consistency on VD #10 is not in progress.
>> Check Consistency on VD #11 is not in progress.
>> Check Consistency on VD #12 is not in progress.
>> Check Consistency on VD #13 is not in progress.
>> Check Consistency on VD #14 is not in progress.
>> Check Consistency on VD #15 is not in progress.
>>
>> Exit Code: 0x00
>> root@colossus:~ # ls -l /dev/mfi*
>> crw-r----- 1 root operator 0x22 Dec 5 17:18 /dev/mfi0
>> crw-r----- 1 root operator 0x68 Dec 5 17:18 /dev/mfid0
>> crw-r----- 1 root operator 0x69 Dec 5 17:18 /dev/mfid1
>> crw-r----- 1 root operator 0x78 Dec 5 17:18 /dev/mfid10
>> crw-r----- 1 root operator 0x79 Dec 5 17:18 /dev/mfid11
>> crw-r----- 1 root operator 0x7a Dec 5 17:18 /dev/mfid12
>> crw-r----- 1 root operator 0x82 Dec 5 17:18 /dev/mfid13
>> crw-r----- 1 root operator 0x83 Dec 5 17:18 /dev/mfid14
>> crw-r----- 1 root operator 0x84 Dec 5 17:18 /dev/mfid15
>> crw-r----- 1 root operator 0x6a Dec 5 17:18 /dev/mfid2
>> crw-r----- 1 root operator 0x6b Dec 5 17:18 /dev/mfid3
>> crw-r----- 1 root operator 0x6c Dec 5 17:18 /dev/mfid4
>> crw-r----- 1 root operator 0x6d Dec 5 17:18 /dev/mfid5
>> crw-r----- 1 root operator 0x6e Dec 5 17:18 /dev/mfid6
>> crw-r----- 1 root operator 0x75 Dec 5 17:18 /dev/mfid7
>> crw-r----- 1 root operator 0x76 Dec 5 17:18 /dev/mfid8
>> crw-r----- 1 root operator 0x77 Dec 5 17:18 /dev/mfid9
>> root@colossus:~ #
>>
>
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
--
Michelle Sullivan
http://www.mhix.org/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5482682A.7090107>