Date: Sat, 06 Dec 2014 02:40:00 +0100 From: Michelle Sullivan <michelle@sorbs.net> To: freebsd-fs@freebsd.org Subject: ZFS weird issue... Message-ID: <54825E70.20900@sorbs.net>
next in thread | raw e-mail | index | archive | help
Here's what happened:
LSI9260-16i, 16x3T SATA drives. 15 LDs each configured as single disk
RAID0 last drive a hot spare.
All LDs configured as a ZFS RaidZ2 pool ('sorbs')
Bay/Drive 8 failed, hot spare kicked in. ZFS resilvered. (mfid15 became
spare-8)
On reboot mfid9-15 were re-named automatically to mfid8-14 ... ZFS
didn't seems to care...
Days later new drive to replace the dead drive arrived and was
inserted. System refused to re-add as there was data in the cache, so
rebooted and cleared the cache (as per many on web faq's) Reconfigured
it to match the others. Can't do a zpool replace mfid8 because that's
already in the pool... (was mfid9) can't use mfid15 because zpool
reports it's not part of the config... can't use the uniq-id it received
(can't find vdev) ... HELP!! :)
This is what I can see currently ...
Thanks, in advance.
root@colossus:~ # zpool status -v
pool: VirtualDisks
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
VirtualDisks ONLINE 0 0 0
zvol/sorbs/VirtualDisks ONLINE 0 0 0
errors: No known data errors
pool: sorbs
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas
exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-2Q
scan: scrub in progress since Fri Dec 5 17:11:29 2014
2.51T scanned out of 29.9T at 89.4M/s, 89h7m to go
0 repaired, 8.40% done
config:
NAME STATE READ WRITE CKSUM
sorbs DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
mfid0 ONLINE 0 0 0
mfid1 ONLINE 0 0 0
mfid2 ONLINE 0 0 0
mfid3 ONLINE 0 0 0
mfid4 ONLINE 0 0 0
mfid5 ONLINE 0 0 0
mfid6 ONLINE 0 0 0
mfid7 ONLINE 0 0 0
spare-8 DEGRADED 0 0 0
1702922605 UNAVAIL 0 0 0 was /dev/mfid8
mfid14 ONLINE 0 0 0
mfid8 ONLINE 0 0 0
mfid9 ONLINE 0 0 0
mfid10 ONLINE 0 0 0
mfid11 ONLINE 0 0 0
mfid12 ONLINE 0 0 0
mfid13 ONLINE 0 0 0
spares
933862663 INUSE was /dev/mfid14
errors: No known data errors
root@colossus:~ # uname -a
FreeBSD colossus.sorbs.net 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898:
Thu Sep 26 22:50:31 UTC 2013
root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
root@colossus:~ # sh ./lsi.sh drives
Slot Number: 0 - Online, Spun Up
Slot Number: 1 - Online, Spun Up
Slot Number: 2 - Online, Spun Up
Slot Number: 3 - Online, Spun Up
Slot Number: 4 - Online, Spun Up
Slot Number: 5 - Online, Spun Up
Slot Number: 6 - Online, Spun Up
Slot Number: 7 - Online, Spun Up
Slot Number: 8 - Online, Spun Up
Slot Number: 9 - Online, Spun Up
Slot Number: 10 - Online, Spun Up
Slot Number: 11 - Online, Spun Up
Slot Number: 12 - Online, Spun Up
Slot Number: 13 - Online, Spun Up
Slot Number: 14 - Online, Spun Up
Slot Number: 15 - Online, Spun Up
root@colossus:~ # sh ./lsi.sh status
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 1 (Target Id: 1)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 2 (Target Id: 2)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 3 (Target Id: 3)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 4 (Target Id: 4)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 5 (Target Id: 5)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 6 (Target Id: 6)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 7 (Target Id: 7)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 8 (Target Id: 8)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 512 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disk's Default
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 9 (Target Id: 9)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 10 (Target Id: 10)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 11 (Target Id: 11)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 12 (Target Id: 12)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 13 (Target Id: 13)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 14 (Target Id: 14)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 15 (Target Id: 15)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 2.728 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 0
State : Optimal
Strip Size : 256 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Exit Code: 0x00
###############################################
Adapter 0: Patrol Read Information:
Patrol Read Mode: Auto
Patrol Read Execution Delay: 672 hours
Number of iterations completed: 2
Next start time: 12/13/2014, 03:00:00
Current State: Stopped
Patrol Read on SSD Devices: Disabled
Exit Code: 0x00
###############################################
Check Consistency on VD #0 is not in progress.
Check Consistency on VD #1 is not in progress.
Check Consistency on VD #2 is not in progress.
Check Consistency on VD #3 is not in progress.
Check Consistency on VD #4 is not in progress.
Check Consistency on VD #5 is not in progress.
Check Consistency on VD #6 is not in progress.
Check Consistency on VD #7 is not in progress.
Check Consistency on VD #8 is not in progress.
Check Consistency on VD #9 is not in progress.
Check Consistency on VD #10 is not in progress.
Check Consistency on VD #11 is not in progress.
Check Consistency on VD #12 is not in progress.
Check Consistency on VD #13 is not in progress.
Check Consistency on VD #14 is not in progress.
Check Consistency on VD #15 is not in progress.
Exit Code: 0x00
root@colossus:~ # ls -l /dev/mfi*
crw-r----- 1 root operator 0x22 Dec 5 17:18 /dev/mfi0
crw-r----- 1 root operator 0x68 Dec 5 17:18 /dev/mfid0
crw-r----- 1 root operator 0x69 Dec 5 17:18 /dev/mfid1
crw-r----- 1 root operator 0x78 Dec 5 17:18 /dev/mfid10
crw-r----- 1 root operator 0x79 Dec 5 17:18 /dev/mfid11
crw-r----- 1 root operator 0x7a Dec 5 17:18 /dev/mfid12
crw-r----- 1 root operator 0x82 Dec 5 17:18 /dev/mfid13
crw-r----- 1 root operator 0x83 Dec 5 17:18 /dev/mfid14
crw-r----- 1 root operator 0x84 Dec 5 17:18 /dev/mfid15
crw-r----- 1 root operator 0x6a Dec 5 17:18 /dev/mfid2
crw-r----- 1 root operator 0x6b Dec 5 17:18 /dev/mfid3
crw-r----- 1 root operator 0x6c Dec 5 17:18 /dev/mfid4
crw-r----- 1 root operator 0x6d Dec 5 17:18 /dev/mfid5
crw-r----- 1 root operator 0x6e Dec 5 17:18 /dev/mfid6
crw-r----- 1 root operator 0x75 Dec 5 17:18 /dev/mfid7
crw-r----- 1 root operator 0x76 Dec 5 17:18 /dev/mfid8
crw-r----- 1 root operator 0x77 Dec 5 17:18 /dev/mfid9
root@colossus:~ #
--
Michelle Sullivan
http://www.mhix.org/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54825E70.20900>