Date: Tue, 15 Dec 2009 15:24:03 -0800 From: Matt Simerson <matt@corp.spry.com> To: Solon Lutz <solon@pyro.de> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS RaidZ2 with 24 drives? Message-ID: <42952D86-6B4D-49A3-8E4F-7A1A53A954C2@spry.com> In-Reply-To: <568624531.20091215163420@pyro.de> References: <568624531.20091215163420@pyro.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On Dec 15, 2009, at 7:34 AM, Solon Lutz wrote:
> Hi,
>
> are there any cons against building a RaidZ2 with 24 1.5TB drives?
Funny, I asked myself a similar question in July 2008. Except I had
1.0TB drives.
$ ssh back01 zpool status
pool: back01
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
back01 ONLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
errors: No known data errors
$ ssh back02 zpool status
pool: back02
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
back02 ONLINE 0 0 934K
raidz1 ONLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
da3 ONLINE 0 0 0
da4 ONLINE 0 0 0
da5 ONLINE 0 0 0
da6 ONLINE 0 0 0
raidz1 ONLINE 0 0 1.83M
da8 ONLINE 0 0 0
da9 ONLINE 0 0 0
da10 ONLINE 0 0 0
da11 ONLINE 0 0 0
da12 ONLINE 0 0 0
da13 ONLINE 0 0 0
da14 ONLINE 0 0 0
raidz1 ONLINE 0 0 1.83M
da16 ONLINE 0 0 0
spare ONLINE 0 0 0
da17 ONLINE 0 0 0
da7 ONLINE 0 0 0
da18 ONLINE 0 0 0
da19 ONLINE 0 0 0
da20 ONLINE 0 0 0
da21 ONLINE 0 0 0
da22 ONLINE 0 0 0
spares
da15 AVAIL
da23 AVAIL
da7 INUSE currently in use
errors: 241 data errors, use '-v' for a list
I tried several combinations and ran benchmarks against ZFS in various
RAID-Z configs and finally determined that how I laid out the disks
didn't affect performance much. That was well before ZFS v13 was
committed, so there were many bug fixes and performance optimizations
since then.
I deployed using the two configurations you see above. Both machines
have a pair of Areca 1231ML RAID controllers with super-sized BBWC
(battery backed write cache). On back01, each controller presents a 12-
disk RAID-5 array and ZFS concatenates them into the zpool you see
above. On back02, the RAID controller is configured in JBOD mode and
disks are pooled as shown.
In 17 months of production, the ZFS pool on back02 has required
maintenance several times, including being down for days while a scrub
was being run. Yes, days. Several times. We've had a couple data
center power outages, and the only ZFS equipped backup servers that
had any issue was back02. The last scrub failed to fixed the data
errors. IIRC, the RAID controller write cache is not active in JBOD
mode. That could explain why back02 had problems and the rest of my
ZFS servers did not.
When another disk in back02 fails, I'll move all the data off back02
and rebuild the disk arrays using hardware RAID. In addition to having
zero disk errors, zero hardware problems, and zero ZFS data errors,
the ZFS backup servers deployed on top of hardware RAID are much
faster. How much faster?
In the past 3 days, I have had a cleanup process running that prunes
stale backups. On back01, the process has cleaned up 4TB of disk
space. On back02, it has only cleaned up 1.2TB. These cleanup
processes run while the machines are performing their 'normal' duties.
On average, the back02 processes take about 3-4x longer to run on
back02. It's not for lack of resources either. These are dual quad-
cores with 16GB of RAM each.
YMMV.
Matt
> In some old postings floating around the net a limit of 9 drives
> is recommended.
> Does this still apply to the current ZFS in 8.0?
>
>
> Best regards,
>
> Solon Lutz
>
>
> +-----------------------------------------------+
> | Pyro.Labs Berlin - Creativity for tomorrow |
> | Wasgenstrasse 75/13 - 14129 Berlin, Germany |
> | www.pyro.de - phone + 49 - 30 - 48 48 58 58 |
> | info@pyro.de - fax + 49 - 30 - 80 94 03 52 |
> +-----------------------------------------------+
>
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?42952D86-6B4D-49A3-8E4F-7A1A53A954C2>