From owner-freebsd-fs@FreeBSD.ORG Tue Dec 15 23:24:07 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE0571065670 for ; Tue, 15 Dec 2009 23:24:07 +0000 (UTC) (envelope-from matt@corp.spry.com) Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id CC85E8FC0A for ; Tue, 15 Dec 2009 23:24:07 +0000 (UTC) Received: by pxi12 with SMTP id 12so275634pxi.3 for ; Tue, 15 Dec 2009 15:24:07 -0800 (PST) Received: by 10.142.5.30 with SMTP id 30mr139889wfe.115.1260919446045; Tue, 15 Dec 2009 15:24:06 -0800 (PST) Received: from mattintosh.spry.com (isaid.donotdelete.com [64.79.222.10]) by mx.google.com with ESMTPS id 20sm210298pzk.5.2009.12.15.15.24.04 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 15 Dec 2009 15:24:05 -0800 (PST) From: Matt Simerson To: Solon Lutz In-Reply-To: <568624531.20091215163420@pyro.de> X-Priority: 3 (Normal) References: <568624531.20091215163420@pyro.de> Message-Id: <42952D86-6B4D-49A3-8E4F-7A1A53A954C2@spry.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Tue, 15 Dec 2009 15:24:03 -0800 X-Mailer: Apple Mail (2.936) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS RaidZ2 with 24 drives? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2009 23:24:08 -0000 On Dec 15, 2009, at 7:34 AM, Solon Lutz wrote: > Hi, > > are there any cons against building a RaidZ2 with 24 1.5TB drives? Funny, I asked myself a similar question in July 2008. Except I had 1.0TB drives. $ ssh back01 zpool status pool: back01 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM back01 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 errors: No known data errors $ ssh back02 zpool status pool: back02 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: none requested config: NAME STATE READ WRITE CKSUM back02 ONLINE 0 0 934K raidz1 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 raidz1 ONLINE 0 0 1.83M da8 ONLINE 0 0 0 da9 ONLINE 0 0 0 da10 ONLINE 0 0 0 da11 ONLINE 0 0 0 da12 ONLINE 0 0 0 da13 ONLINE 0 0 0 da14 ONLINE 0 0 0 raidz1 ONLINE 0 0 1.83M da16 ONLINE 0 0 0 spare ONLINE 0 0 0 da17 ONLINE 0 0 0 da7 ONLINE 0 0 0 da18 ONLINE 0 0 0 da19 ONLINE 0 0 0 da20 ONLINE 0 0 0 da21 ONLINE 0 0 0 da22 ONLINE 0 0 0 spares da15 AVAIL da23 AVAIL da7 INUSE currently in use errors: 241 data errors, use '-v' for a list I tried several combinations and ran benchmarks against ZFS in various RAID-Z configs and finally determined that how I laid out the disks didn't affect performance much. That was well before ZFS v13 was committed, so there were many bug fixes and performance optimizations since then. I deployed using the two configurations you see above. Both machines have a pair of Areca 1231ML RAID controllers with super-sized BBWC (battery backed write cache). On back01, each controller presents a 12- disk RAID-5 array and ZFS concatenates them into the zpool you see above. On back02, the RAID controller is configured in JBOD mode and disks are pooled as shown. In 17 months of production, the ZFS pool on back02 has required maintenance several times, including being down for days while a scrub was being run. Yes, days. Several times. We've had a couple data center power outages, and the only ZFS equipped backup servers that had any issue was back02. The last scrub failed to fixed the data errors. IIRC, the RAID controller write cache is not active in JBOD mode. That could explain why back02 had problems and the rest of my ZFS servers did not. When another disk in back02 fails, I'll move all the data off back02 and rebuild the disk arrays using hardware RAID. In addition to having zero disk errors, zero hardware problems, and zero ZFS data errors, the ZFS backup servers deployed on top of hardware RAID are much faster. How much faster? In the past 3 days, I have had a cleanup process running that prunes stale backups. On back01, the process has cleaned up 4TB of disk space. On back02, it has only cleaned up 1.2TB. These cleanup processes run while the machines are performing their 'normal' duties. On average, the back02 processes take about 3-4x longer to run on back02. It's not for lack of resources either. These are dual quad- cores with 16GB of RAM each. YMMV. Matt > In some old postings floating around the net a limit of 9 drives > is recommended. > Does this still apply to the current ZFS in 8.0? > > > Best regards, > > Solon Lutz > > > +-----------------------------------------------+ > | Pyro.Labs Berlin - Creativity for tomorrow | > | Wasgenstrasse 75/13 - 14129 Berlin, Germany | > | www.pyro.de - phone + 49 - 30 - 48 48 58 58 | > | info@pyro.de - fax + 49 - 30 - 80 94 03 52 | > +-----------------------------------------------+ > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"