From owner-freebsd-questions@FreeBSD.ORG Mon Jul 13 19:23:41 2009 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 39EC6106566B for ; Mon, 13 Jul 2009 19:23:41 +0000 (UTC) (envelope-from mkhitrov@gmail.com) Received: from mail-yx0-f181.google.com (mail-yx0-f181.google.com [209.85.210.181]) by mx1.freebsd.org (Postfix) with ESMTP id E69168FC1B for ; Mon, 13 Jul 2009 19:23:40 +0000 (UTC) (envelope-from mkhitrov@gmail.com) Received: by yxe11 with SMTP id 11so4500496yxe.3 for ; Mon, 13 Jul 2009 12:23:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :from:date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=1eptCsRyvZmmXf1LwHLRYvkp1npPhfySA0xccUpGZW4=; b=Fd8atwETYPysrAoU01N9NaJ2fPEoFfbtwwd4sFSUEX2hIgYZ4aQz5Dyax+4t4lxP+W H1/ooH/yzqvxLT2kXl+jaEEIN51xmPSknIg/7ZUt3gPTdFinyk93M38mT/MrC8iJAale NEK+9muz/E0IBmPneyGPmd+zjOogMEyQplKAk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; b=CnTw3LcfAWsu6ig6NX/hEXYVLdXyS6kqO/4Z/wB3BA0a6aS7ZvbrwtAk4flO9zRnYl PqIGw8biFyDMzff+pPsxBt1L4LUfkBt+QnMGeDAl5uSS6pcoX6bzlXnUe6UhMg8JQe7y 6hL0TAeFrKiCmriSLdeQ4I7P5J3m4AOf92Rgk= MIME-Version: 1.0 Received: by 10.90.96.1 with SMTP id t1mr2472136agb.46.1247513020139; Mon, 13 Jul 2009 12:23:40 -0700 (PDT) In-Reply-To: <937260.17107.qm@web51002.mail.re2.yahoo.com> References: <937260.17107.qm@web51002.mail.re2.yahoo.com> From: Maxim Khitrov Date: Mon, 13 Jul 2009 15:23:20 -0400 Message-ID: <26ddd1750907131223k9e20142n1fbc41e16d82bf87@mail.gmail.com> To: mahlerrd@yahoo.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Free BSD Questions list Subject: Re: ZFS or UFS for 4TB hardware RAID6? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2009 19:23:41 -0000 On Mon, Jul 13, 2009 at 2:13 PM, Richard Mahlerwein wro= te: > > --- On Mon, 7/13/09, Maxim Khitrov wrote: > >> From: Maxim Khitrov >> Subject: Re: ZFS or UFS for 4TB hardware RAID6? >> To: mahlerrd@yahoo.com >> Cc: "Free BSD Questions list" >> Date: Monday, July 13, 2009, 2:02 PM >> On Mon, Jul 13, 2009 at 1:46 PM, >> Richard Mahlerwein >> wrote: >> >> >> >> Your mileage may vary, but... >> >> >> >> I would investigate either using more spindles if >> you want >> >> to stick to RAID6, or perhaps using another RAID >> level if >> >> you will be with 4 drives for a while.=C2=A0 The >> reasoning >> >> is that there's an overhead with RAID 6 - parity >> blocks are >> >> written to 2 disks, so in a 4 drive combination >> you have 2 >> >> drives with data and 2 with parity. >> >> >> >> With 4 drives, you could get much, much higher >> performance >> >> out of RAID10 (which is alternatively called >> RAID0+1 or >> >> RAID1+0 depending on the manufacturer and on how >> accurate >> >> they wish to be, and on how they actually >> implemented it, >> >> too). This would also mean 2 usable drives, as >> well, so >> >> you'd have the same space available in RAID10 as >> your >> >> proposed RAID6. >> >> >> >> I would confirm you can, on the fly, convert from >> RAID10 to >> >> RAID6 after you add more drives.=C2=A0 If you can not, >> then >> >> by all means stick with RAID6 now! >> >> >> >> With 4 1 TB drives (for simpler examples) >> >> RAID5 =3D 3 TB available, 1 TB worth used in >> "parity". >> >> Fast reads, slow writes. >> >> RAID6 =3D 2 TB available, 2 TB worth used in >> "parity". >> >> Moderately fast reads, slow writes. >> >> RAID10 =3D 2 TB available, 2TB in duplicate copies >> (easier >> >> work than parity calculations).=C2=A0 Very fast >> reads, >> >> moderately fast writes. >> >> >> >> When you switch to, say, 8 drives, the numbers >> start to >> >> change a bit. >> >> RAID5 =3D 7TB available, 1 lost. >> >> RAID6 =3D 6TB available, 2 lost. >> >> RAID10 =3D 4TB available, 4 lost. >> >> >> > >> > Sorry, consider myself chastised for having missed the >> "Security is more important than performance" bit. I tend >> toward solutions that show the most value, and with 4 >> drives, it seems that I'd stick with the same "data >> security" only pick up the free speed of RAID10. =C2=A0Change >> when you get to 6 or more drives, if necessary. >> > >> > For data security, I can't answer for the UFS2 vs. >> ZFS. =C2=A0For hardware setup, let me amend everything I said >> above with the following: >> > >> > Since you are seriously focusing on data integrity, >> ignore everything I said but make sure you have good >> backups! =C2=A0:) >> > >> > Sorry, >> > -Rich >> >> No problem :) I've been doing some reading since I posted >> this >> question and it turns out that the controller will actually >> not allow >> me to create a RAID6 array using only 4 drives. 3ware >> followed the >> same reasoning as you; with 4 drives use RAID10. >> >> I know that you can migrate from one to the other when a >> 5th disk is >> added, but RAID10 can only handle 2 failed drives if they >> are from >> separate RAID1 groups. In this way, it is just slightly >> less resilient >> to failure than RAID6. With this new information, I think I >> may as >> well get one more 2TB drive and start with 6TB of RAID6 >> space. This >> will be less of a headache later on. >> >> - Max > > Just as a question: how ARE you planning on backing this beast up? =C2=A0= While I don't want to sound like a worry-wort, I have had odd things happen= at the worst of times. =C2=A0RAID cards fail, power supplies let out the m= agic smoke, users delete items they really want back... *sigh* Rsync over ssh to another server. Most of the data stored will never change after the first upload. A daily rsync run will transfer one or two gigs at the most. History is not required for the same reason; this is an append-only storage for the most part. A backup for the previous day is all that is required, but I will keep a weekly backup as well until I start running out of space. > A bit of reading shows that ZFS, if it's stable enough, has some really g= reat features that would be nice on such a large pile o' drives. > > See http://wiki.freebsd.org/ZFSQuickStartGuide > > I guess the last question I'll ask (as any more may uncover my ignorance)= is if you need to use hardware RAID at all? It seems both UFS2 and ZFS ca= n do software RAID which seems to be quite reasonable with respect to perfo= rmance and in many ways seems to be more robust since it is a bit more port= able (no specialized hardware). I've thought about this one a lot. In my case, the hard drives are in a separate enclosure from the server and the two had to be connected via SAS cables. The 9690SA-8E card was the best choice I could find for accessing an external SAS enclosure with support for 8 drives. I could configure it in JBOD mode and then use software to create a RAID array. In fact, I will likely do this to compare performance of a hardware vs. software RAID5 solution. The ZFS RAID-Z option does not appeal to me, because the read performance does not benefit from additional drives, and I don't think RAID6 is available in software. For those reasons I'm leaning toward a hardware implementation. If I go the hardware route, I'll try to purchase a backup controller in a year or two. :) > There are others who may respond with better information on that front. = I've been a strong proponent of hardware RAID, but have recently begun to r= ealize many of the reasons for that are only of limited validity now. Agreed, and many simple RAID setups (0, 1, 10) will give you much better performance in software. In my case, I have to have some piece of hardware just to get to the drives, and I'm guessing that hardware RAID5/6 will be faster than the closest software equivalent. Maybe my tests will convince me otherwise. - Max