From owner-freebsd-fs@FreeBSD.ORG Tue Nov 13 21:19:27 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 59433D01 for ; Tue, 13 Nov 2012 21:19:27 +0000 (UTC) (envelope-from jas@cse.yorku.ca) Received: from bronze.cs.yorku.ca (bronze.cs.yorku.ca [130.63.95.34]) by mx1.freebsd.org (Postfix) with ESMTP id 0BFB58FC0C for ; Tue, 13 Nov 2012 21:19:26 +0000 (UTC) Received: from [130.63.97.125] (ident=jas) by bronze.cs.yorku.ca with esmtp (Exim 4.76) (envelope-from ) id 1TYNtG-0000Wd-2D; Tue, 13 Nov 2012 16:19:26 -0500 Message-ID: <50A2B95D.4000400@cse.yorku.ca> Date: Tue, 13 Nov 2012 16:19:25 -0500 From: Jason Keltz User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:10.0.9) Gecko/20121011 Thunderbird/10.0.9 MIME-Version: 1.0 To: Bob Friesenhahn Subject: Re: RHEL to FreeBSD file server References: <50A130B7.4080604@cse.yorku.ca> <20121113043409.GA70601@neutralgood.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -1.0 X-Spam-Level: - X-Spam-Report: Content preview: On 11/13/2012 12:41 PM, Bob Friesenhahn wrote: > On Mon, 12 Nov 2012, kpneal@pobox.com wrote: >> >> With your setup of 11 mirrors you have a good mixture of read and write >> performance, but you've compromised on the safety. The reason that >> RAID 6 >> (and thus raidz2) and up were invented was because drives that get used >> together tend to fail together. If you lose a drive in a mirror there is >> an elevated probability that the replacement drive will not be in place >> before the remaining leg of the mirror fails. If that happens then >> you've >> lost the pool. (Drive failures are _not_ independent.) > > Do you have a reference to independent data which supports this claim > that drive failures are not independent? The whole function of RAID > assumes that drive failures are independent. > > If drives share a chassis, care should be taken to make sure that > redundant drives are not in physical proximity to each other and that > they are supported via a different controller, I/O path, and power > supply. If the drives are in a different chassis then their failures > should be completely independent outside of a shared event like power > surge, fire, EMP, flood, or sun-spot activity. > > The idea of raidz2 vdevs of four drives each sounds nice but will > suffer from decreased performance and increased time to replace a > failed disk. There are always tradeoffs. [...] Content analysis details: (-1.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SHORTCIRCUIT Not all rules were run, due to a shortcircuited rule -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Nov 2012 21:19:27 -0000 On 11/13/2012 12:41 PM, Bob Friesenhahn wrote: > On Mon, 12 Nov 2012, kpneal@pobox.com wrote: >> >> With your setup of 11 mirrors you have a good mixture of read and write >> performance, but you've compromised on the safety. The reason that >> RAID 6 >> (and thus raidz2) and up were invented was because drives that get used >> together tend to fail together. If you lose a drive in a mirror there is >> an elevated probability that the replacement drive will not be in place >> before the remaining leg of the mirror fails. If that happens then >> you've >> lost the pool. (Drive failures are _not_ independent.) > > Do you have a reference to independent data which supports this claim > that drive failures are not independent? The whole function of RAID > assumes that drive failures are independent. > > If drives share a chassis, care should be taken to make sure that > redundant drives are not in physical proximity to each other and that > they are supported via a different controller, I/O path, and power > supply. If the drives are in a different chassis then their failures > should be completely independent outside of a shared event like power > surge, fire, EMP, flood, or sun-spot activity. > > The idea of raidz2 vdevs of four drives each sounds nice but will > suffer from decreased performance and increased time to replace a > failed disk. There are always tradeoffs. Hi Bob. Initially, I had one storage chassis, split between 2 LSI 9205-8e controllers with a 22 disk pool comprised of 11 mirrored vdevs. I think that I'm still slightly uncomfortable with the fact that 2 disks, which were all purchased at the same time, could essentially die at the same time, killing my whole pool. Yet, while moving to raidz2 would allow better redundancy, I'm not sure if the raidz2 rebuild time and decrease in performance would be worth it.. After all, this would be a primary file server, without which, I'd be in big trouble.. As a result, I'm considering this approach.. I'll buy another md1220, a few more disks, add another 9205-8e card... and use triple mirrored vdevs instead of dual.... I only really need about 8 x 900 GB storage, so if I can multiply this by 3, add a few spares... in addition, each set of disks would be on its own controller. I should be able to lose a controller, and maintain full redundancy.... I should be able to lose an entire disk enclosure and still be up ... I believe read performance would probably go up, but I suspect that write performance would suffer a little -- not sure exactly by how much. When I first speced out the server, the LSI 9205-8e was the best choice for a card since the PCI Express 3 HBAs (which the R720 supports) weren't out yet ... now, there's the LSI 9207-8e which is PCIE3, but I guess it doesn't make much sense to buy one of those now that I have another 2 x LSI 9205-8e cards already ... (a shame though since there is less than $50 difference between the cards). By the way - on another note - what do you or other list members think of the new Intel SSD DC S3700 as ZIL? Sounds very promising when it's finally available. I spent a lot of time researching ZILs today, and one thing I can say is that I have a major headache now because of it!! Jason.