Date: Tue, 18 Feb 2014 21:43:39 -0800 From: <dteske@FreeBSD.org> To: <dweimer@dweimer.net> Cc: dteske@FreeBSD.org, questions@freebsd.org, freebsd@fongaboo.com, owner-freebsd-questions@freebsd.org Subject: RE: ZFS on Hardware RAID controller Message-ID: <0fe101cf2d35$8b988960$a2c99c20$@FreeBSD.org> In-Reply-To: <0fdf01cf2d34$65d9de00$318d9a00$@FreeBSD.org> References: <alpine.BSF.2.00.1402111550250.5347@helix.wtfayla.net> <CAF6rxgku91Rz037H-rpBCzc9G5bu5=KOiAm1Wms3%2B1Xcz%2BJjtQ@mail.gmail.com> <alpine.BSF.2.00.1402181717580.4903@helix.wtfayla.net> <CAF6rxgkVWm9AbM8SWCA3oHg8DHwSrRbSTW%2Ba%2BE9o%2Bky9SKNc2w@mail.gmail.com> <alpine.BSF.2.00.1402182250080.4903@helix.wtfayla.net> <0fdd01cf2d29$dca0aba0$95e202e0$@FreeBSD.org> <4e4aa28301cedd61cb5693e79a90d0ab@dweimer.net> <0fdf01cf2d34$65d9de00$318d9a00$@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
> -----Original Message----- > From: dteske@FreeBSD.org [mailto:dteske@FreeBSD.org] > Sent: Tuesday, February 18, 2014 9:35 PM > To: dweimer@dweimer.net; dteske@freebsd.org > Cc: freebsd@fongaboo.com; questions@freebsd.org; owner-freebsd- > questions@freebsd.org; daver@vicor.com > Subject: RE: ZFS on Hardware RAID controller > > > > > -----Original Message----- > > From: dweimer [mailto:dweimer@dweimer.net] > > Sent: Tuesday, February 18, 2014 8:57 PM > > To: dteske@freebsd.org > > Cc: freebsd@fongaboo.com; questions@freebsd.org; owner-freebsd- > > questions@freebsd.org > > Subject: RE: ZFS on Hardware RAID controller > > > > On 02/18/2014 10:20 pm, dteske@FreeBSD.org wrote: > > >> -----Original Message----- > > >> From: freebsd@fongaboo.com [mailto:freebsd@fongaboo.com] > > >> Sent: Tuesday, February 18, 2014 7:58 PM > > >> To: questions@freebsd.org > > >> Subject: Re: ZFS on Hardware RAID controller > > >> > > >> > > >> I was speaking to someone else about this today, and it eventually > > >> became apparent that we were getting into a sort-of > > > Abbott-and-Costello-Who's-on-First > > >> confusion... because apparently people can mean different things > > >> when they use the term 'JBOD'. > > >> > > >> What I have always meant when I say 'JBOD' is a (not really) RAID > > >> mode > > > that > > >> simply concatenates the drives into one volume in a serial fashion, > > >> ie. > > >> 'spanning'. Most RAID controllers and RAID-enabled NAS units that I > > >> have interacted with in my life have offered this mode and referred > > >> to it as > > > 'JBOD'. > > >> > > > > > > This is not entirely correct. JBOD and RAID-SPAN are two different > > > things. > > > Your > > > controller either supports one, or both (aside a host of other > > > options, such as RAID-1, RAID-10; and if often RAID-5, maybe RAID-6). > > > > JBOD, stands for "just a bunch of disks", > [Devin Teske] > > Correct, I stated as much. > > its defined as one or more disks > > spanned together or concatenated in a linear fashion. > > > [Devin Teske] > > I beg to differ. Google begs to differ. Type in "JBOD definition" to Google > and you get... > > A collection of hard disks that aren't configured according to RAID; A hard > disk enclosure for several disks, especially one lacking a RAID controller. > > I don't know where you got the so-called definition of "spanned together..." > which directly contradicts JBOD. > > > > > > > RAID-SPAN is RAID-0. Which is as you describe "simply concatenates the > > > drives into one volume in a serial fashion, ie. 'spanning'." > > > > Raid-0 is a stripe, its splits the data evenly across the disks, instead > of spanning > > them. > > > [Devin Teske] > > How are you classifying "spanning" as something different than RAID-0? > > I advise you that some RAID controllers call RAID-0 "SPAN" and RAID-1 > "MIRROR". > If you've not personally seen this, I offer it as evidence that they are one > and the > same. On a more technical note, "span" could technically be used for any > topology > that causes capacity to increase as you add more disks (but I inform you, I > have > _never_ seen any RAID controller call any parity-balanced topology; e.g., > RAID-5; > a "SPAN" but rather they call it a RAID-5; only RAID-1 has ever been called > SPAN). [Devin Teske] Minor typo in that last bit... "only RAID-0 has ever been called SPAN" > > I'm somewhat mystified by your above statement that somehow implies that > RAID-0 is not a spanned topology. > > > > > > > > JBOD on the other hand stands for "Just a Bunch of Disks" and is not > > > the same as "concatenating the disks" but rather allows your > > > controller to throw the drives at the Operating System with the > > > attitude of "here, you do it then." > > > > > > We can actually therefore call ZFS's RAID capabilities, "software > > > RAID". > > > However, it's a very _good_ software RAID that breaks the old adage > > > that says "hardware RAID is faster than software RAID." > > > > JBOD has been generally used as in putting just one disk through to the > O/S, > > and Spanning is generally used to describe multiple disks concatenated > linearly > > however its definition is interchangeable. > > > [Devin Teske] > > No sir, the definitions of JBOD versus SPAN are never interchangeable. I'm > not > here to pick a fight or argue, but you're misinformed. > > > > FWIW, if the raid controller doesn't have a JBOD/SPAN option, > [Devin Teske] > > I repeat that JBOD and SPAN are not the same thing. > > > sometimes you > > can define a single disk Raid-0 instead, > [Devin Teske] > > This would be a SPAN of 1 disk (counter-intuitive as this may sound). Again, > it > all depends on what your controller calls it -- be it 3ware (where I've seen > it) > or something else. > > > if there is only one drive its functionally > > equivalent. Keep in mind though that often the O/S will see a Raid > controller > > LUN device, which likely wont except SMART commands, and might not > display > > correct sector size etc. All depends on the Raid controller, so make sure > you > > correctly define partitions to match the drives or you may get sub-par > > performance. > > > [Devin Teske] > > That much is true. > -- > Devin > > > > > > > To highlight the difference of "JBOD" versus "spanning" (aka RAID-0; > > > aka > > > RAID- > > > SPAN), take the following use-case example: > > > > > > 1. You take 12 drives and apply spanning logic on the controller > > > > > > 2. You take a controller and put it into JBOD mode > > > > > > In scenario one (1), your OS still sees a single drive. > > > In scenario two (2), your OS sees all twelve drives. > > > > > > Just wanted to clarify that putting a controller into JBOD-mode is not > > > to be confused with RAID-0 or spanning (which would still be utilizing > > > the controller's software; the best benefit of ZFS comes from letting > > > it have direct access to each/every disk). > > > > > >> In this kind of mode, the motherboard and the OS still thinks it sees > > >> only > > > a > > >> single volume. So now I am gathering that this is also not ideal for > > >> ZFS, > > > since it > > >> would still not be aware of multiple physical volumes and be unable > > >> to > > > optimize > > >> accordingly. > > >> > > >> I'm learning for the first time that sometimes 'JBOD' can also refer > > >> to > > > each > > >> individual drive being mounted separately at least as far as the > > > controller and > > >> the motherboard is concerned. > > >> > > >> I just want to confirm 100% that this is how you are recommending > > >> multiple drives be configured for ZFS. Because when I started the > > >> thread I was > > > thinking > > >> of JBOD as 'spanning'. > > >> > > > > > > Correct; JBOD never means span (which is RAID-0). Rather, JBOD usually > > > means using non-RAID capable hardware with a RAID-capable operating > > > system (e.g., using ZFS). We are again in-essence talking about using > > > software RAID to create a software pool. > > > > > > I don't want to go drawing similarities to many other software RAID > > > solutions, because ZFS is truly in a class its own. But if you're > > > familiar with the concepts of setting up a software RAID (mdadm, > > > vinum, graid*, etc.) then ZFS should be more familiar. That's not to > > > say that you need to know these things to use ZFS (in practice, ZFS > > > has predictable commands with predictable syntax) but if you have ever > > > created a software RAID you will be in unique position to better > > > understand the JBOD mentality. > > > > > > So in transitioning to a test platform that uses "JBOD" you > > > essentially have two choices that look attractive... > > > > > > + Ditch the RAID card and use a standard adapter for connecting your > > > drives > > > > > > + Find a way to change your RAID controller to export all the disks > > > NB: This may involve using a LSI provided utility or flashing a QLogic > > > card > > > NB: If you give us your exact card info, someone may have information > > > on how to transition it into JBOD mode for ZFS > > > > > > + Create a bunch of single-disk RAID-0 arrays (12 disks? 12x > > > single-disk > > > SPAN > > > or RAID-0 arrays; producing 12x single LUNs for use in ZFS). > > > > > > Hope this gives some ideas. > > > -- > > > Devin > > > > > >> On Tue, 18 Feb 2014, Eitan Adler wrote: > > >> > > >> > On Tue, Feb 18, 2014 at 5:27 PM, <freebsd@fongaboo.com> wrote: > > >> >> > > >> >> When we spoke, you noted that when installing ZFS on multiple > > >> >> disks connected to a hardware RAID controller, it is best to > > >> >> config it to > > > JBOD. > > >> > > > >> > There are a few reasons for this. > > >> > (a) Hardware RAID serves as a single point of failure: if the > > >> > contoller dies you have neither disk > > >> > (b) As Andrew noted , using hardware RAID means that ZFS won't be > > >> > able to tell which disk is which. The ZFS management tools won't > > >> > work as expected (they will show only one disk). > > >> > (c) Since ZFS implements RAID itself it can use knowledge about the > > >> > physical disks for better performance > > >> > > > >> > Also see: > > >> > https://urldefense.proofpoint.com/v1/url?u=https://en.wikipedia.org > > >> > > > > /wiki/ZFS%23ZFS_and_hardware_RAID&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D > > %3 > > >> > > > > D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=rrGF66zgDe1thlbKNL > > 85L2 > > >> > > > > J80eLI6wBr0AEIltTFLaM%3D%0A&s=4799b4ef2dd7cc6060a9545178e315329de > > 3a > > >> > 8909141aff0d90992798c0379ee > > >> > > > >> >> I tried to explain this to a colleague, but they were skeptical. > > >> >> Would you (or anyone) be willing to give me a one or two line > > >> >> sales-pitch on > > >> > > > >> > "ZFS does RAID better than the controller." > > >> > > > >> >> why one > > >> >> should abandon traditional notions of RAID performance in favor of > > >> >> allowing ZFS to do disk management? > > >> > > > >> > The goal isn't to give up on RAID but move its implementation to ZFS. > > >> > > > >> > -- > > >> > Eitan Adler > > >> > Source, Ports, Doc committer > > >> > Bugmeister, Ports Security teams > > >> > _______________________________________________ > > >> > freebsd-questions@freebsd.org mailing list > > >> > https://urldefense.proofpoint.com/v1/url?u=http://lists.freebsd.org > > >> > /mailman/listinfo/freebsd- > > questions&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D > > >> > > > > %3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=rrGF66zgDe1thlbK > > NL85 > > >> > > > > L2J80eLI6wBr0AEIltTFLaM%3D%0A&s=6a7f3cd494e1bb3cfbdaf4ab0797f7fc97e > > >> > da2a1f02e097d7c6924792dc8fc7d To unsubscribe, send any mail to > > >> > "freebsd-questions- > > >> unsubscribe@freebsd.org" > > >> > > > >> _______________________________________________ > > >> freebsd-questions@freebsd.org mailing list > > >> https://urldefense.proofpoint.com/v1/url?u=http://lists.freebsd.org/m > > >> ailman/listinfo/freebsd- > > questions&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D% > > >> > > > 0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=rrGF66zgDe1thlbKNL85L > > 2J80e > > >> > > > LI6wBr0AEIltTFLaM%3D%0A&s=6a7f3cd494e1bb3cfbdaf4ab0797f7fc97eda2a1f > > 02 > > >> e097d7c6924792dc8fc7d > > >> To unsubscribe, send any mail to > > > "freebsd-questions-unsubscribe@freebsd.org" > > > > > > _____________ > > > The information contained in this message is proprietary and/or > > > confidential. If you are not the intended recipient, please: (i) > > > delete the message and all copies; (ii) do not disclose, distribute or > > > use the message in any manner; and (iii) notify the sender > > > immediately. In addition, please be aware that any message addressed > > > to our domain is subject to archiving and review by persons other than > > > the intended recipient. Thank you. > > > _______________________________________________ > > > freebsd-questions@freebsd.org mailing list > > > https://urldefense.proofpoint.com/v1/url?u=http://lists.freebsd.org/ma > > > ilman/listinfo/freebsd- > > questions&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A > > > > > > &r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=rrGF66zgDe1thlbKNL85L2J8 > > 0eLI6 > > > > > > wBr0AEIltTFLaM%3D%0A&s=6a7f3cd494e1bb3cfbdaf4ab0797f7fc97eda2a1f02 > > e097 > > > d7c6924792dc8fc7d > > > To unsubscribe, send any mail to > > > "freebsd-questions-unsubscribe@freebsd.org" > > > > -- > > Thanks, > > Dean E. Weimer > > > > > https://urldefense.proofpoint.com/v1/url?u=http://www.dweimer.net/&k=%2Fb > > > kpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D > > > %3D%0A&m=rrGF66zgDe1thlbKNL85L2J80eLI6wBr0AEIltTFLaM%3D%0A&s=353 > > ac15c8c6902bd0ac8552e57e039cba7fd0d09a06ac5860a9e2b0cde67d897 > > > _____________ > The information contained in this message is proprietary and/or confidential. If > you are not the intended recipient, please: (i) delete the message and all > copies; (ii) do not disclose, distribute or use the message in any manner; and > (iii) notify the sender immediately. In addition, please be aware that any > message addressed to our domain is subject to archiving and review by persons > other than the intended recipient. Thank you. _____________ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?0fe101cf2d35$8b988960$a2c99c20$>