Date: Thu, 31 May 2012 11:24:41 -0700 (PDT) From: Dennis Glatting <dg@pki2.com> To: Oscar Hodgson <oscar.hodgson@gmail.com> Cc: freebsd-questions@freebsd.org Subject: Re: Anyone using freebsd ZFS for large storage servers? Message-ID: <alpine.BSF.2.00.1205311102290.48219@btw.pki2.com> In-Reply-To: <CACxnZKM__Lt9LMabyUC_HOCg2zsMT=3bpqwVrGj16py1A=qffg@mail.gmail.com> References: <CACxnZKM__Lt9LMabyUC_HOCg2zsMT=3bpqwVrGj16py1A=qffg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 31 May 2012, Oscar Hodgson wrote: > The subject is pretty much the question. Perhaps there's a better > place to be asking this question ... > > We have (very briefly) discussed the possibility of using FreeBSD > pizza boxes as a storage heads direct attached to external JBOD arrays > with ZFS. In perusing the list, I haven't stumbled across indications > of people actually doing this. External JBODs would be running 24 to > 48TB each, roughly. There would be a couple of units. The pizza > boxes would be used for computational tasks, and nominally would have > 8 cores and 96G+ RAM. > > Obvious questions are hardware compatibility and stability. I've set > up small FreeBSD 9 machines with ZFS roots and simple mirrors for > other tasks here, and those have been successful so far. > > Observations would be appreciated. > mc: real memory = 120259084288 (114688 MB) FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs FreeBSD/SMP: 4 package(s) x 16 core(s) mc> zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT disk-1 14.5T 4.95T 9.55T 34% 1.00x ONLINE - disk-2 270G 297M 270G 0% 1.00x ONLINE - disk-1, RAIDz1, uses Hitachi 4TB drives. iirc: real memory = 68719476736 (65536 MB) FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs FreeBSD/SMP: 2 package(s) x 16 core(s) iirc> zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT disk-1 18.1T 6.70T 11.4T 36% 1.00x ONLINE - disk-2 5.44T 3.05G 5.43T 0% 1.00x ONLINE - disk-1, RAIDz1, uses a bunch of 2TB drives I have another storage server named bd3 that has a RAIDz2 array of 2.5T drives (11 of them, IIRC) but it is presently powered down for maintenance. btw: real memory = 25769803776 (24576 MB) FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 SMT threads btw> zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT disk-1 9.06T 97.3G 8.97T 1% 1.00x ONLINE - disk-2 9.06T 5.13T 3.93T 56% 1.00x ONLINE - Those are smaller RAIDz1 arrays of 1TB and 2TB drives, IIRC. I also have three other systems, over clocked to 4GHz with 16GB of RAM and presently powered off, each with 3 or 4 2TB disks RAIDz1. None of these systems have external arrays. The storage systems use common technologies, such as NFS, to export their space but their primary mission is manipulating (sort-of) big data and crypto attacks, though one is being converted to a Hadoop node for experimentation. I have only had four issues over the past year and a half: 1) It is important to keep your ZFS patches up to date and the firmware in you controllers up to date. Failure to do this results in a => :( 2) Under heavy I/O my systems freeze for a few seconds. I haven't looked into why but they are completely unresponsive. Note I am also using compressed volumes (gzip), which puts a substantual load on the kernel. 3) I have had a number of disk failures -- not too many and not too few. These are merely an annoyance with no loss of data. 4) In two systems I use OCZ Revo drives. After several months of operating they go Tango Uniform, requiring a system boot where they return from the dead. None of my other SSD technologies exhibit the same problem.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.1205311102290.48219>