From owner-freebsd-stable@FreeBSD.ORG Wed Apr 9 12:47:56 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 622A5106566C; Wed, 9 Apr 2008 12:47:56 +0000 (UTC) (envelope-from arnaud.houdelette@tzim.net) Received: from alaranth.tzim.net (unknown [IPv6:2001:41d0:1:968f:219:d1ff:fedf:4f29]) by mx1.freebsd.org (Postfix) with ESMTP id ECA9B8FC0A; Wed, 9 Apr 2008 12:47:55 +0000 (UTC) (envelope-from arnaud.houdelette@tzim.net) Received: from localhost ([127.0.0.1]) by alaranth.tzim.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1JjZi6-0003mQ-5P; Wed, 09 Apr 2008 14:47:30 +0200 Message-ID: <47FCBAFB.9060508@tzim.net> Date: Wed, 09 Apr 2008 14:47:55 +0200 From: Arnaud Houdelette User-Agent: Thunderbird 2.0.0.12 (Windows/20080213) MIME-Version: 1.0 To: lev@FreeBSD.org References: <396418019.20080409104542@serebryakov.spb.ru> In-Reply-To: <396418019.20080409104542@serebryakov.spb.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Authenticated-User: tzim@tzim.net X-Authenticator: plain Cc: freebsd-stable@freebsd.org Subject: Re: [SO]HO Software RAID5 server: which implementation should I choice? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Apr 2008 12:47:56 -0000 Lev Serebryakov a écrit : > Hello, freebsd-stable. > > Does somebody use some software RAID5 on FreeBSD in real production > system? > > I want to build storage server for my home: RAW photos, multi-layer > PhotoShop files and FLAC-encoded music consume a lot of space, and > they should be availible both from desktop & notebook. > > Also, all photo-content is unique, so I need some insuranse from > single HDD crash. I understand, that I will not safe from fire, PSU > failure and thing slike this. > > I selected hardware platform: Intel Q35-based MoBo with 6xSATA-II ports > (all of them is chipset-based, so no SiliconImage/JMicron/Whatever > crappy controllers), some low-end Core2Duo, 2Gb of memory. > Storage will be 5x500Gb WD HDDs for RAID + one small HDD for boot, system, > swap, etc. I want to have 2Tb (ok, not real Tb, I know) of "protected" > storage. > I want to have maximum speed via 1Gb network, because graphic files > are big and should open fast. Not as fast as local ones, I understand > that, but speeds about 12-15Mb/s is not enough for sure :) > > Only problem I see: which software RAID5 solution should I prefer? > FreeBSD-based, of course! > > I see these variants: > > (1) FreeBSD 6(7?) + graid3. Slow, one disk for checksums is bottleneck, > as far as I understand. > > (2) FreeBSD 6(7?) + gvinum/radi5. Is it stable enough?! Is it complete? > when I try it about 6 months ago in VMWare installation with 5 > virtual disks, I got panics and strange behaviour after "crashing" > one of virtual disks. > > (3) FreeBSD 6(7?) + graid5. Again, is it stable enough? There are > THREE versions of it. Which one should I prefer? There was long > thread about it some times ago without any clear conclusion. Does > something changed? > > (4) FreeBSD 7 + ZFS "zraid". And again: stability. Too many messages > about locks, crashes, etc. Code is experemental. Is it only for > 32 bit systems? > > (5) Do I miss something? > > (6) Solaris + ZFS? I don't want it, I know a little about Solaris > administaration, and I already have FreeBSD servers and routers. > > I know, that 3ware or Areca controllers are very good. I know, that > "gmirror" is very stable. But these variants are too expensive for > home server :( > > Does somebody use some software RAID5 on FreeBSD in real production? > Any advices? > Hi ! I personally use the 3 option for my personal Home File Server. I got approximatly the same usage for the file server (mostly video, music, photo). I built my own about 12 month ago. I reviewed the about the same variants as the one you propose : (1) Discarded for performance issues. Raid3 is slow. Really. (2) raid5/vinum is also slow. And as I understood at that time, recovery from lost hard drive wasn't easy enough for the freebsd niubee I was then. (4) ZFS wasn't there yet. But I did test it on a test VMWare, and wasn't convinced (mostly stability and memory issues). So I use geom_raid5. I sticked to the main distributions, which seemed more stable at the moment. The kernel module is fairly simple to build/install. Performance is (very) good for a software raid. I successfully switched the raid array from an i386 6.2-RELEASE to an an amd64 7.0-RELEASE (with motherboard and CPU change) without any assle. For the moment, I use one big UFS+SU (and snapshots) on the whole array. I successfuly tried unpplugging then replugging on of the drives, suddent power loss, using the array with a missing disk (degraded mode). All did work fine. (still, I use an UPS on the file server). The sole issue I had is with ataidle. I had to patch ata-disc.c to increase the IO timeout. Without, the raid5 module detected temporary disk loss and constantly launched rebuilds of the array. With 7.0, I wondered if I should use gjournal, but I'm not sure if it's really the way to go on a file system dedicated to store many big files. So I stick to soft updates. Current configuration is : / on a 2GB usb key /tmp on memory ports and source trees (and some portsnap stuff) on a small disk 4x250 GB sata for the raid5 array. AMD A64 3200+ and 512 GB DDRII Realtek Gigabit nics. Copy from raid5 to /dev/null gives about 100MB/s Copy from /dev/random to raid5 about 40MB/s I use samba shares. I get about 40MB/s in both ways from another computer on the network (enabling jumbo-frames gives a big boost). Hope my own story can help you in any way. Regards, Arnaud Houdelette