From owner-freebsd-questions@FreeBSD.ORG Fri Nov 28 20:18:19 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF76C1065677 for ; Fri, 28 Nov 2008 20:18:19 +0000 (UTC) (envelope-from SRS0=TiVw6V=4D=shell.siscom.net=vogelke@siscom.net) Received: from lamorack.siscom.net (lamorack.siscom.net [209.251.2.116]) by mx1.freebsd.org (Postfix) with ESMTP id 8A57B8FC13 for ; Fri, 28 Nov 2008 20:18:19 +0000 (UTC) (envelope-from SRS0=TiVw6V=4D=shell.siscom.net=vogelke@siscom.net) Received: from shell.siscom.net ([209.251.2.80]) by lamorack.siscom.net with esmtp (Exim 4.62) (envelope-from ) id 1L69Lh-0005A7-C6 for freebsd-questions@freebsd.org; Fri, 28 Nov 2008 14:49:57 -0500 Received: by shell.siscom.net (Postfix, from userid 2198) id 506E4115551; Fri, 28 Nov 2008 14:49:57 -0500 (EST) Received: by kev.msw.wpafb.af.mil (Postfix, from userid 32768) id 7B17FB7C5; Fri, 28 Nov 2008 14:45:31 -0500 (EST) To: freebsd-questions@freebsd.org In-reply-to: <139b44430811280548x36915301i766bfb15f162c8ca@mail.gmail.com> (valentin.bud@gmail.com) Organization: Oasis Systems Inc. X-Disclaimer: I don't speak for the USAF or Oasis. X-GPG-ID: 1024D/711752A0 2006-06-27 Karl Vogel X-GPG-Fingerprint: 56EB 6DBF 4224 C953 F417 CC99 4C7C 7D46 7117 52A0 Message-Id: <20081128194531.7B17FB7C5@kev.msw.wpafb.af.mil> Date: Fri, 28 Nov 2008 14:45:31 -0500 (EST) From: vogelke+software@pobox.com (Karl Vogel) Subject: Re: 5 TB server X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: vogelke+software@pobox.com List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Nov 2008 20:18:19 -0000 >> On Fri, 28 Nov 2008 15:48:45 +0200, >> "Valentin Bud" said: V> I have to come up with a solution for a company that has as we speak 4 TB V> of data spread among 3 computers with lots of HDDs. Of course I've V> recommend them to buy a server for that storage capacity and for data V> organization. Good idea. We had a similar problem here; a SuperMicro server tanked, and the company that provided the warranty went bankrupt, so all we had were 12 perfectly good 400-Gb SATA drives and nowhere to put them. The drives were the big-cost item, so I didn't want to just dump them. We finally bought two empty IBM x3400 8-bay enclosures plus some IBM SAS 3.5" hot-swap trays, and it's working like a charm. V> I thought of going on the ZFS way (on FreeBSD of course) with some raidz. In my experience, completely new filesystems or operating systems need at least 5 years in the field to weed out all the weird corner-cases. I might trust ZFS on Sun hardware (*with* vendor support) at this point, but I'd wait awhile before trying it on anything else. This isn't a slam at ZFS or the FreeBSD porters, it's just recognition of the fact that some types of software development are *not* time- compressible, regardless of who's doing the work. V> One of the problems is that the server will stay in their office so it V> has to be quite silent. Not a good idea, especially if this data is their bread-and-butter. You can walk out the door with a system this size on your shoulder, so I'd recommend a locked room with reasonable cooling and *clean* power. You don't need a 10-foot-tall zillion-dollar Liebert A/C, but you absolutely need a UPS that can take care of power spikes; the more moderate the environment, the less likely you are to have a hardware failure. I don't use disk mirroring because 99% of our problems come from humans rather than hardware. If someone zaps the wrong file, a mirror will simply replicate that mistake; we have two matching servers in separate rooms, and we run rsync nightly to back up the production box without deleting any files. I also run hourly backups on the production box to store anything that's been modified in the last 60 minutes, which gives us a nice file history and takes care of most recovery problems. With two servers, I can use basic UFS filesystems and get fine I/O performance with minimum maintenance. -- Karl Vogel I don't speak for the USAF or my company The first product Motorola started to develop was a record player for cars. At that time, the most known player on the market was Victrola, so they called themselves Motorola. --possibly-true item for a lull in conversation