From owner-freebsd-current@FreeBSD.ORG Sun Jan 25 02:38:17 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 96037106566B for ; Sun, 25 Jan 2009 02:38:17 +0000 (UTC) (envelope-from terry@tmk.com) Received: from server.tmk.com (server.tmk.com [204.141.35.63]) by mx1.freebsd.org (Postfix) with ESMTP id 6D7F78FC0A for ; Sun, 25 Jan 2009 02:38:17 +0000 (UTC) (envelope-from terry@tmk.com) Received: from tmk.com by tmk.com (PMDF V6.3-x13 #37010) id <01N4OSG12R2O002BF9@tmk.com> for freebsd-current@freebsd.org; Sat, 24 Jan 2009 21:38:15 -0500 (EST) Date: Sat, 24 Jan 2009 21:21:41 -0500 (EST) From: Terry Kennedy In-reply-to: "Your message dated Sat, 24 Jan 2009 08:17:50 -0800" Sender: Terry Kennedy To: Freddie Cash Message-id: <01N4OT471TUS002BF9@tmk.com> MIME-version: 1.0 Content-type: TEXT/PLAIN; charset=UTF-8 References: <01N4NEOEB7LY00EQWX@tmk.com> Cc: freebsd-current@freebsd.org Subject: Re: Help me select hardware and software options for very large server X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Jan 2009 02:38:17 -0000 > We did something similar for our off-site, automated backups box. > > One box has 2x 2 GB CompactFlash in IDE adapters, the other has 2x 2 > GB USB flash drives. I assume this was for FreeBSD itself? I was concerned about write cycles (and, to a lesser extent, storage capacity) on CF or USB media. I haven't seen much degradation due to seeks when the FreeBSD space is a separate logical drive on the AMCC controller. That also gets me inherent RAID 6 protection (assuming I carve it from the main batch of drives). > - 64-bit FreeBSD 7.1-RELEASE (started with 7.1-STABLE from August 08) > - UFS for / partition, using gmirror across the two CF/USB drives > - ZFS for everything else This seems to mirror what I'll be doing. It is good to hear that this has been working well for you. > - uses rsync and ssh to backup 83 remote servers everynight, > creating ZFS snapshots every night > - uses rsync to transfer "snapshots" between the two servers In my case, the data will originate on one of the servers (instead of being backups of other servers) and will be synchronized with an off- site server (same hardware) via either 1Gbit or 10Gbit Ethernet (right now I have regular Gigabit hardware), but all I'd need is new add-in cards and an upgrade to the switches at each end of the fiber. The synchonization is currently dones nightly via rdiff-backup. But that package can occasionally loose its marbles, even with < 2TB of data, so I may have to consider alternatives. > The drives on each of the RAID controllers are configured as "Single > Disk Array", so they appear as 24 separate drives to the OS, but still > benefit from the controller's disk cache, management interface, and so > on (as compared to JBOD where it acts like nothing more than a SATA > controller). Hmmm. I was planning to use the hardware RAID 6 on the AMCC, for a number of reasons: 1) that gives me front-panel indications of broken RAID sets, controller-hosted rebuild, and so forth. 2) I'd be using fewer ZFS features (basically, just large partitions and snapshots) so if anything went wrong, I'd have a larger pool of expertise to draw on to fix things (all AMCC users, rather than all FreeBSD ZFS users). Did you consider this option and reject it? If so, can you tell me why? > The drives on one box are configured as 1 large 24-drive raidz2 in ZFS > (this box also has 12x 400 GB drives). > The drives on the other box are configured as 2 separate 11-drive > raidz2 arrays, with 2 hot spares. > The usable space on box boxes is 9 TB. So a single ZFS partition of 8TB would be manageable without long delays for backup snapshots? > Other than a bit of kernel tuning back in August/September, these > boxes have been running nice and smooth. Just waiting for either the > release of FreeBSD 8.0 or an MFC of ZFS v13 to 7-STABLE to get support > for auto-rebuild using hot spares. That's good to hear. What sort of tuning was involved (if it is still needed)? Thanks, Terry Kennedy http://www.tmk.com terry@tmk.com New York, NY USA