From owner-freebsd-cluster@FreeBSD.ORG Mon Sep 26 20:38:09 2005 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2F96F16A496; Mon, 26 Sep 2005 20:38:09 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6D82C43D58; Mon, 26 Sep 2005 20:38:07 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j8QKc6nK077563; Mon, 26 Sep 2005 15:38:06 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <43385C29.5060406@centtech.com> Date: Mon, 26 Sep 2005 15:38:01 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.11) Gecko/20050914 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Isaac Levy References: <20050924141025.GA1236@uk.tiscali.com> In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/1102/Sun Sep 25 09:04:56 2005 on mh1.centtech.com X-Virus-Status: Clean Cc: freebsd-isp@freebsd.org, freebsd-cluster@freebsd.org Subject: Re: Options for synchronising filesystems X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Sep 2005 20:38:09 -0000 Isaac Levy wrote: > Hi Brian, All, > > This email has one theme: GEOM! :) > > On Sep 24, 2005, at 10:10 AM, Brian Candler wrote: > >> Hello, >> >> I was wondering if anyone would care to share their experiences in >> synchronising filesystems across a number of nodes in a cluster. I >> can think >> of a number of options, but before changing what I'm doing at the >> moment I'd >> like to see if anyone has good experiences with any of the others. >> >> The application: a clustered webserver. The users' CGIs run in a chroot >> environment, and these clearly need to be identical (otherwise a CGI >> running >> on one box would behave differently when running on a different box). >> Ultimately I'd like to synchronise the host OS on each server too. >> >> Note that this is a single-master, multiple-slave type of filesystem >> synchronisation I'm interested in. > > > I just wanted to throw out some quick thoughts on a totally different > approach which nobody has really explored in this thread, solutions > which are production level software. (Sorry if I'm repeating things or > giving out info yall' already know:) > > -- > Geom: > http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/geom- intro.html > > The core Disk IO framework for FreeBSD, as of 5.x, led by PHK: > http://www.bsdcan.org/2004/papers/geom.pdf > > This framework itself is not as useful to you as the utilities which > make use of it, > > -- > Geom Gate: > http://kerneltrap.org/news/freebsd?from=20 > > Network device-level client/server disk mapping tool. > (VERY IMPORTANT COMPONENT, it's reportedly faster, and more stable than > NFS has ever been- so people have immediately and happily deployed it > in production systems!) > > -- > Gvinum and Gmirror: > > Gmirror > http://people.freebsd.org/~rse/mirror/ > http://www.ie.freebsd.org/doc/en_US.ISO8859-1/books/handbook/geom.html > > (Sidenote: even Greg Lehey (original author of Vinum), has stated that > it's better to use Geom-based tools than Vinum for the forseeable future.) > > -- > In a nutshell, to address your needs, let me toss out the following > example setup: > > I know of one web-shop in Canada, which is running 2 machines for every > virtual cluster, in the following configuration: > > 2 servers, > 4 SATA drives per box, > quad copper/ethernet gigabit nic on each box > > each drive is mirrored using gmirror, over each of the gigabit ethernet > nics > each box is running Vinum Raid5 across the 4 mirrored drives > > The drives are then sliced appropriately, and server resources are > distributed across the boxes- with various slices mounted on each box. > The folks I speak of simply have a suite of failover shell scripts > prepared, in the event of a machine experiencing total hardware failure. > > Pretty tough stuff, very high-performance, and CHEAP. > > -- > With that, I'm working towards similar setups, oriented around > redundant jailed systems, with an eventual end to tie CARP (from pf) > into the mix to make for nearly-instantaneous jailed failover > redundancy- (but it's going to be some time before I have what I want > worked out for production on my own). > > Regardless, it's worth tapping into the GEOM dialogues, as there are > many new ways of working with disks coming into existence- and the GEOM > framework itself provides an EXTREMELY solid base to bring 'exotic' > disk configurations up to production level quickly. > (Also noteworthy, there's a couple of encrypted disk systems based on > GEOM emerging now too...) I think the original poster (and I at least) knew about this already, but what I still fail to see is how you can get several machines using the same data at the same time, and still do updates to that data? The only way I know of is to use a syncing tool (like rsync) or a shared filesystem (like NFS, or CXFS, or Polyserve FS, opengfs, etc), none of which run on FreeBSD. What I read from above, is a redundant server setup, not a high-performance setup (meaning multiple machines serving the same data to many clients). If I'm missing something, please fill me in.. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------