From owner-freebsd-questions@FreeBSD.ORG Mon Jul 14 15:04:46 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 75A9D1065673 for ; Mon, 14 Jul 2008 15:04:46 +0000 (UTC) (envelope-from david.robillard@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.232]) by mx1.freebsd.org (Postfix) with ESMTP id 55F9A8FC0A for ; Mon, 14 Jul 2008 15:04:46 +0000 (UTC) (envelope-from david.robillard@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so5924491rvf.43 for ; Mon, 14 Jul 2008 08:04:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:mime-version:content-type:content-transfer-encoding :content-disposition; bh=jQnrRgGpkJahKXDxpkzqV7woJpXIVvNilSDHDlw2cAQ=; b=BBx6H+cpuT0JI2ZPf6CdqrGViFYk2Tl6poitAakaedNor8BthSxYwuBrKsBhEfQ7xb pP9vYcB5E0v1faKptLtSM0ypkwdi/SR855Mcu0kAOZdDnDms14Z654xueMcl/L4e2lTH dKPGe7rT5utG/8Qgo7Ob7abDgaLHo0F3ocmtA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:mime-version:content-type :content-transfer-encoding:content-disposition; b=v9d+4Vy4FxnPf1tfwchwkN3CbOqAUx9RtT4pa7gETOC5hW4uSBitgiVHWWcG4DZOuZ df7cwDehu3RQumx8ZDsJFd51UZdwVYBw1t8JSgDqnq8/jpPMBPju3Qbd5mdJkpp3aKl7 G+5X1B9B5YaX5SH9+5G4rhT6Cf2J2oPutHrGQ= Received: by 10.141.29.18 with SMTP id g18mr6660101rvj.162.1216047885698; Mon, 14 Jul 2008 08:04:45 -0700 (PDT) Received: by 10.141.19.8 with HTTP; Mon, 14 Jul 2008 08:04:45 -0700 (PDT) Message-ID: <226ae0c60807140804g467facdn37756a244e0b7de@mail.gmail.com> Date: Mon, 14 Jul 2008 11:04:45 -0400 From: "David Robillard" To: "Maurice Volaski" MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Cc: FreeBSD Questions Subject: Re: Would ZFS and gmirror work well together in a two-node failover cluster? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jul 2008 15:04:46 -0000 > I am looking to put together a two-node high-availability cluster > where each node has identical data storage consisting of a set of > internal data drives (separate from the boot drive). I want ZFS to > manage the drives as a JDBOD in a RAIDZ2 configuration. Thus, if an > individual drive misbehaves or fails, ZFS detects and handles the > fault. > > But I'm also looking to mirror this entire setup in real time to a > second identical server. > > Basically, my question is can this work well on FreeBSD while taking > full advantage of ZFS? > > Specifically, my understanding is that the only way to handle the > real time mirror is with gmirror and ggated, but it's not clear how > gmirror would interact with ZFS. > > I am assuming that gmirror operates only on individual drives, so if > I had a set of 24 drives on each server, there would be 24 mirrored > drive pairs. > > One concern I have is that this setup could run into trouble with > gmirror's potentially sabotaging ZFS's RAIDZ2. For example, when a > drive starts failing, won't gmirror see it before ZFS does and take > the unfavorable action of substituting the corresponding drive in the > failover server in subsequent I/O, leaving ZFS's RAIDZ2 out of the > loop? > > This is just one particular scenario, but in general, it's not > entirely clear that it's possible to have fine-grained control of > when, how much and in what direction gmirror manages synchronization > among drive pairs. Hello Maurice, Which type of connection do you intend to use for the shared storage JBOD? SAN or direct attached SCSI? Don't forget to change the SCSI initiator ID on one of the nodes if you go the direct attached SCSI road. I had this setup running back in 1999 with two Solaris boxes using Solstice Disk Suite with shared disks. Both nodes knew about the existance of the other and hence it worked quite well. But I don't know if it can work with two FreeBSD nodes? Now for the filesystem choice, keep in mind that ZFS is not a native cluster, distributed, or parallel file system and cannot provide concurrent access from multiple hosts as ZFS is a local file system. Which means your two node cluster won't be active/active. You'll have an active node and a failover node. That may be alright or it may not. Depends on your application, how deep your pockets are and your the level of risk your organization is willing to live with. You might want to take a look at clustered file systems for your setup. Check out Lustre (http://wiki.lustre.org/) or OpenGFS (http://opengfs.sourceforge.net/) for instance. If your cluster requires mostly reads and not much write, check out OpenAFS (http://www.openafs.org/) which is a distributed filesystem. You could always use NFS too, but then it depends on where you want to deploy the cluster, as NFS is rather hard to secure. Now if we come back to the problem at hand, mainly using zfs under gmirror. I've never heard of anyone using this. It does sound a bit strange to me since both zfs and gmirror will do mirroring. I would advise to test and retest very carefully before you go into production with such a setup. If you do try it, I'd be interested in reading what you've tried and what conclusions you came to. Good luck! HTH, David -- David Robillard UNIX systems administrator & Oracle DBA CISSP, RHCE & Sun Certified Security Administrator Montreal: +1 514 966 0122 If you receive something that says "Send this to everyone you know", then please pretend you don't know me.