From owner-freebsd-stable@FreeBSD.ORG Wed Mar 24 22:19:54 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CCC5E1065672 for ; Wed, 24 Mar 2010 22:19:54 +0000 (UTC) (envelope-from freebsd-stable@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 5A19D8FC0A for ; Wed, 24 Mar 2010 22:19:54 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1NuYvK-0000uF-5H for freebsd-stable@freebsd.org; Wed, 24 Mar 2010 23:19:38 +0100 Received: from 78-1-170-106.adsl.net.t-com.hr ([78.1.170.106]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 24 Mar 2010 23:19:38 +0100 Received: from ivoras by 78-1-170-106.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 24 Mar 2010 23:19:38 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-stable@freebsd.org From: Ivan Voras Date: Wed, 24 Mar 2010 23:19:16 +0100 Lines: 45 Message-ID: References: <4BAA3409.6080406@ionic.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: 78-1-170-106.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.21 (X11/20090612) In-Reply-To: Subject: Re: Multi node storage, ZFS X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Mar 2010 22:19:54 -0000 Freddie Cash wrote: > On Wed, Mar 24, 2010 at 8:47 AM, Michal wrote: > >> I wrote a really long e-mail but realised I could ask this question far >> far easier, if it doesn't make sense, the original e-mail is bellow >> >> Can I use ZFS to create a multinode storage area. Multiple HDD's in >> Multiple servers to create one target of, for example, //officestorage >> Allowing me to expand the storage space when needed and clients being >> able to retrieve data (like RAID0 but over devices not HDD) >> >> Here is an example I found which is where I'm getting some ideas from >> http://www.howtoforge.com/how-to-build-a-low-cost-san-p3 >> >> Horribly, horribly, horribly complex. But, then, that's the Linux world. > :) > > Server 1: bunch of disks exported via iSCSI > Server 2: bunch of disks exported via iSCSI > Server 3: bunch of disks exported via iSCSI > > "SAN" box: uses all those iSCSI exports to create a ZFS pool For what it's worth - I think this is a good idea! iSCSI and ZFS make it extraordinarily flexible to do this. You can have a RAIS - redundant array of inexpensive servers :) For example: each server box hosts 8-12 drives - use a hardware controller with RAID6 and a BBU to create a single volume (if FreeBSD booting issues allow, but that can be worked around). Export this volume via iSCSI. Repeat for the rest of the servers. Then, on the client, create a RAIDZ. or if you trust your setup that much. a straight striped ZFS volume. If you do it the RAIDZ way, one of your storage servers can fail completely. As you need more space, add more servers in batches of three (if you did RAIDZ, else the number doesn't matter), add them to the client as usual. The "client" in this case can be a file server, and you can achieve failover between several of those by using e.g. carp, heartbeat, etc. - if the master node fails, some other one can reconstitute the ZFS pool ad make it available. But, you need very fast links between the nodes, and I wouldn't use something like this without extensively testing the failure modes.