From owner-freebsd-fs@FreeBSD.ORG  Thu Dec  9 17:40:18 2010
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 1B40E1065670
	for <freebsd-fs@freebsd.org>; Thu,  9 Dec 2010 17:40:18 +0000 (UTC)
	(envelope-from bakul@bitblocks.com)
Received: from mail.bitblocks.com (mail.bitblocks.com [64.142.15.60])
	by mx1.freebsd.org (Postfix) with ESMTP id 0035B8FC20
	for <freebsd-fs@freebsd.org>; Thu,  9 Dec 2010 17:40:17 +0000 (UTC)
Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1])
	by mail.bitblocks.com (Postfix) with ESMTP id DB2CD5B8B;
	Thu,  9 Dec 2010 09:20:58 -0800 (PST)
To: zeus@ibs.dn.ua
In-reply-to: Your message of "Thu, 09 Dec 2010 16:14:12 +0200."
	<20101209141412.GA62013@relay.ibs.dn.ua> 
References: <20101209141412.GA62013@relay.ibs.dn.ua>
Comments: In-reply-to Zeus V Panchenko <zeus@ibs.dn.ua>
	message dated "Thu, 09 Dec 2010 16:14:12 +0200."
Date: Thu, 09 Dec 2010 09:20:58 -0800
From: Bakul Shah <bakul@bitblocks.com>
Message-Id: <20101209172058.DB2CD5B8B@mail.bitblocks.com>
Cc: freebsd-fs@freebsd.org
Subject: Re: what is the best way to use free space on several boxes? 
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 09 Dec 2010 17:40:18 -0000

On Thu, 09 Dec 2010 16:14:12 +0200 Zeus V Panchenko <zeus@ibs.dn.ua>  wrote:
> Hi All,
> 
> we have several FreeBSD 8-STABLE boxes, each has 2x500GB HDD in gmirror 
> UFS dedicated
> 
> but really it's used less than 30% each box ...
> 
> so, advice please, what is the best/correct/right way to use the free
> space on the boxes? nfs? zfs? iscsi?
> 
> the aim is to have storage place ...

This is probably more "researchy" than you want and this is
probably more suited to a much larger number of nodes but how
about some sort of a distributed hash table filesystem? Hash
a block address and map it to N different nodes.  That way
you can add and remove boxes without having to reconfigure.
When a block is initially written, its `address' is computed
from its contents.  One thing you have to worry about is that
the amount of available storage can change dynamically.

Lots of useful information in the wikipedia entry for this.

Even more researchy: even if you move such disks arround to
different nodes, or lose some disk blocks, the remaining data
on such disks can be reattached. A truly resilient storage
system.