From owner-freebsd-stable@FreeBSD.ORG Wed Mar 24 17:14:16 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC121106567A for ; Wed, 24 Mar 2010 17:14:16 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-pz0-f196.google.com (mail-pz0-f196.google.com [209.85.222.196]) by mx1.freebsd.org (Postfix) with ESMTP id 803708FC27 for ; Wed, 24 Mar 2010 17:14:16 +0000 (UTC) Received: by pzk34 with SMTP id 34so1051736pzk.3 for ; Wed, 24 Mar 2010 10:14:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=Nnf/Ao03WytsMAEZqBZLcIEr1DzyDtr5y7j4qIFQ4s0=; b=YPyXM+7qLkxnnwE97jOSJdEuKT9NiBCgj0J8wn3242WrvioNa4R7Fe+Tiv/JztP2cU m13lKM1WibZB9mFPk+uLicx5mYupLKkPut8F+gn1mW0izW5KFlZ9h2vsbXuUnLlHhVPS o4dh96UWIYUPvVMUbMsDaVbb+e9nGBHuaVnc0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=urFnEAsGoCuQOBzM3wlNTk3y8Tf5quxgGSwvdvrviDYcmhRYD1ZioFG2tvczIrXs+L LKVFLd7IKM30QGjJw5Kb2PUtb7t65jqEFKOc8yX3eUikR5v6+rLK6PZiFVq7JWVYz1XQ DpdCeDogWWY1wP7wZEzMk6YWoQJ8dcxh6Z604= MIME-Version: 1.0 Received: by 10.141.15.17 with SMTP id s17mr5576596rvi.14.1269450854253; Wed, 24 Mar 2010 10:14:14 -0700 (PDT) In-Reply-To: References: <4BAA3409.6080406@ionic.co.uk> Date: Wed, 24 Mar 2010 10:14:14 -0700 Message-ID: From: Freddie Cash To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: Multi node storage, ZFS X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Mar 2010 17:14:16 -0000 On Wed, Mar 24, 2010 at 10:04 AM, Michael Loftis wrote: > --On Wednesday, March 24, 2010 9:20 AM -0700 Freddie Cash < > fjwcash@gmail.com> wrote: > On Wed, Mar 24, 2010 at 8:47 AM, Michal wrote: > >> >> I wrote a really long e-mail but realised I could ask this question far >>> far easier, if it doesn't make sense, the original e-mail is bellow >>> >>> Can I use ZFS to create a multinode storage area. Multiple HDD's in >>> Multiple servers to create one target of, for example, //officestorage >>> Allowing me to expand the storage space when needed and clients being >>> able to retrieve data (like RAID0 but over devices not HDD) >>> >>> Here is an example I found which is where I'm getting some ideas from >>> http://www.howtoforge.com/how-to-build-a-low-cost-san-p3 >>> >>> Horribly, horribly, horribly complex. But, then, that's the Linux world. >>> >> :) >> >> Server 1: bunch of disks exported via iSCSI >> Server 2: bunch of disks exported via iSCSI >> Server 3: bunch of disks exported via iSCSI >> >> "SAN" box: uses all those iSCSI exports to create a ZFS pool >> >> Use 1 iSCSI export from each server to create a raidz vdev. Or multiple >> mirror vdevs. When you need more storage, just add another server full of >> disks, export them via iSCSI to the "SAN" box, and expand the ZFS pool. >> >> And, if you need fail-over, on your "SAN" box, you can use HAST at the >> lower layers (currently only available in 9-CURRENT) to mirror the >> storage across two systems, and use CARP to provide a single IP for the >> two boxes. >> > > If you were to do something like this, I'd make sure to have a fast local > ZIL (log) device on the head node. That would reduce latency for writes, > you might also do the same for reads. Then your bulk storage comes from the > iSCSI boxes. > Yes, that would be helpful (mirrored slogs, until we get slog removal support). As would an L2ARC (cache) device in the head node. As well as lots and lots and lots of RAM. And as fast of ethernet NICs as you can get between the head node and the storage nodes. And, and, and, and ... :) -- Freddie Cash fjwcash@gmail.com