From owner-freebsd-fs@FreeBSD.ORG Fri Jul 1 12:33:53 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1D1CA16A41F for ; Fri, 1 Jul 2005 12:33:53 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id C1AE243D53 for ; Fri, 1 Jul 2005 12:33:52 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id j61CXjPu077286; Fri, 1 Jul 2005 07:33:48 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <42C5381E.9080708@centtech.com> Date: Fri, 01 Jul 2005 07:33:34 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.8) Gecko/20050603 X-Accept-Language: en-us, en MIME-Version: 1.0 To: yfyoufeng@263.net References: <42B825CC.806@centtech.com> <20050622023727.GA19408@afields.ca> <42B954FE.2070406@centtech.com> <1119490133.2298.14.camel@localhost.localdomain> In-Reply-To: <1119490133.2298.14.camel@localhost.localdomain> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, Allan Fields Subject: Re: Cluster Filesystem for FreeBSD - any interest? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jul 2005 12:33:53 -0000 yf-263 wrote: > =E5=9C=A8 2005-06-22=E4=B8=89=E7=9A=84 07:09 -0500=EF=BC=8CEric Anderso= n=E5=86=99=E9=81=93=EF=BC=9A >=20 >>Allan Fields wrote: >> >>>On Tue, Jun 21, 2005 at 09:35:56AM -0500, Eric Anderson wrote: >>> >>> >>>>This is something I've brought up before on other lists, but I'm curi= ous=20 >>>>if anyone is interested in developing a BSD licensed clustered=20 >>>>filesystem for FreeBSD (and anyone else)? >>> >>> >>>A few questions: >>> >>>Could this be done as a stackable file system (vnode layer distributed= >>>file system) or did you have something else in mind (i.e. specifically= >>>a full implementation of a network filesystem including storage >>>layer)? >> >>Hmm. I'm not sure if it can or not. I'll try to explain what I'm=20 >>dreaming of. I currently have about 1000 clients needing access to the= =20 >=20 >=20 > This is exactly what we are doing now :) but only a proof-of-concept > toy. >=20 > Now it has a asymmetric arch, i.e. a client, some metadata server, and > some filedata server. >=20 > thanks for your professional description as what we are target to ;) >=20 >=20 >>same pools of data (read/write) all the time. The data changes=20 >>constantly. There is a lot of this data. We use NFS currently.=20 >>FreeBSD is *very* fast and stable at serving NFS data. The problem is,= =20 >>that even though it is very fast and stable, I still cannot pump out=20 >>enough bits fast enough with one machine, and if that one machine fails= =20 >>(hardware problems, etc), then all my machines are hung waiting for me = >>to bring it back online. >> >>So, what I would love to have, is this kind of setup: shared media=20 >>storage (fibre channel SAN, iscsi, or something like ggated possibly), = >>connected up to a cluster of hosts running FreeBSD. Each FreeBSD serve= r=20 >>has access to the logical disks, same partitions, and can mount them al= l=20 >>r/w. Now, I can kind of do this now, however there are obviously some = >>issues with this currently. I want all machines in this cluster to be = >>able to serve the data via NFS (or http, or anything else for that=20 >>matter really - if you can make NFS work, anything will pretty much=20 >>work) simultaneously from the same partitions, and see writes=20 >>immediately as the other hosts in the cluster commit them. >> >>I currently have a solution just like this for Linux - Polyserve=20 >>(http://www.polyserve.com) has a clustered filesystem for linux, that=20 >>works very well. I've even tried to convince them to port it to=20 >>FreeBSD, but it falls on deaf ears, so it's time to make our own. >> >> >> >>>Why not a port of an existing network filesystem say from Linux? >>>(A BSD rewrite could be done, if the code was GPLed.) Would >>>cross-platform capabilities make sense? >> >>That would work fine I'm sure - but I have found some similar threads i= n=20 >>the past that claim it would be just as hard and time consuming to port= =20 >>one as it would be to create one from scratch. Cross platform=20 >>capabilities would be great, but I'm mostly interested in getting=20 >>FreeBSD into this arena (as it will soon be an extremely important one = >>to be in). >> >> >> >>>How do you see this comparing to device-level solutions? I know >>>the argument can be made to implement file systems/storage >>>abstractions at multiple layers, but I thought I might ask. >> >>I'm not sure of a device level solution that does this. I think the OS= =20 >>has to know to commit the meta-data to a journal, or otherwise let the = >>other machines know about locking, etc, in order for this to work. >> >> >> >>>The other thing is there a wealth of filesystem papers out there, >>>any in specific caught your eye? >> >>No - can you point me to some? >> >>I'll be honest here - I'm not a code developer. I would love to learn = >>some C here, and 'just do it', but filesystems aren't exactly simple, s= o=20 >>I'm looking for a group of people that would love to code up something = >>amazing like this - I'll support the developers and hopefully learn=20 >>something in the process. My goal personally would be to do anything I= =20 >>could to make the developers work most productively, and do testing. I= =20 >>can probably provide equipment, and a good testbed for it. You've mentioned this a few times before - is this something you will be = offering publicly sometime soon? Eric --=20 ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------