From owner-freebsd-fs@FreeBSD.ORG Sun Nov 27 04:26:43 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 576D216A420 for ; Sun, 27 Nov 2005 04:26:43 +0000 (GMT) (envelope-from delphij@gmail.com) Received: from zproxy.gmail.com (zproxy.gmail.com [64.233.162.204]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7318743D60 for ; Sun, 27 Nov 2005 04:26:42 +0000 (GMT) (envelope-from delphij@gmail.com) Received: by zproxy.gmail.com with SMTP id z31so759229nzd for ; Sat, 26 Nov 2005 20:26:42 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=N9u819rScZGZp08qe9t9y0qmdNaQWjJIXEfJzRTKfVeFEOt3+ayif0t0LVz5Vy15hQpejTO3SEU5g9N3iOGEHWuZ+KHgnozbhJHB5NgxYZan1MSqogy/+mflC5c0nX3Ez3L/zKnPw7wLH4G4l6xIIzpymZDynjDTNH7W9Gu4ScA= Received: by 10.65.15.3 with SMTP id s3mr177984qbi; Sat, 26 Nov 2005 20:26:42 -0800 (PST) Received: by 10.65.72.5 with HTTP; Sat, 26 Nov 2005 20:26:42 -0800 (PST) Message-ID: Date: Sun, 27 Nov 2005 12:26:42 +0800 From: Xin LI To: bv@wjv.com In-Reply-To: <20051126180211.GA69773@wjv.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 Content-Disposition: inline References: <43889E6F.80008@samsco.org> <20051126180211.GA69773@wjv.com> Cc: freebsd-fs@freebsd.org, user Subject: Re: remind me ... (file undelete on FreeBSD 5.4) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: delphij@delphij.net List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Nov 2005 04:26:43 -0000 SGksIEJpbGwsCgpPbiAxMS8yNy8wNSwgQmlsbCBWZXJtaWxsaW9uIDxidkB3anYuY29tPiB3cm90 ZToKW3NuaXBdCj4gPiBPbmUgc3VnZ2VzdGlvbiBpcyB5b3Ugc3BlY2lmeSAnLUknIHdoZW4gZG9p bmcgcm0sIHdoaWNoIHdpbGwgcHJvbXB0IGlmCj4gPiB5b3UgYXJlIGdvaW5nIHRvIHJlY3Vyc2l2 ZWx5IHJlbW92ZSBkaXJlY3RvcnksIG9yIHJlbW92aW5nIG1vcmUgdGhhbiAzCj4gPiBmaWxlcy4g IFRoaXMgaXMgbW9yZSBvciBsZXNzIGEgZm9vdC1zaG9vdGluZyBwcm9vZi4gIEFub3RoZXIKPiA+ IHN1Z2dlc3Rpb24gaXMgdGhhdCBldmVyeXRoaW5nIGltcG9ydGFudCBzaG91bGQgYmUgYmFja2Vk IHVwIGVsc2V3aGVyZSwKPiA+IHBlcmlvZGljYWxseSBhbmQgZXhwbGljdGx5Lgo+Cj4gSGUgaGFk IGEgbWFqb3IgcHJvYmxlbSB3aGVuIGhlIHNwZWNpZmllZCAnZicgYXMgb24gb3B0aW9uLCBhbmQg J2YnCj4gd2lsbCBvdmVyLXJpZGUgYW55ICdpJyBvcHRpb24uCj4KPiBVc2luZyB0aGUgaSBvcHRp b24gZm9yIGxhcmdlIGFtb3VudHMgb2YgZmlsZXMgd2lsbCBtYWtlIHlvdSBnaXZlCj4gdGhhdCB1 cCBpbiBhIGh1cnJ5LiAgQXMgSSBzdWdnZXN0ZWQgZWFybGllciBwZXJmb3JtIGFuICdscycgd2l0 aAo+IHRoZSBhcHByb3ByaWF0ZSBvcHRpb25zLCBhbmQgdGhlbiBpZiBpdCBhbGwgbG9va3Mgd2Vs bCB1c2UgdGhlCj4gY29tbWFuZCBlZGl0b3IgdG8gY2hhbmdlIHRoZSAnbHMnIHRvICdybScuCj4K CkZvcnR1bmF0ZWx5LCAnSScgaXMgYSBuZXcgb3B0aW9uIGludHJvZHVjZWQgaW4gNi4wIGFuZCA1 LjMuICAnZicgZG9lcwpub3Qgb3ZlcnJpZGUgdGhpcyBvcHRpb24gOi0pICBJdCBpcyBkaWZmZXJl bnQgZnJvbSB0aGUgJ2knIG9wdGlvbiwgYXMKaXQgcHJvbXB0IGZvciBvbmNlIHRvIGNvbmZpcm0g aWYgdGhlIG9wZXJhdGlvbiBzaG91bGQgYmUgY29tbWl0dGVkLiAKRnJvbSB1c2VyIGV4cGVyaWVu Y2UgcGVyc3BlY3RpdmUsIGl0IGlzIGJldHRlciB0aGFuIHRoZSBhbm5veWluZyAnaScKb3B0aW9u IHdoZW4geW91IGFyZSBnb2luZyB0byByZW1vdmUgYSBsb3Qgb2YgZmlsZXMsIGFuZCBlbm91Z2gg dG8KcHJldmVudCBtb3N0IGZvb3Qtc2hvb3RpbmcuCgo+IEFuZCB3aGVuIHlvdSBzdGFydCB1c2lu ZyB3aWxkY2FyZHMsIGFsd2F5cyBtYWtlIHN1cmUgdG8gcnVuCj4gJ3B3ZCcgYmVmb3JlIHlvdSBt YWtlIHRoZSBmaW5hbCBjaG9pY2UuICBTb21ldGltZXMgeW91IHdpbGwgYmUKPiBzdXJwcmlzZWQg YW5kIHlvdSB3aWxsIHNhdmUgeW91cnNlbGYgYSBsb3Qgb2YgZ3JpZWYuCgpTdXJlLCB0aGF0J3Mg d2hhdCB3ZSBjYWxsICJiZXN0IHByYWN0aWNlIi4gIE1heWJlIHdlIHNob3VsZCBzdWdnZXN0CnRo aXMgaW4gb3VyIEhhbmRib29rPwoKQ2hlZXJzLAotLQpYaW4gTEkgPGRlbHBoaWpAZGVscGhpai5u ZXQ+IGh0dHA6Ly93d3cuZGVscGhpai5uZXQK From owner-freebsd-fs@FreeBSD.ORG Mon Nov 28 03:04:37 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 63F6116A41F for ; Mon, 28 Nov 2005 03:04:37 +0000 (GMT) (envelope-from mike503@gmail.com) Received: from wproxy.gmail.com (wproxy.gmail.com [64.233.184.198]) by mx1.FreeBSD.org (Postfix) with ESMTP id E3F2943D46 for ; Mon, 28 Nov 2005 03:04:36 +0000 (GMT) (envelope-from mike503@gmail.com) Received: by wproxy.gmail.com with SMTP id i20so1631461wra for ; Sun, 27 Nov 2005 19:04:36 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=hZ+vIx4s/faVTvI3qa8UB/44uSN/KyPQ5daxY/EV6nlwlsNrjfgGIajIWI6LOLnf+k6zbjc+ZHVNlYOQXw0pL9TJzYuWrbtW6SR/6iaEPu6MTeXtsebClfcZZubqmq+pog2vn0d0VNhs1Rii+dc17R/jbAHTV6toOmuruhBjI6U= Received: by 10.54.86.11 with SMTP id j11mr7922035wrb; Sun, 27 Nov 2005 19:04:36 -0800 (PST) Received: by 10.54.77.2 with HTTP; Sun, 27 Nov 2005 19:04:36 -0800 (PST) Message-ID: Date: Sun, 27 Nov 2005 19:04:36 -0800 From: mike To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Subject: Possible way to distribute NFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Nov 2005 03:04:37 -0000 I've been looking for clustered FS or (if I have to) clustered NFS solutions, and there's not many out there. Unless you want to go with an expensive appliance. Currently you can't "roll your own" at all with FreeBSD. However, I think I've figured out a way that might work; basically using the memcached[1] idea of hashing the requests to identify which server is serving the data could alleviate any issues with NFS not being "globally" aware, and allow for multiple NFS servers to read from the same physical shared storage (note that the expectation is the shared storage is being accessed from multiple physical servers, and redundancy is done below the actual FS layer) If someone could write an NFS client using FUSE (perhaps?) and have it integrate with memcached[1], couldn't that basically allow for n+1 scaling out of NFS servers and storage? Basically any time the FS would need to issue or check a file lock, it would hit the memcache API to check if it's been locked or "represented" yet by a specific NFS server. (Note: whether or not it actually needs a "lock" or it's just saying "I am NFS server #1 and /foo/bar/baz.txt will be served from my server" is something I'd let someone smarter figure out - perhaps just taking ownership of files regardless of the need to lock it would work?) I'm just brain dumping here. Note that I am not that advanced of a programmer (I'm a PHP scripter) so filesystem and hardware I/O semantics are not my specialty; this is just what I've got from Googling until I'm blue in the face and why multi-path storage solutions are still not very cheap/open source (only a couple possibilities, each with limitations and caveats...) Perhaps someone could shed some light, tell me it's stupid, or tell me it's worthwhile? NFS is pretty much tried and true, adding a small middle layer in between to allow for distribution of NFS requests and alleviating the lock conflicts might be a much more usable solution? Perhaps this could fill the void of no GFS for FreeBSD? My idea for design would be N number of NFS servers behind a single virtual IP (or this "middleware" layer would be listening on this virtual IP intercepting all the NFS requests and routing them appropriately to the NFS servers behind them (also, for all intents and purposes, this middleware daemon *could* just be on the same physical machine as each NFS server - saving the need for separate middleware servers too) Thanks, mike [1] "memcached" is a distributed memory caching system - http://www.danga.com/memcached/ - it can be scaled up or out, and can handle failover inside of the protocol (if a memcached server disappears, it's info is forgotten and freed up for another active memcached server) From owner-freebsd-fs@FreeBSD.ORG Tue Nov 29 14:02:41 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9938116A41F for ; Tue, 29 Nov 2005 14:02:41 +0000 (GMT) (envelope-from cel@citi.umich.edu) Received: from citi.umich.edu (citi.umich.edu [141.211.133.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1A98D43D66 for ; Tue, 29 Nov 2005 14:02:41 +0000 (GMT) (envelope-from cel@citi.umich.edu) Received: from [10.58.52.94] (nat-198-95-226-230.netapp.com [198.95.226.230]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by citi.umich.edu (Postfix) with ESMTP id C5DC01BB7B; Tue, 29 Nov 2005 09:02:39 -0500 (EST) Message-ID: <438C5F80.8050800@citi.umich.edu> Date: Tue, 29 Nov 2005 09:02:40 -0500 From: Chuck Lever Organization: Network Appliance, Inc. User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: mike References: In-Reply-To: Content-Type: multipart/mixed; boundary="------------020803020906010708010902" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: Possible way to distribute NFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: cel@citi.umich.edu List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Nov 2005 14:02:41 -0000 This is a multi-part message in MIME format. --------------020803020906010708010902 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit mike- you want pNFS. this is something that is currently being worked out by the IETF, various NFS vendors, and members of the open source community. google for pNFS or Parallel NFS. mike wrote: > I've been looking for clustered FS or (if I have to) clustered NFS > solutions, and there's not many out there. Unless you want to go with > an expensive appliance. Currently you can't "roll your own" at all > with FreeBSD. > > However, I think I've figured out a way that might work; basically > using the memcached[1] idea of hashing the requests to identify which > server is serving the data could alleviate any issues with NFS not > being "globally" aware, and allow for multiple NFS servers to read > from the same physical shared storage (note that the expectation is > the shared storage is being accessed from multiple physical servers, > and redundancy is done below the actual FS layer) > > If someone could write an NFS client using FUSE (perhaps?) and have it > integrate with memcached[1], couldn't that basically allow for n+1 scaling > out of NFS servers and storage? Basically any time the FS would need > to issue or check a file lock, it would hit the memcache API to check > if it's been locked or "represented" yet by a specific NFS server. > > (Note: whether or not it actually needs a "lock" or it's just saying > "I am NFS server #1 and /foo/bar/baz.txt will be served from my > server" is something I'd let someone smarter figure out - perhaps just > taking ownership of files regardless of the need to lock it would > work?) > > I'm just brain dumping here. Note that I am not that advanced of a > programmer (I'm a PHP scripter) so filesystem and hardware I/O > semantics are not my specialty; this is just what I've got from > Googling until I'm blue in the face and why multi-path storage > solutions are still not very cheap/open source (only a couple > possibilities, each with limitations and caveats...) > > Perhaps someone could shed some light, tell me it's stupid, or tell me > it's worthwhile? > > NFS is pretty much tried and true, adding a small middle layer in > between to allow for distribution of NFS requests and alleviating the > lock conflicts might be a much more usable solution? Perhaps this > could fill the void of no GFS for FreeBSD? > > My idea for design would be N number of NFS servers behind a single > virtual IP (or this "middleware" layer would be listening on this > virtual IP intercepting all the NFS requests and routing them > appropriately to the NFS servers behind them (also, for all intents > and purposes, this middleware daemon *could* just be on the same > physical machine as each NFS server - saving the need for separate > middleware servers too) > > Thanks, > mike > > [1] "memcached" is a distributed memory caching system - > http://www.danga.com/memcached/ - it can be scaled up or out, and can > handle failover inside of the protocol (if a memcached server > disappears, it's info is forgotten and freed up for another active > memcached server) --------------020803020906010708010902-- From owner-freebsd-fs@FreeBSD.ORG Tue Nov 29 19:29:56 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2757716A420 for ; Tue, 29 Nov 2005 19:29:56 +0000 (GMT) (envelope-from mike503@gmail.com) Received: from wproxy.gmail.com (wproxy.gmail.com [64.233.184.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 840D743D9E for ; Tue, 29 Nov 2005 19:29:41 +0000 (GMT) (envelope-from mike503@gmail.com) Received: by wproxy.gmail.com with SMTP id i30so2384178wra for ; Tue, 29 Nov 2005 11:29:38 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=I6ZyhR5QSzxxJq98UtimOafopoZKgiL0FzD1jSC45o/ijTQA9DvdjvwPFNLiaUEbJkK3qiX8nR7vzYJScAPE1mCk7FpD0dsnZrtbLSg0dNlHuM0fuVrtkK9sniKNAVPbndfE+LBg21PLefPvWV3hK+8DP9w/ANRixVle/WgUflo= Received: by 10.54.93.10 with SMTP id q10mr10404751wrb; Tue, 29 Nov 2005 11:29:38 -0800 (PST) Received: by 10.54.77.2 with HTTP; Tue, 29 Nov 2005 11:29:38 -0800 (PST) Message-ID: Date: Tue, 29 Nov 2005 11:29:38 -0800 From: mike To: freebsd-fs@freebsd.org In-Reply-To: <438C5F80.8050800@citi.umich.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <438C5F80.8050800@citi.umich.edu> Subject: Re: Possible way to distribute NFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Nov 2005 19:29:56 -0000 I did find that. But isn't that really paired with NFSv4? That's how I got into all the NFSv4 information about it's future plans for clustering/redundancy/whatever was when I Googled for that. Do you have some specific URLs that might help? Is this production-ready/usable, or just something to talk about over coffee? On 11/29/05, Chuck Lever wrote: > mike- > > you want pNFS. this is something that is currently being worked out by > the IETF, various NFS vendors, and members of the open source community. > google for pNFS or Parallel NFS. From owner-freebsd-fs@FreeBSD.ORG Tue Nov 29 21:23:26 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1D5AF16A41F for ; Tue, 29 Nov 2005 21:23:26 +0000 (GMT) (envelope-from cel@citi.umich.edu) Received: from citi.umich.edu (citi.umich.edu [141.211.133.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8F7DC43D46 for ; Tue, 29 Nov 2005 21:23:25 +0000 (GMT) (envelope-from cel@citi.umich.edu) Received: from [141.211.133.33] (dexter.citi.umich.edu [141.211.133.33]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by citi.umich.edu (Postfix) with ESMTP id CF5861BB70; Tue, 29 Nov 2005 16:23:24 -0500 (EST) Message-ID: <438CC6CC.8090303@citi.umich.edu> Date: Tue, 29 Nov 2005 16:23:24 -0500 From: Chuck Lever Organization: Network Appliance, Inc. User-Agent: Mozilla Thunderbird 1.0.7-1.4.1 (X11/20050929) X-Accept-Language: en-us, en MIME-Version: 1.0 To: mike References: <438C5F80.8050800@citi.umich.edu> In-Reply-To: Content-Type: multipart/mixed; boundary="------------080908020308010906010001" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: Possible way to distribute NFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: cel@citi.umich.edu List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Nov 2005 21:23:26 -0000 This is a multi-part message in MIME format. --------------080908020308010906010001 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit mike wrote: > I did find that. > > But isn't that really paired with NFSv4? That's how I got into all the > NFSv4 information about it's future plans for > clustering/redundancy/whatever was when I Googled for that. > > Do you have some specific URLs that might help? Is this > production-ready/usable, or just something to talk about over coffee? pNFS is planned to be part of a minor revision of NFSv4. that means something like NFSv4.1 or NFSv4.2. there is nothing production ready right now, but there is significant industry momentum behind this, and several prototypes already exist. most of the design is happening within the IETF (mailing lists and f2f meetings), and i'm not especially close to the dialog. > On 11/29/05, Chuck Lever wrote: > >>mike- >> >>you want pNFS. this is something that is currently being worked out by >>the IETF, various NFS vendors, and members of the open source community. >> google for pNFS or Parallel NFS. > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --------------080908020308010906010001--