From owner-freebsd-fs@freebsd.org Mon Sep 7 06:13:16 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1D1FF9CCA0F for ; Mon, 7 Sep 2015 06:13:16 +0000 (UTC) (envelope-from jordanhubbard@icloud.com) Received: from pv33p03im-asmtp001.me.com (pv33p03im-asmtp001.me.com [17.143.180.10]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E8B20640 for ; Mon, 7 Sep 2015 06:13:15 +0000 (UTC) (envelope-from jordanhubbard@icloud.com) Received: from [10.20.30.11] (75-101-82-48.static.sonic.net [75.101.82.48]) by pv33p03im-asmtp001.me.com (Oracle Communications Messaging Server 7.0.5.35.0 64bit (built Mar 31 2015)) with ESMTPSA id <0NUA00PSOMLT2810@pv33p03im-asmtp001.me.com> for freebsd-fs@freebsd.org; Mon, 07 Sep 2015 06:13:09 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2015-09-07_01:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 kscore.is_bulkscore=0 compositescore=0.994460221061599 phishscore=0 kscore.is_spamscore=0 rbsscore=0.994460221061599 recipient_to_sender_totalscore=0 spamscore=0 urlsuspectscore=0.994460221061599 adultscore=0 kscore.compositescore=0 circleOfTrustscore=0 suspectscore=0 recipient_domain_to_sender_totalscore=0 bulkscore=0 recipient_domain_to_sender_domain_totalscore=0 recipient_to_sender_domain_totalscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1412110000 definitions=main-1509070115 Content-type: text/plain; charset=utf-8 MIME-version: 1.0 (Mac OS X Mail 9.0 \(3093\)) Subject: glusterfs + FreeBSD [was Re: CEPH + FreeBSD] From: Jordan Hubbard In-reply-to: <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> Date: Sun, 06 Sep 2015 23:13:05 -0700 Cc: freebsd-fs@freebsd.org, Rakshith Venkatesh , Justin Clift Content-transfer-encoding: quoted-printable Message-id: <382BA1FD-032E-4F21-9460-D2119340E21B@icloud.com> References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> To: Rick Macklem X-Mailer: Apple Mail (2.3093) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Sep 2015 06:13:16 -0000 [ Adding Justin Clift to CC line ] > On Sep 6, 2015, at 3:18 PM, Rick Macklem wrote: >=20 > Hmm. =46rom a quick look at their web page (I looked once before as = well), I don't > think RiakCS has what I need to do pNFS in a reasonable (for me) = amount of effort. > Two things that glusterFS has that I am hoping to use (and I don't = think RiakCS has > either of these) are: > - A Fuse file system interface which allows the kernel nfsd to access = the store as > a file system, so that it can provide the metadata services (NFS = without the reads/writes). > - A userland NFSv3 server in each node which will allow the node to = act as a data server. >=20 > If I am wrong and RiakCS does support a VFS file system interface (via = Fuse or ???), then > please correct me. You are not wrong. RiakCS is basically just a distributed object store = with an Amazon S3 compatible API. It doesn=E2=80=99t attempt to expose = the object store as a filesystem or provide any other modes of access to = it other than via S3 or its own =E2=80=9Cdatabase API=E2=80=9D since = it=E2=80=99s also (at its heart) a distributed database. In all fairness to Basho, I also think that was probably the right call. = It=E2=80=99s not a file system and doesn=E2=80=99t pretend to be one, = it=E2=80=99s just a distributed database that can store arbitrary = numbers (and sizes) of objects in a cluster with fault-tolerance, = multi-data center tenancy and so on (I also have no connection with = Basho so if it sounds like I=E2=80=99m trying to sales pitch it or = something, I=E2=80=99m not, I=E2=80=99m simply trying to be accurate in = describing it given that we=E2=80=99re also evaluating it as a possible = technology to incorporate into FreeNAS, for which the topic of = S3-compatible object storage comes up fairly frequently. Now glusterfs is another kettle of fish, and like I said, I think the = bigger question there is just what FreeBSD=E2=80=99s relationship with = it is. The https://wiki.freebsd.org/GlusterFS wiki was created over a = year ago and seems to be suffering from an overall lack of specific = goals. Does the FreeBSD project want glusterfs and/or is there anyone = in the FreeBSD community even interested in deploying it in a production = scenario? Is anyone willing to actually maintain a port, just as the = minimum price of admission? The bug tracking this, namely = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194409, has been = open for almost a year now and while it looks like =E2=80=98craig001=E2=80= =9D has been trying fairly hard to get 3.7.2 up and working, he=E2=80=99s = hitting some walls and it=E2=80=99s not even clear he=E2=80=99s in = contact with the right folks. As the previous glusterfs mail thread indicated, I was also happy to add = an earlier version of his port to the FreeNAS repo and play with it = there, though I was forced to take it back out of FreeNAS when we ran = into a number of issues just trying to make it work with the Linux = glusterfs reference VMs we set up. Should the underlying port get some = love again, it would be the simplest thing in the world for me to = refresh our copy of it and uncomment the line which adds glusterfs to = the FreeNAS manifest (and being a storage appliance, it=E2=80=99s rather = of an obvious test bed for this since people running FreeNAS are also = the target audience for technologies like this). If getting glusterfs up as a stable, working substrate is also a = prerequisite for getting pNFS, then I think a lot of us in the storage = industry would be the first to say =E2=80=9CWhat do we have to do? How = can we help?=E2=80=9D I already volunteered to send Rick an AMD64 = architecture machine for development, but I suspect that=E2=80=99s = actually the least of our obstacles right now. :-) - Jordan