From owner-freebsd-fs@freebsd.org Sun Sep 6 22:18:41 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5894D9CC07E for ; Sun, 6 Sep 2015 22:18:41 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id E9C0012E1 for ; Sun, 6 Sep 2015 22:18:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:O0U6JxVv4D2UCaggKfsecM4/fOHV8LGtZVwlr6E/grcLSJyIuqrYZhyDt8tkgFKBZ4jH8fUM07OQ6PC8HzRRqs/e7DgrS99laVwssY0uhQsuAcqIWwXQDcXBSGgEJvlET0Jv5HqhMEJYS47UblzWpWCuv3ZJQk2sfTR8Kum9IIPOlcP/j7n0oM2PJV0Zz2PiPftbF1afk0b4joEum4xsK6I8mFPig0BjXKBo/15uPk+ZhB3m5829r9ZJ+iVUvO89pYYbCf2pN4xxd7FTDSwnPmYp/4Wr8ECbFUrcrkcbB0cRiBZBBUDl8RvwV439+n/4sfBx0S+aIMf8RKo4cTWp66B2RFnjjyJRZBAj92SCsM17j+p+qRmioxF6i9rOZYieN/5ze4vAetwHSG5ZXoBaXnoSUcuHc4ITAr9Zbq5jpI7nqg5L9EPmCA== X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BYAgAyu+xV/61jaINeg3dpBoMeuk0BCYF3hXkCgVcUAQEBAQEBAQGBCYIdggcBAQQjVhACAQgSBgICDRkCAkkOAgQTiC4NtC6TeAEBAQEGAQEBAQEZBIEihVGDdoEFhEEXATMHgmmBQwWVFz6FCok8hDORDINqAiaEHCIzh0SBBQEBAQ X-IronPort-AV: E=Sophos;i="5.17,481,1437451200"; d="scan'208";a="237002272" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 06 Sep 2015 18:18:33 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 71E2915F563; Sun, 6 Sep 2015 18:18:33 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id zNzkbqJlzNOa; Sun, 6 Sep 2015 18:18:32 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 9402615F565; Sun, 6 Sep 2015 18:18:32 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id y4UTipf5FowJ; Sun, 6 Sep 2015 18:18:32 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 7652C15F563; Sun, 6 Sep 2015 18:18:32 -0400 (EDT) Date: Sun, 6 Sep 2015 18:18:32 -0400 (EDT) From: Rick Macklem To: Jordan Hubbard Cc: freebsd-fs@freebsd.org, Rakshith Venkatesh Message-ID: <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> Subject: Re: CEPH + FreeBSD MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: CEPH + FreeBSD Thread-Index: KowScjIbpQ2TT+IvpGwjo+f1Q1ORrg== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2015 22:18:41 -0000 Jordan Hubbard wrote: >=20 > > On Sep 3, 2015, at 4:17 AM, Rick Macklem wrote: > >=20 > > Slightly off topic but, btw, there is a port of GLusterFS and those fol= ks > > do seem > > interested in seeing it brought "up to speed". I am not sure how mature= it > > is at > > this point, but it has been known to build on amd64. (I don't have an a= md64 > > machine, > > so I haven't gotten around to building/testing it, but I do plan to try= and > > use > > it as a basis for a pNFS server, if I can figure out how to get the FH = info > > out of it. > > I'm working on that;-) >=20 > There are at least two distributed (multi-node) object stores for FreeBSD > that I know of. >=20 > One is glusterfs, for which I=E2=80=99m not even really clear on the stat= us of the > ports for. I don=E2=80=99t see any glusterfs port in the master branch o= f > https://github.com/freebsd/freebsd-ports (or > https://github.com/freebsd/freebsd-ports/tree/branches/2015Q3 for that > matter). >=20 > Our FreeNAS ports tree (https://github.com/freenas/ports), in which we ha= ve a > bit more latitude to add and curate our own ports, has both a net/gluster= fs > and sysutils/glusterfs, from separate sources (looks like we need to clea= n > things up) - net/glusterfs lists craig001@lerwick.hopto.org as the > MAINTAINER and is at version 3.6.2. The sysutils/glusterfs port lists > bapt@FreeBSD.org as the MAINTAINER and is at version 20140811. >=20 > I=E2=80=99m not really sure about the provenance since we were simply eva= luating > glusterfs for awhile and may have pulled in interim versions from those > sources, but obviously it would be best to have an official maintainer an= d > someone in the FreeBSD project actually curating a glusterfs port so that > all users of FreeBSD can use it. It would also be fairly key to your own > efforts, assuming you decide to pursue glusterfs as a foundation technolo= gy > for pNFS. >=20 > The other object store, which is pretty mature and is currently leading t= he > pack (of two :) ) for inclusion into FreeNAS is RiakCS from Basho. There= is > a port currently in databases/riak but it=E2=80=99s pretty out of date at= version > 1.4.12 (the current version is 2.0.1, with 2.0 being a major upgrade of > RiakCS). >=20 > We are very interested in investigating various ways of shimming RiakCS t= o > NFS, using RiakCS a back-end store. Is that something you=E2=80=99d be = amenable to > discussing? I=E2=80=99d be happy to send you an amd64 architecture mach= ine to > develop on. :) >=20 Hmm. From a quick look at their web page (I looked once before as well), I = don't think RiakCS has what I need to do pNFS in a reasonable (for me) amount of = effort. Two things that glusterFS has that I am hoping to use (and I don't think Ri= akCS has either of these) are: - A Fuse file system interface which allows the kernel nfsd to access the s= tore as a file system, so that it can provide the metadata services (NFS without = the reads/writes). - A userland NFSv3 server in each node which will allow the node to act as = a data server. If I am wrong and RiakCS does support a VFS file system interface (via Fuse= or ???), then please correct me. With that, it might be a reasonable alternative. I'll admit I've spent a little time looking at the glusterFS sources and ha= ven't yet solved the problem of how to generate the file handles I need, but that sou= nds trivial compared with an entire Fuse and/or VFS file system interface, I think? In general, using a cloud object store to implement a pNFS server is a *mis= *use of the technology, imho. I think it may be possible with glusterFS, since that= technology seems to be based on a cluster file system, which is what a pNFS server can= also use. I think there would be a lot of work involved in mapping a POSIX file syste= m onto the Riak database and then exporting that via NFS, etc. It might also be more p= ractical to do this via a userland NFS service than the kernel based one currently in F= reeBSD. (glusterFS is starting to use the NFS-ganesha server, but I believe it is p= retty Linux specific, so I doubt it would be useful for Riak running on FreeBSD?) rick > - Jordan >=20 >=20