Ceph on FreeBSD

From owner-svn-doc-all@freebsd.org Fri Jul 8 02:42:46 2016 Return-Path: Delivered-To: svn-doc-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B0F5CB831C9; Fri, 8 Jul 2016 02:42:46 +0000 (UTC) (envelope-from dru@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8ABA817DB; Fri, 8 Jul 2016 02:42:46 +0000 (UTC) (envelope-from dru@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id u682gjD4055624; Fri, 8 Jul 2016 02:42:45 GMT (envelope-from dru@FreeBSD.org) Received: (from dru@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id u682gjNC055623; Fri, 8 Jul 2016 02:42:45 GMT (envelope-from dru@FreeBSD.org) Message-Id: <201607080242.u682gjNC055623@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dru set sender to dru@FreeBSD.org using -f From: Dru Lavigne Date: Fri, 8 Jul 2016 02:42:45 +0000 (UTC) To: doc-committers@freebsd.org, svn-doc-all@freebsd.org, svn-doc-head@freebsd.org Subject: svn commit: r49066 - head/en_US.ISO8859-1/htdocs/news/status X-SVN-Group: doc-head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-doc-all@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "SVN commit messages for the entire doc trees \(except for " user" , " projects" , and " translations" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 02:42:46 -0000 Author: dru Date: Fri Jul 8 02:42:45 2016 New Revision: 49066 URL: https://svnweb.freebsd.org/changeset/doc/49066 Log: Add Ceph status report submitted by wjw@digiware.nl. Reviewed by: wblock Sponsored by: iXsystems Modified: head/en_US.ISO8859-1/htdocs/news/status/report-2016-04-2016-06.xml Modified: head/en_US.ISO8859-1/htdocs/news/status/report-2016-04-2016-06.xml ============================================================================== --- head/en_US.ISO8859-1/htdocs/news/status/report-2016-04-2016-06.xml Fri Jul 8 00:43:59 2016 (r49065) +++ head/en_US.ISO8859-1/htdocs/news/status/report-2016-04-2016-06.xml Fri Jul 8 02:42:45 2016 (r49066) @@ -680,4 +680,172 @@ Microsoft + + + Ceph on FreeBSD + + + + + Willem Jan + Withagen + + wjw@digiware.nl + + + + + Ceph main site + Main repository + My Fork + The git PULL with all changes + + + +

Ceph is a distributed object store and file system designed + to provide excellent performance, reliability, and + scalability. It provides the following features:

+ +

Object Storage: Ceph provides seamless access to objects + using native language bindings or radosgw, a REST + interface that is compatible with applications written for + S3 and Swift.
Block Storage: Ceph’s RADOS Block Device (RBD) provides + access to block device images that are striped and + replicated across the entire storage cluster.
File System: Ceph provides a POSIX-compliant network file + system that aims for high performance, large data storage, + and maximum compatibility with legacy applications.

+ +

I started looking into Ceph as using HAST with CARP and + ggate did not meet my requirements. My primary goal + with Ceph is to run a storage cluster of ZFS storage nodes + where the clients run bhyve on RBD disks stored in Ceph.

+ +

The &os; build process can build most of the tools in + Ceph. However, the RBD-dependent items do not work since + &os; does not yet provide RBD support.

+ +

Since the last quarterly report, the following progress was + made:

+ +

The changeover from using CMake to Automake results in a + much cleaner development environment and better test output. + The changes can be found in the + wip-wjw-freebsd-cmake branch.
Throttling code has been overhauled to prevent live locks. + These mainly occur on &os; but also manifest on Linux.
Fixed a few more tests. On one occasion, I was able to + complete the full test set without errors.

+ +

11-CURRENT is used to compile and build test Ceph. The + Clang toolset needs to be at least version 3.7 as Clang 3.4 + does not have all of the capabilities required to compile + everything.

+ +

This setup will get things running for &os;:

+ +

+
Install bash and link it in /bin + (requires root privileges):
+ +
sudo pkg install bash
+ +
sudo ln -s /usr/local/bin/bash /bin/bash
+
+
Build Ceph:
+ +
git clone https://github.com/wjwithagen/ceph.git
+ +
cd ceph
+ +
git checkout wip-wjw-freebsd-tests
+ +
./do_freebsd.sh
+

+ +

Parts Not Yet Included:

+ +

RBD: Rados Block Devices are currently implemented in the + Linux kernel but there used to be a userspace + implementation. It is possible that ggated could + be used as a template since it provides some of the same + functionality and it has a userspace counterpart.
BlueStore: &os; and Linux have a different AIO API which + needs to be made compatible. There has been some + discussion about aio_cancel not working for all + device types in &os;.
CephFS: Cython tries to access an internal field in dirent + which does not compile.
Tests that verify the correct working of the above are + also excluded from the testset.

+ +

Tests Not Yet Included:

+ +

ceph-detect-init/run-tox.sh: the current + implementation does not know anything about &os; + rc-init.
Tests that make use of nosestests do not really + work since nostests is not in /usr/bin, + and calling /usr/bin/env/nosetests does not work on + &os;.
test/pybind/test_ceph_argparse.py
test/pybind/test_ceph_daemon.py

+ + + + The current and foremost task it to get the test set to + complete without errors. + + Build an automated test platform that will build + ceph/master on &os; and report the results back to + the Ceph developers. This will increase the maintainability + of the &os; side of things as developers are signaled that + they are using Linux-isms that will not compile or run on + &os;. Ceph has several projects that support this: Jenkins, + teuthology, and palpito. But even a + while { compile } loop that reports the build data on + a static webpage is a good start. + + Run integration tests to see if the &os; daemons will work + with a Linux Ceph platform. + + Get the currently excluded Python tests to work. + + Compile and test the user space RBD (Rados Block + Device). + + Investigate if an in-kernel RBD device could be developed + ala ggate. + + Investigate the keystore which currently prevents the + building of Cephfs and some other parts. + + Integrate the &os; /etc/rc.d init scripts in the + Ceph stack for testing and for running Ceph on production + machines. + +