From owner-svn-doc-all@freebsd.org Sat Jan 14 22:42:07 2017 Return-Path: Delivered-To: svn-doc-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9BABCB0795; Sat, 14 Jan 2017 22:42:07 +0000 (UTC) (envelope-from bjk@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9B0DB119D; Sat, 14 Jan 2017 22:42:07 +0000 (UTC) (envelope-from bjk@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v0EMg67L003801; Sat, 14 Jan 2017 22:42:06 GMT (envelope-from bjk@FreeBSD.org) Received: (from bjk@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v0EMg6hM003800; Sat, 14 Jan 2017 22:42:06 GMT (envelope-from bjk@FreeBSD.org) Message-Id: <201701142242.v0EMg6hM003800@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bjk set sender to bjk@FreeBSD.org using -f From: Benjamin Kaduk Date: Sat, 14 Jan 2017 22:42:06 +0000 (UTC) To: doc-committers@freebsd.org, svn-doc-all@freebsd.org, svn-doc-head@freebsd.org Subject: svn commit: r49850 - head/en_US.ISO8859-1/htdocs/news/status X-SVN-Group: doc-head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-doc-all@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "SVN commit messages for the entire doc trees \(except for " user" , " projects" , and " translations" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jan 2017 22:42:07 -0000 Author: bjk Date: Sat Jan 14 22:42:06 2017 New Revision: 49850 URL: https://svnweb.freebsd.org/changeset/doc/49850 Log: Add Ceph entry from Willem Jan Withagen Modified: head/en_US.ISO8859-1/htdocs/news/status/report-2016-10-2016-12.xml Modified: head/en_US.ISO8859-1/htdocs/news/status/report-2016-10-2016-12.xml ============================================================================== --- head/en_US.ISO8859-1/htdocs/news/status/report-2016-10-2016-12.xml Sat Jan 14 19:19:39 2017 (r49849) +++ head/en_US.ISO8859-1/htdocs/news/status/report-2016-10-2016-12.xml Sat Jan 14 22:42:06 2017 (r49850) @@ -888,4 +888,176 @@ couple of weeks.

+ + + Ceph on &os; + + + + + Willem Jan + Withagen + + wjw@digiware.nl + + + + + Ceph Main Site + Main Repository + My &os; Fork + + + +

Ceph is a distributed object store and file system designed + to provide excellent performance, reliability and + scalability:

+ +
    +

  • Object Storage

    + +

    Ceph provides seamless access to objects using native + language bindings or radosgw, a REST interface that’s + compatible with applications written for S3 and + Swift.

    • + +

  • Block Storage

    + +

    Ceph’s RADOS Block Device (RBD) provides access to block + device images that are striped and replicated across the + entire storage cluster.

    • + +

  • File System

    + +

    Ceph provides a POSIX-compliant network file system that + aims for high performance, large data storage, and maximum + compatibility with legacy applications.

    • +
+ +

I started looking into Ceph because the HAST solution with + CARP and ggate did not really do what I was looking + for. But I aim to run a Ceph storage cluster of storage nodes + that are running ZFS. User stations would be running + bhyve on RBD disks that are stored in Ceph.

+ +

The &os; build will build most of the tools in Ceph. Note + that the RBD dependant items will not work, since &os; does not + have RBD (yet).

+ +

Most notable progress since the last report:

+ +
    +
  • RBD is actually buildable and can be used to manage + RADOS BLOCK DEVICEs.
    • + +
  • All tests run to completion for the current selection of + tools, though the neded (minor) patches have yet to be + pulled into HEAD.
    • + +
  • Cmake is now the only way of building Ceph.
    • + +
  • The threading/polling code has been reworked for the + simple socket code. It now uses a self-pipe, instead of using + an odd shutdown()-signaling Linux feature.
    • + +
  • The EventKqueue code was modified to work around the + "feature" that starting threads destroys the kqueue + handles. The code was just finshed, so it is not yet + submitted to the main repository.
    • + +
  • We investigated differences between &os; and Linux for + SO_REUSEADDR and SO_REUSEPORT. Fortunately, + the code is only used during testing, so disabling these + features only delays progress in the tests.
    • + +
  • A jenkins instances is regularly testing both + ceph/ceph/master and + wjwithagen/ceph/wip.FreeBSD, so there is regular + verification of buildability and the tests: http://cephdev.digiware.nl:8180/jenkins/.
    • +
+ + +

Build Prerequisites

+ +

Compiling and building Ceph is tested on 12-CURRENT with + its clang 3.9.0, but 11-RELEASE will probably also work, given + experience with clang 3.7.0 from 11-CURRENT. Interestingly, + when 12-CURRENT had clang 3.8.0, that did not work as well as + either 3.7.0 or 3.9.0..

+ +

The clang 3.4 present in 10-STABLE does not have the + required capabilities to compile everything.

+ +

The following setup will get things running for &os;:

+ +
    +
  1. Install bash and link it in /bin
    2. + +
  2. It is no longer necessary to add a definition of + ENODATA to /usr/include/errno.h
    3. + +
  3. Clone the github repo + (http://github.com/wjwithagen/ceph.git) and checkout the + "wip.FreeBSD" branch
    4. + +
  4. Run ./do_FreeBSD.sh to start the build.
    5. +
+ +

The old build method using automake is no longer used; see + the README.FreeBSD for more details.

+ +

Parts not (yet) included:

+ +
    +
  • KRBD: Kernel Rados Block Devices is implemented in the + Linux kernel, but not in the &os; kernel. Perhaps + ggated could be used as a template since it does some + of the same things as KRBD, just between 2 disks. It also has + a userspace counterpart, which could ease development.
    • + +
  • BlueStore: &os; and Linux have different AIO APIs, and + that incompatibility needs to be resolved somehow. + Additionally, there is discussion in &os; about + aio_cancel not working for all devicetypes.
    • + +
  • CephFS: Cython tries to access an internal field in + struct dirent, which does not compile.
    • + +
  • Tests that verify the correct working of the above are + also excluded from the testset.
    • +
+ + + + Run integration tests to see if the &os; daemons will work + with a Linux Ceph platform. + + Compile and test the user space RBD (Rados Block Device). + This currently works, but testing has been limitted. + + Investigate and see if an in-kernel RBD device could be + developed akin to &os;'s ggate. + + Investigate the keystore, which could be embedded in the + kernel on Linux, and currently prevents building Cephfs and some + other components. The first question whether it is really + required, or if only KRBD require it. + + Scheduler information is not used at the moment, because + the schedulers work rather differently between &os; and Linux. + But a a certain point in time, this would need some attention in + src/common/Thread.cc. + + Integrate the &os; /etc/rc.d initscripts in the + Ceph stack. This helps with testing, but also enables running + Ceph on production machines. + + Build a testcluster and start running some of the + teuthology integration tests on it. + + Design a vitual disk implementation that can be used with + bhyve and attached to an RBD image. + +