From owner-freebsd-fs@freebsd.org Wed Aug 17 15:38:06 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8C69BBC661 for ; Wed, 17 Aug 2016 15:38:06 +0000 (UTC) (envelope-from joe@getsomewhere.net) Received: from prak.gameowls.com (prak.gameowls.com [IPv6:2001:19f0:5c00:950b:5400:ff:fe14:46b7]) by mx1.freebsd.org (Postfix) with ESMTP id 84A3D15EE for ; Wed, 17 Aug 2016 15:38:06 +0000 (UTC) (envelope-from joe@getsomewhere.net) Received: from [IPv6:2001:470:c412:beef:135:c8df:2d0e:4ea6] (unknown [IPv6:2001:470:c412:beef:135:c8df:2d0e:4ea6]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by prak.gameowls.com (Postfix) with ESMTPSA id B6C001863E; Wed, 17 Aug 2016 10:37:58 -0500 (CDT) From: Joe Love Message-Id: Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: MooseFS on FreeBSD (was: HAST + ZFS + NFS + CARP) Date: Wed, 17 Aug 2016 10:37:57 -0500 References: <20160704183643.GI41276@mordor.lan> <20160704193131.GJ41276@mordor.lan> <20160811091016.GI70364@mordor.lan> <1AA52221-9B04-4CF6-97A3-D2C2B330B7F9@sarenet.es> <472bc879-977f-8c4c-c91a-84cc61efcd86@internetx.com> <20160817085413.GE22506@mordor.lan> <465bdec5-45b7-8a1d-d580-329ab6d4881b@internetx.com> <20160817095222.GG22506@mordor.lan> <20160817113339.GH22506@mordor.lan> To: krad , FreeBSD FS In-Reply-To: X-Mailer: Apple Mail (2.3124) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Aug 2016 15:38:06 -0000 On Aug 17, 2016, at 8:33 AM, krad wrote: >=20 > What are peoples experiences on running something like moosfs on top = of > zfs? It looks really compelling on certain levels, but i'm not sure = about > the reality in a production network yet. >=20 I did some experimenting with MooseFS on a test cluster (using ZFS as = the local storage on the nodes). That was nearly a year ago, when I = decided it wasn=E2=80=99t a good fit as a storage backend to vmware = (primarily because of the overhead involved with traversing from vmware = over nfs to moosefs). They did a bunch of tweaking back then as I kept prodding a bit for = better operations on FreeBSD, but they ultimately ran into a stumbling = block with I/O caching in FreeBSD. Here is their analysis of what they ran into, and the solution they came = up with for higher throughput: https://sourceforge.net/p/moosefs/mailman/message/34483159/ = I=E2=80=99d love to get around to testing MooseFS as a backing store for = bhyve, but I currently lack the power & network ports to connect up my = test nodes again. -Joe