From owner-freebsd-current@freebsd.org Wed Dec 19 16:31:39 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4713D133BBD0 for ; Wed, 19 Dec 2018 16:31:39 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-ot1-x32d.google.com (mail-ot1-x32d.google.com [IPv6:2607:f8b0:4864:20::32d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 092F188807 for ; Wed, 19 Dec 2018 16:31:38 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-ot1-x32d.google.com with SMTP id v23so19635550otk.9 for ; Wed, 19 Dec 2018 08:31:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=cPuAQaogYXEWkURGYic2nUqfsUFib1b9wz0LZwZEeVI=; b=kcFWAXxZ7wl7ViYFZK1s30H/x9w0xJTWqfJANuPO1S8xmV0O8c9EpU9bhxS9pKxUiA h2Wb91QicEz/QropBzNuuF3OZV5Q6s1p+A+iiISIWcxks4a+ouBx8MYT0ZmJzZje4gOT biKsZMwJsJMTCXJdJzPL/HxAdvEa1IRGkP8FLdNBNT1uTm15CEeuaMX1I/+fXZ/RkgTw psXe/1Apj87XqE4IujYGOZI7L1rAiEcdZ2dqikp8DdnIWY7bLmbF7hUs+iawJi8wtsIV DhEgoVKL1DVdusgcUB+GWX6Tpjrrck98N25Xw2HCuJR6qV2ckJho4ngNWCVJY/2FDRuZ ylCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=cPuAQaogYXEWkURGYic2nUqfsUFib1b9wz0LZwZEeVI=; b=T3o/2uT2m7Y0ywkbJKCDlsVSSRED2SnYO/Dj4tLjzQM/V2p8GMz3qMCLy/AEqSILmC XKqwdPykyQ/jN3d13swHXWoMvYZ9Sncf0zu6FPBBlbjeYpKXnA/tQaz4oOI1/YzlAbia I0idqZgfMY2iTiur8mEOTw2rsxmTBTPNOexScypENoVymrKHw3Rc8LQovQXiWNhqL4Lv aipIyOLXkqTZwrR4ydYz9bP4JuhG3eQDqXdTtKeALQddY6t8a4otcLPXneQsvQSsvxuw UNlvyG/gIonrU7QNtBqy/F5q5kk5jkhf/mxbtgd6JwiNGNTp3wMqhpBSogZfUe4fi7p+ bB4Q== X-Gm-Message-State: AA+aEWaVVcPPXiPwwUEmoQrcmI7lSZ7j8uguWsWmNscAM85Y2i6dytre wOq2BkmJLk5L+RiMzPEQ/XxNBSn6vrmPpA== X-Google-Smtp-Source: AFSGD/VsWFsJZrpM4UL1eBiWLxBEyZeAV76xSq+Dwa1Sv4Q7C7wQI0qq0T0iuLQmHid2v1B1w9+Ing== X-Received: by 2002:a9d:3b77:: with SMTP id z110mr5049525otb.352.1545237096850; Wed, 19 Dec 2018 08:31:36 -0800 (PST) Received: from mutt-hbsd ([185.100.87.129]) by smtp.gmail.com with ESMTPSA id k13sm16950451otj.19.2018.12.19.08.31.33 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 19 Dec 2018 08:31:36 -0800 (PST) Date: Wed, 19 Dec 2018 11:30:33 -0500 From: Shawn Webb To: Matthew Macy Cc: freebsd-fs , freebsd-current Subject: Re: The future of ZFS in FreeBSD Message-ID: <20181219163033.jwm7opiwmdhbk6p3@mutt-hbsd> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="rlmvscpkcfr6pslq" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hbsd 13.0-CURRENT FreeBSD 13.0-CURRENT HARDENEDBSD-13-CURRENT amd64 X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: NeoMutt/20180716 X-Rspamd-Queue-Id: 092F188807 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hardenedbsd.org header.s=google header.b=kcFWAXxZ; spf=pass (mx1.freebsd.org: domain of shawn.webb@hardenedbsd.org designates 2607:f8b0:4864:20::32d as permitted sender) smtp.mailfrom=shawn.webb@hardenedbsd.org X-Spamd-Result: default: False [-2.17 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(0.00)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt1.aspmx.l.google.com]; DKIM_TRACE(0.00)[hardenedbsd.org:+]; NEURAL_HAM_SHORT(-0.91)[-0.915,0]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; IP_SCORE(-0.64)[ipnet: 2607:f8b0::/32(-1.75), asn: 15169(-1.39), country: US(-0.08)]; ARC_NA(0.00)[]; RECEIVED_SPAMHAUS_XBL(3.00)[129.87.100.185.zen.spamhaus.org : 127.0.0.4]; R_DKIM_ALLOW(0.00)[hardenedbsd.org:s=google]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[hardenedbsd.org]; BAD_REP_POLICIES(0.10)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[d.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MID_RHS_NOT_FQDN(0.50)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 16:31:39 -0000 --rlmvscpkcfr6pslq Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Dec 18, 2018 at 10:49:48PM -0800, Matthew Macy wrote: > The sources for FreeBSD's ZFS support are currently taken directly > from Illumos with local ifdefs to support the peculiarities of FreeBSD > where the Solaris Portability Layer (SPL) shims fall short. FreeBSD > has regularly pulled changes from Illumos and tried to push back any > bug fixes and new features done in the context of FreeBSD. In the past > few years the vast majority of new development in ZFS has taken place > in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced > that they will be moving to ZoL > https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means > that there will be little to no net new development of Illumos. While > working through the git history of ZoL I have also discovered that > many races and locking bugs have been fixed in ZoL and never made it > back to Illumos and thus FreeBSD. This state of affairs has led to a > general agreement among the stakeholders that I have spoken to that it > makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf > has graciously encouraged me to add FreeBSD support directly to ZoL > https://github.com/zfsonfreebsd/ZoF so that we might all have a single > shared code base. >=20 > A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port > Before it can be committed some additional functionality needs to be > added to the FreeBSD opencrypto framework. These can be found at > https://reviews.freebsd.org/D18520 >=20 > This port will provide FreeBSD users with multi modifier protection, > project quotas, encrypted datasets, allocation classes, vectorized > raidz, vectorized checksums, and various command line improvements. >=20 > Before ZoF can be merged back in to ZoL several steps need to be taken: > - Integrate FreeBSD support into ZoL CI > - Have most of the ZFS test suite passing > - Complete additional QA testing at iX >=20 > We at iX Systems need to port ZoL's EC2 CI scripts to work with > FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes. > Being integrated in to their CI will mean that, among other things, > most integration issues will be caught before a PR is merged upstream > rather than many months later when it is MFVed into FreeBSD. I???m > hoping to submit the PR to ZoL some time in January. >=20 > This port will make it easy for end users on a range of releases to > run the latest version of ZFS. Nonetheless, transitioning away from an > Illumos based ZFS is not likely to be entirely seamless. The > stakeholders I???ve spoken to all agree that this is the best path > forward but some degree of effort needs to be made to accommodate > downstream consumers. The current plan is to import ZoF and unhook the > older Illumos based sources from the build on April 15th or two months > after iX systems QA deems ZoF stable - which ever comes later. The > Illumos based sources will be removed some time later - but well > before 13. This will give users a 3 month period during which both the > port and legacy Illumos based ZFS will be available to users. Pools > should interoperate between ZoF and legacy provided the user does not > enable any features available only in ZoF. We will try to accommodate > any downstream consumers in the event that they need that date pushed > back. We ask that any downstream consumers who are particularly > sensitive to changes start testing the port when it is formally > announced and report back any issues they have. I will do my best to > ensure that this message is communicated to all those who it may > concern. However, I can???t ensure that everyone reads these lists. That > is the responsibility of -CURRENT users. Hey Matt, Thank you for your detailed and informative post. It really helps downstream consumers of FreeBSD. I'm curious what this means for OpenZFS. I was under the impression that OpenZFS was the upstream for all the ZFS implementations (sans Oracle). Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD Tor-ified Signal: +1 443-546-8752 Tor+XMPP+OTR: lattera@is.a.hacker.sx GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --rlmvscpkcfr6pslq Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEKrq2ve9q9Ia+iT2eaoRlj1JFbu4FAlwaciUACgkQaoRlj1JF bu7GnxAAkKRQBJ1ypOD0J+FxS/K5fqXH+mBD2DW66/iU0EPcAsTZFPj5nhsCuhLL znBjvCH4nnMu1vYoTwn0Yeism1ioYz6EGxfdxVfyz/X+cOv1Bf9tGiKwvm5U8TvM AK2YIaRHL8lWLeQzMapJx5xm1PVPIo6FPyVOVZSCP0JetfJ96ahTPOOWcOozk8h6 vYurzIXnsA7IBxEy22/h3h1ZyJ4IxRFW81XAGsdQPdhWA2V2eBGka9UGisk5cgmZ vxRnZH/8RBjDaGNnzrpZBlND7Ukyu1PZeJmdF01wInB7ruTfDmdTYeQK2VMfmHOP feU70wyFI7AhVPQ2SbM1fUDVvwkG0kS1Q+qIX6X5EHqbOER9g0cUbphEYIvFpMQD ICr2m2LpeJGem1HMCtZb/j+rChRJtvKytIk60QC+lXDzhRhiR9irM7pfCNiJ1RW2 xCLhjEkCNtMYVw/5KpdqmN91GNBnp7BdhzKSKSW6xiB5CECWzEv9qexQlmxT7OzS 0bsg2EdYLc4C4YkL2fw2UF2y5oYUMI9niS7n0rRZpMbD8eTK6yhAlhHEBWDvhDUT c/ITYnduUf0yW39n9uQNQgYdauxnj8yJwK0eIQNFAIMgBxDZAgm+s0KmxAyCNg/P c8uSzZBPFdpHCBbeSybGCBEpABn9EULervJJ8KEFBeoaoNSqMvg= =IVGV -----END PGP SIGNATURE----- --rlmvscpkcfr6pslq--