From owner-freebsd-fs@freebsd.org Sun Jul 3 19:30:00 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4B760B90163 for ; Sun, 3 Jul 2016 19:30:00 +0000 (UTC) (envelope-from julien@perdition.city) Received: from relay-b02.edpnet.be (relay-b02.edpnet.be [212.71.1.222]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "edpnet.email", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ED8B721B1 for ; Sun, 3 Jul 2016 19:29:59 +0000 (UTC) (envelope-from julien@perdition.city) X-ASG-Debug-ID: 1467574186-0a7b8d120f3f3c640001-3nHGF7 Received: from mordor.lan (213.219.165.225.bro01.dyn.edpnet.net [213.219.165.225]) by relay-b02.edpnet.be with ESMTP id 8v4NG8TM8VMht7iG (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 03 Jul 2016 21:29:47 +0200 (CEST) X-Barracuda-Envelope-From: julien@perdition.city X-Barracuda-Effective-Source-IP: 213.219.165.225.bro01.dyn.edpnet.net[213.219.165.225] X-Barracuda-Apparent-Source-IP: 213.219.165.225 Date: Sun, 3 Jul 2016 21:29:46 +0200 From: Julien Cigar To: Ben RUBSON Cc: freebsd-fs@freebsd.org Subject: Re: HAST + ZFS + NFS + CARP Message-ID: <20160703192945.GE41276@mordor.lan> X-ASG-Orig-Subj: Re: HAST + ZFS + NFS + CARP References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="M/SuVGWktc5uNpra" Content-Disposition: inline In-Reply-To: <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> User-Agent: Mutt/1.6.1 (2016-04-27) X-Barracuda-Connect: 213.219.165.225.bro01.dyn.edpnet.net[213.219.165.225] X-Barracuda-Start-Time: 1467574186 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://212.71.1.222:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 4639 X-Virus-Scanned: by bsmtpd at edpnet.be X-Barracuda-BRTS-Status: 1 X-Barracuda-Bayes: INNOCENT GLOBAL 0.5000 1.0000 0.0100 X-Barracuda-Spam-Score: 0.51 X-Barracuda-Spam-Status: No, SCORE=0.51 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=6.0 tests=BSF_SC1_TG070 X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.30983 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.50 BSF_SC1_TG070 Custom Rule TG070 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Jul 2016 19:30:00 -0000 --M/SuVGWktc5uNpra Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jul 02, 2016 at 05:04:22PM +0200, Ben RUBSON wrote: >=20 > > On 30 Jun 2016, at 20:57, Julien Cigar wrote: > >=20 > > On Thu, Jun 30, 2016 at 11:32:17AM -0500, Chris Watson wrote: > >>=20 > >>=20 > >> Sent from my iPhone 5 > >>=20 > >>>=20 > >>>>=20 > >>>> Yes that's another option, so a zpool with two mirrors (local +=20 > >>>> exported iSCSI) ? > >>>=20 > >>> Yes, you would then have a real time replication solution (as HAST), = compared to ZFS send/receive which is not. > >>> Depends on what you need :) > >>>=20 > >>>>=20 > >>>>> ZFS would then know as soon as a disk is failing. > >>=20 > >> So as an aside, but related, for those watching this from the peanut g= allery and for the benefit of the OP perhaps those that run with this setup= might give some best practices and tips here in this thread on making this= a good reliable setup. I can see someone reading this thread and tossing t= wo crappy Ethernet cards in a box and then complaining it doesn't work well= =2E=20 > >=20 > > It would be more than welcome indeed..! I have the feeling that HAST > > isn't that much used (but maybe I am wrong) and it's difficult to find= =20 > > informations on it's reliability and concrete long-term use cases... > >=20 > > Also the pros vs cons of HAST vs iSCSI >=20 > I made further testing today. >=20 > # serverA, serverB : > kern.iscsi.ping_timeout=3D5 > kern.iscsi.iscsid_timeout=3D5 > kern.iscsi.login_timeout=3D5 > kern.iscsi.fail_on_disconnection=3D1 >=20 > # Preparation : > - serverB : let's make 2 iSCSI targets : rem3, rem4. > - serverB : let's start ctld. > - serverA : let's create a mirror pool made of 4 disks : loc1, loc2, rem3= , rem4. > - serverA : pool is healthy. >=20 > # Test 1 : > - serverA : put a lot of data into the pool ; > - serverB : stop ctld ; > - serverA : put a lot of data into the pool ; > - serverB : start ctld ; > - serverA : make all pool disks online : it works, pool is healthy. >=20 > # Test 2 : > - serverA : put a lot of data into the pool ; > - serverA : export the pool ; > - serverB : import the pool : it does not work, as ctld locks the disks != Good news, nice protection (both servers won't be able to access the same = disks at the same time). > - serverB : stop ctld ; > - serverB : import the pool : it works, 2 disks missing ; > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > - serverB : make all pool disks online : it works, pool is healthy. >=20 > # Test 3 : > - serverA : put a lot of data into the pool ; > - serverB : stop ctld ; > - serverA : put a lot of data into the pool ; > - serverB : import the pool : it works, 2 disks missing ; > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > - serverB : make all pool disks online : it works, pool is healthy, but o= f course data written at step3 is lost. >=20 > # Test 4 : > - serverA : put a lot of data into the pool ; > - serverB : stop ctld ; > - serverA : put a lot of data into the pool ; > - serverA : export the pool ; > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > - serverB : import the pool : it works, pool is healthy, data written at = step3 is here. >=20 > # Test 5 : > - serverA : rsync a huge remote repo into the pool in the background ; > - serverB : stop ctld ; > - serverA : 2 disks missing, but rsync still runs flawlessly ; > - serverB : start ctld ; > - serverA : make all pool disks online : it works, pool is healthy. > - serverB : ifconfig down ; > - serverA : 2 disks missing, but rsync still runs flawlessly ; > - serverB : ifconfig up ; > - serverA : make all pool disks online : it works, pool is healthy. > - serverB : power reset ! > - serverA : 2 disks missing, but rsync still runs flawlessly ; > - serverB : let's wait for server to be up ; > - serverA : make all pool disks online : it works, pool is healthy. >=20 > Quite happy with these tests actually :) Thank you very much for thoses quick tests! I'll start my own ones tomorrow, but based on your preliminary it *seems* that ZFS + iSCSI combinaison could be a potential candidate for what I'd like to do..! >=20 > Ben >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --M/SuVGWktc5uNpra Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCgAGBQJXeWemAAoJELK7NxCiBCPA674P/2dc4AIDZxgnRYbp4qkhL1nB k+tdijxn1Hiqx1n2sRcR4SDVl+hVLXcPHrM6vH6QPCyXmVFhSwdmJBjhLN/dRGII nT2Bnu0ta4UuwCeRpLcJMGjwfPlqJTjTCrohF1Poa6nAa12O9F4NPcqwexj5/ogR lH7/+moVTFyp9jGea7V+iPxbR95RIbT4m1Juq8L8q7PmRWTv9PD1+kgkSOZ6HB+E 2NdjRiQ5CP+Bl5V09goJ4kzovAQSfGnnU+WPUkiPcX2L2jXx1gSau/vJPCpV1Z/6 eWrETdFc9XIsXB9hFGBQSK1cF+qER27rQ6ChN4hA1fgpZjc7I42j0YjuXxBRAWlq FXYpC79aSkwBR9khmNJrsDH5eQ/tx+irlTZjIZPY0fgBGqUZcVxReIvipS6fcVDw cXI8sFzEkNxBdqq4tLhs6p88lM5sTkXa/hlULRaqlhBPznDzDXr+T5MBYJDT26S+ OAugINKwVHjVq2SzWfH8hbJuEvGY9G8X234OUZmXEwAcMwyI8A3n1gYrDcUcF9Nf ClwLL0rTpJ+tpyA7kKmoIVdWwOmDMI28o1CBaw5h10bOwf5OTugqQ9jQyn292JXB PSFxxsohz6ICbgwstDssgQ7QgyUbto/SsFG5V2/eNkKWrZt1SOLbICFCOjSjiuRI Rilz6dULr2CRatqBaS+p =535T -----END PGP SIGNATURE----- --M/SuVGWktc5uNpra-- From owner-freebsd-fs@freebsd.org Sun Jul 3 21:47:35 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 685E6B90C8D for ; Sun, 3 Jul 2016 21:47:35 +0000 (UTC) (envelope-from julien@perdition.city) Received: from relay-b03.edpnet.be (relay-b03.edpnet.be [212.71.1.220]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "edpnet.email", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1957A2B02 for ; Sun, 3 Jul 2016 21:47:34 +0000 (UTC) (envelope-from julien@perdition.city) X-ASG-Debug-ID: 1467582443-0a88181ce78120c0001-3nHGF7 Received: from mordor.lan (213.219.165.225.bro01.dyn.edpnet.net [213.219.165.225]) by relay-b03.edpnet.be with ESMTP id 3LvHQYWjClWQyiMq (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 03 Jul 2016 23:47:25 +0200 (CEST) X-Barracuda-Envelope-From: julien@perdition.city X-Barracuda-Effective-Source-IP: 213.219.165.225.bro01.dyn.edpnet.net[213.219.165.225] X-Barracuda-Apparent-Source-IP: 213.219.165.225 Date: Sun, 3 Jul 2016 23:47:23 +0200 From: Julien Cigar To: Ben RUBSON Cc: freebsd-fs@freebsd.org Subject: Re: HAST + ZFS + NFS + CARP Message-ID: <20160703214723.GF41276@mordor.lan> X-ASG-Orig-Subj: Re: HAST + ZFS + NFS + CARP References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> <20160703192945.GE41276@mordor.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="w3uUfsyyY1Pqa/ej" Content-Disposition: inline In-Reply-To: <20160703192945.GE41276@mordor.lan> User-Agent: Mutt/1.6.1 (2016-04-27) X-Barracuda-Connect: 213.219.165.225.bro01.dyn.edpnet.net[213.219.165.225] X-Barracuda-Start-Time: 1467582443 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://212.71.1.220:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 5666 X-Virus-Scanned: by bsmtpd at edpnet.be X-Barracuda-BRTS-Status: 1 X-Barracuda-Bayes: INNOCENT GLOBAL 0.5000 1.0000 0.0100 X-Barracuda-Spam-Score: 0.51 X-Barracuda-Spam-Status: No, SCORE=0.51 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=6.0 tests=BSF_SC1_TG070 X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.30986 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.50 BSF_SC1_TG070 Custom Rule TG070 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Jul 2016 21:47:35 -0000 --w3uUfsyyY1Pqa/ej Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jul 03, 2016 at 09:29:46PM +0200, Julien Cigar wrote: > On Sat, Jul 02, 2016 at 05:04:22PM +0200, Ben RUBSON wrote: > >=20 > > > On 30 Jun 2016, at 20:57, Julien Cigar wrote: > > >=20 > > > On Thu, Jun 30, 2016 at 11:32:17AM -0500, Chris Watson wrote: > > >>=20 > > >>=20 > > >> Sent from my iPhone 5 > > >>=20 > > >>>=20 > > >>>>=20 > > >>>> Yes that's another option, so a zpool with two mirrors (local +=20 > > >>>> exported iSCSI) ? > > >>>=20 > > >>> Yes, you would then have a real time replication solution (as HAST)= , compared to ZFS send/receive which is not. > > >>> Depends on what you need :) > > >>>=20 > > >>>>=20 > > >>>>> ZFS would then know as soon as a disk is failing. > > >>=20 > > >> So as an aside, but related, for those watching this from the peanut= gallery and for the benefit of the OP perhaps those that run with this set= up might give some best practices and tips here in this thread on making th= is a good reliable setup. I can see someone reading this thread and tossing= two crappy Ethernet cards in a box and then complaining it doesn't work we= ll.=20 > > >=20 > > > It would be more than welcome indeed..! I have the feeling that HAST > > > isn't that much used (but maybe I am wrong) and it's difficult to fin= d=20 > > > informations on it's reliability and concrete long-term use cases... > > >=20 > > > Also the pros vs cons of HAST vs iSCSI > >=20 > > I made further testing today. > >=20 > > # serverA, serverB : > > kern.iscsi.ping_timeout=3D5 > > kern.iscsi.iscsid_timeout=3D5 > > kern.iscsi.login_timeout=3D5 > > kern.iscsi.fail_on_disconnection=3D1 > >=20 > > # Preparation : > > - serverB : let's make 2 iSCSI targets : rem3, rem4. > > - serverB : let's start ctld. > > - serverA : let's create a mirror pool made of 4 disks : loc1, loc2, re= m3, rem4. > > - serverA : pool is healthy. > >=20 > > # Test 1 : > > - serverA : put a lot of data into the pool ; > > - serverB : stop ctld ; > > - serverA : put a lot of data into the pool ; > > - serverB : start ctld ; > > - serverA : make all pool disks online : it works, pool is healthy. > >=20 > > # Test 2 : > > - serverA : put a lot of data into the pool ; > > - serverA : export the pool ; > > - serverB : import the pool : it does not work, as ctld locks the disks= ! Good news, nice protection (both servers won't be able to access the sam= e disks at the same time). > > - serverB : stop ctld ; > > - serverB : import the pool : it works, 2 disks missing ; > > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > > - serverB : make all pool disks online : it works, pool is healthy. > >=20 > > # Test 3 : > > - serverA : put a lot of data into the pool ; > > - serverB : stop ctld ; > > - serverA : put a lot of data into the pool ; > > - serverB : import the pool : it works, 2 disks missing ; > > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > > - serverB : make all pool disks online : it works, pool is healthy, but= of course data written at step3 is lost. > >=20 > > # Test 4 : > > - serverA : put a lot of data into the pool ; > > - serverB : stop ctld ; > > - serverA : put a lot of data into the pool ; > > - serverA : export the pool ; > > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > > - serverB : import the pool : it works, pool is healthy, data written a= t step3 is here. > >=20 > > # Test 5 : > > - serverA : rsync a huge remote repo into the pool in the background ; > > - serverB : stop ctld ; > > - serverA : 2 disks missing, but rsync still runs flawlessly ; > > - serverB : start ctld ; > > - serverA : make all pool disks online : it works, pool is healthy. > > - serverB : ifconfig down ; > > - serverA : 2 disks missing, but rsync still runs flawlessly ; > > - serverB : ifconfig up ; > > - serverA : make all pool disks online : it works, pool is healthy. > > - serverB : power reset ! > > - serverA : 2 disks missing, but rsync still runs flawlessly ; > > - serverB : let's wait for server to be up ; > > - serverA : make all pool disks online : it works, pool is healthy. > >=20 > > Quite happy with these tests actually :) >=20 > Thank you very much for thoses quick tests! I'll start my own ones > tomorrow, but based on your preliminary it *seems* that ZFS + iSCSI > combinaison could be a potential candidate for what I'd like to do..! another question from a performance point of view, imagine that you=20 create a single mirror zpool, something like: $> zpool create storage mirror loc1 loc2 rem1 rem2 (where rem1 and rem2 are iSCSI disks) I guess that ZFS will split the read requests accross all devices in order to maximize performance... which could lead to contrary to what is expecpted when iSCSI disks are involved, no? Is there some sysctl params which could prevent this unexpected behavior? >=20 > >=20 > > Ben > >=20 > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >=20 > --=20 > Julien Cigar > Belgian Biodiversity Platform (http://www.biodiversity.be) > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > No trees were killed in the creation of this message. > However, many electrons were terribly inconvenienced. --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --w3uUfsyyY1Pqa/ej Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCgAGBQJXeYfmAAoJELK7NxCiBCPAllkQAJq5GNwdUeTIErA12G7u8T4t mXL2fBao7Pg864cS60FnBLnBcaOnkYoRpVbY3vcp2Zs8Xjj2z/Hc0fDgYrOC0h1T OLTPt/wftlmyhh2slJ1xMyRhiax1LhKRAgYxu8okk+ArGUkzmKUmHNsXptJ6ZGwm p0ZQSiMkDgRkImgV4oFWzpe6rrlsaamYhzt37JCsAjwhb7vxewZ7G9+FgOShHaaB VHVmvucKadGG8mzRrBjnj7Y4wYF2gDKG3c7Gf7VnfQPLX/Fk3v3C4D92U13I4hYn NDtL6R9UQsCzcw4jjjBv/GK+3Ha68Kou8kloP+usFyxzDBBd7H5JE8ggEyE8NZAE cViSScxNwZeVEdjC1sqvoBbl7DErqe6pZg34V0jnp/i2ueLQ7mn4DjOV6XaNZvVb DaTIHtoxabBuvjMCHafr86jfQB9KrdetiqRi/GphmmrJhalb6w4e9AqctVZc9ZYO qoWU/OOUqqVrsIB5GuM8emjpeoaDKY2V0ZNxvWQO2v50hK4mYkSLthCLoXbK0gKk qSe4RtilwaSEyVdsvg4ICDALKnW4SaAXRIRxYPRb2nC0gJUXi1Do8ciZEztUZ9rW wYGFdqn8BVYeJdse8HBQ8gwafp72bRrMf7ecrCmkiPCuxi1kVeAXxvhuYjGi5e8a BG82TpIku2fHYsdrbK0D =eQxz -----END PGP SIGNATURE----- --w3uUfsyyY1Pqa/ej-- From owner-freebsd-fs@freebsd.org Sun Jul 3 22:24:59 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6D00B905B2 for ; Sun, 3 Jul 2016 22:24:59 +0000 (UTC) (envelope-from jkh@ixsystems.com) Received: from barracuda.ixsystems.com (barracuda.ixsystems.com [12.229.62.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ixsystems.com", Issuer "Go Daddy Secure Certificate Authority - G2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C83D02A55 for ; Sun, 3 Jul 2016 22:24:59 +0000 (UTC) (envelope-from jkh@ixsystems.com) X-ASG-Debug-ID: 1467584698-08ca0410fd47d10001-3nHGF7 Received: from zimbra.ixsystems.com ([10.246.0.20]) by barracuda.ixsystems.com with ESMTP id ZiIXz8PhNmTmsjwX (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 03 Jul 2016 15:24:58 -0700 (PDT) X-Barracuda-Envelope-From: jkh@ixsystems.com X-Barracuda-RBL-Trusted-Forwarder: 10.246.0.20 X-ASG-Whitelist: Client Received: from localhost (localhost [127.0.0.1]) by zimbra.ixsystems.com (Postfix) with ESMTP id 0E256CA22F0; Sun, 3 Jul 2016 15:24:58 -0700 (PDT) Received: from zimbra.ixsystems.com ([127.0.0.1]) by localhost (zimbra.ixsystems.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id Qj9rJWWKgSE0; Sun, 3 Jul 2016 15:24:57 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.ixsystems.com (Postfix) with ESMTP id 49C91CA2329; Sun, 3 Jul 2016 15:24:57 -0700 (PDT) X-Virus-Scanned: amavisd-new at ixsystems.com Received: from zimbra.ixsystems.com ([127.0.0.1]) by localhost (zimbra.ixsystems.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id WCLMZWVeGxpF; Sun, 3 Jul 2016 15:24:57 -0700 (PDT) Received: from [172.20.0.42] (vpn.ixsystems.com [10.249.0.2]) by zimbra.ixsystems.com (Postfix) with ESMTPSA id 01BF6CA22F0; Sun, 3 Jul 2016 15:24:56 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: HAST + ZFS + NFS + CARP From: Jordan Hubbard X-ASG-Orig-Subj: Re: HAST + ZFS + NFS + CARP In-Reply-To: <20160703214723.GF41276@mordor.lan> Date: Sun, 3 Jul 2016 15:24:56 -0700 Cc: Ben RUBSON , freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <65906F84-CFFC-40E9-8236-56AFB6BE2DE1@ixsystems.com> References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> <20160703192945.GE41276@mordor.lan> <20160703214723.GF41276@mordor.lan> To: Julien Cigar X-Mailer: Apple Mail (2.3124) X-Barracuda-Connect: UNKNOWN[10.246.0.20] X-Barracuda-Start-Time: 1467584698 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://10.246.0.26:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at ixsystems.com X-Barracuda-BRTS-Status: 1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Jul 2016 22:25:00 -0000 > On Jul 3, 2016, at 2:47 PM, Julien Cigar = wrote: >=20 > I guess that ZFS will split the read requests accross all devices in > order to maximize performance... which could lead to contrary to what = is > expecpted when iSCSI disks are involved, no? > Is there some sysctl params which could prevent this unexpected > behavior? Nope. You will suffer the performance implications of layering a = filesystem that expects =E2=80=9Crotating media or SSDs=E2=80=9D (with = the innate ability to parallelize multiple requests in a way that ADD = performance) on top of a system which is now serializing the requests = across an internet connection to another software layer which may offer = no performance benefits to having multiple LUNs at all. You can try = iSCSI-specific tricks like MPIO to try and increase performance, but ZFS = itself is just going to treat everything it sees as =E2=80=9Ca disk=E2=80=9D= and so physical concepts like mirrors or multiple vdevs for performance = won=E2=80=99t translate across. Example question: What=E2=80=99s the point of writing multiple copies of = data across virtual disks in a mirror configuration if the underlying = storage for the virtual disks is already redundant and the I/Os to it = serialize? Example Answer: There is no point. In fact, it=E2=80=99s a = pessimization to do so. This is not a lot different than running ZFS on top of RAID controllers = that turn N physical disks into 1 or more virtual disks. You have to = make entirely different performance decisions based on such scenarios = and that=E2=80=99s just the way it is, which is also why we don=E2=80=99t = recommend doing that. - Jordan From owner-freebsd-fs@freebsd.org Mon Jul 4 05:56:32 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2943BB91E3F for ; Mon, 4 Jul 2016 05:56:32 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22e.google.com (mail-wm0-x22e.google.com [IPv6:2a00:1450:400c:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ADDB42E9F for ; Mon, 4 Jul 2016 05:56:31 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22e.google.com with SMTP id f126so89614848wma.1 for ; Sun, 03 Jul 2016 22:56:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=taqJckWj2YiqIN870q/Hv6OXvldTcsXXo436yNIjIs8=; b=0an3ErElBAlnKicuF1RsQkzNDddhUqIlw3DRGOY7pt5VEM4fDILbocAr08W6/HsbUD KAZeiLA9WLB2TERjZt5RfrTwSQ22bKGG9ijaIhqRFR8QXp5zoqLn9rNPD4x/4IhHYzFQ mnaYuPXelW0pT9sltCiKDK6QyCvwTcf6UV5PlTTmpdhYWHsxXeSTYik9ahSL4qoT7D07 LG4Aysee/1BkRjFeb2rqzNCHSklTkv5TfXR4f/V2I+apu2lX61EgwU9gaYle8n30L2MF eXajfxX6N7ddNUFIeKnnyhguy46kPuRE1CZLLFv13iquCKL4PZTB5Y5XQudLH/zKgUO9 hZwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=taqJckWj2YiqIN870q/Hv6OXvldTcsXXo436yNIjIs8=; b=YFHAOuJRZ6VQiW56ahCJkZd0+fbfZKCb04OXRXkujIlHmU/dbqNtD7pEoEpcvSJFQ7 C7p8K9YZK/ABcaq30izs7Np/dUt5CJqJVyAbnLPEkCGML4EpyQ2+q0RMrh8N76P3T5Za 6oYIvtmIaVzXRZ61D/10PgQvmSCSDGX3BBFLgXVFKuL4p6ulSZLk1EnNfOCBeUa8gAlr x7NPwTTu7J5Juw3BrjC8223dSkzyoOaFGLJr7IhEQMqru7acgiAhDw9WjH+7YRyYSzkf r5FdeFowllqqlRdN0KqJIFL58SfPFV/sx8CnttVMFkQQF0vX1QrWy4IQdo8sLv4Z797l RpGQ== X-Gm-Message-State: ALyK8tJMur8D2rdCXTLzFZ13C4XPqs6xQ7PX/dLQjPbq4bcdLHtnvSBiqif5itujie1INQ== X-Received: by 10.28.144.80 with SMTP id s77mr9064046wmd.41.1467611789412; Sun, 03 Jul 2016 22:56:29 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id dv8sm3583321wjd.49.2016.07.03.22.56.28 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 03 Jul 2016 22:56:28 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: HAST + ZFS + NFS + CARP From: Ben RUBSON In-Reply-To: <20160703214723.GF41276@mordor.lan> Date: Mon, 4 Jul 2016 07:56:27 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> <20160703192945.GE41276@mordor.lan> <20160703214723.GF41276@mordor.lan> To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 05:56:32 -0000 > On 03 Jul 2016, at 23:47, Julien Cigar wrote: >=20 > On Sun, Jul 03, 2016 at 09:29:46PM +0200, Julien Cigar wrote: >> On Sat, Jul 02, 2016 at 05:04:22PM +0200, Ben RUBSON wrote: >>>=20 >>>> On 30 Jun 2016, at 20:57, Julien Cigar = wrote: >>>>=20 >>>> On Thu, Jun 30, 2016 at 11:32:17AM -0500, Chris Watson wrote: >>>>>=20 >>>>>=20 >>>>> Sent from my iPhone 5 >>>>>=20 >>>>>>=20 >>>>>>>=20 >>>>>>> Yes that's another option, so a zpool with two mirrors (local +=20= >>>>>>> exported iSCSI) ? >>>>>>=20 >>>>>> Yes, you would then have a real time replication solution (as = HAST), compared to ZFS send/receive which is not. >>>>>> Depends on what you need :) >>>>>>=20 >>>>>>>=20 >>>>>>>> ZFS would then know as soon as a disk is failing. >>>>>=20 >>>>> So as an aside, but related, for those watching this from the = peanut gallery and for the benefit of the OP perhaps those that run with = this setup might give some best practices and tips here in this thread = on making this a good reliable setup. I can see someone reading this = thread and tossing two crappy Ethernet cards in a box and then = complaining it doesn't work well.=20 >>>>=20 >>>> It would be more than welcome indeed..! I have the feeling that = HAST >>>> isn't that much used (but maybe I am wrong) and it's difficult to = find=20 >>>> informations on it's reliability and concrete long-term use = cases... >>>>=20 >>>> Also the pros vs cons of HAST vs iSCSI >>>=20 >>> I made further testing today. >>>=20 >>> # serverA, serverB : >>> kern.iscsi.ping_timeout=3D5 >>> kern.iscsi.iscsid_timeout=3D5 >>> kern.iscsi.login_timeout=3D5 >>> kern.iscsi.fail_on_disconnection=3D1 >>>=20 >>> # Preparation : >>> - serverB : let's make 2 iSCSI targets : rem3, rem4. >>> - serverB : let's start ctld. >>> - serverA : let's create a mirror pool made of 4 disks : loc1, loc2, = rem3, rem4. >>> - serverA : pool is healthy. >>>=20 >>> # Test 1 : >>> - serverA : put a lot of data into the pool ; >>> - serverB : stop ctld ; >>> - serverA : put a lot of data into the pool ; >>> - serverB : start ctld ; >>> - serverA : make all pool disks online : it works, pool is healthy. >>>=20 >>> # Test 2 : >>> - serverA : put a lot of data into the pool ; >>> - serverA : export the pool ; >>> - serverB : import the pool : it does not work, as ctld locks the = disks ! Good news, nice protection (both servers won't be able to access = the same disks at the same time). >>> - serverB : stop ctld ; >>> - serverB : import the pool : it works, 2 disks missing ; >>> - serverA : let's make 2 iSCSI targets : rem1, rem2 ; >>> - serverB : make all pool disks online : it works, pool is healthy. >>>=20 >>> # Test 3 : >>> - serverA : put a lot of data into the pool ; >>> - serverB : stop ctld ; >>> - serverA : put a lot of data into the pool ; >>> - serverB : import the pool : it works, 2 disks missing ; >>> - serverA : let's make 2 iSCSI targets : rem1, rem2 ; >>> - serverB : make all pool disks online : it works, pool is healthy, = but of course data written at step3 is lost. >>>=20 >>> # Test 4 : >>> - serverA : put a lot of data into the pool ; >>> - serverB : stop ctld ; >>> - serverA : put a lot of data into the pool ; >>> - serverA : export the pool ; >>> - serverA : let's make 2 iSCSI targets : rem1, rem2 ; >>> - serverB : import the pool : it works, pool is healthy, data = written at step3 is here. >>>=20 >>> # Test 5 : >>> - serverA : rsync a huge remote repo into the pool in the background = ; >>> - serverB : stop ctld ; >>> - serverA : 2 disks missing, but rsync still runs flawlessly ; >>> - serverB : start ctld ; >>> - serverA : make all pool disks online : it works, pool is healthy. >>> - serverB : ifconfig down ; >>> - serverA : 2 disks missing, but rsync still runs flawlessly ; >>> - serverB : ifconfig up ; >>> - serverA : make all pool disks online : it works, pool is healthy. >>> - serverB : power reset ! >>> - serverA : 2 disks missing, but rsync still runs flawlessly ; >>> - serverB : let's wait for server to be up ; >>> - serverA : make all pool disks online : it works, pool is healthy. >>>=20 >>> Quite happy with these tests actually :) >>=20 >> Thank you very much for thoses quick tests! I'll start my own ones >> tomorrow, but based on your preliminary it *seems* that ZFS + iSCSI >> combinaison could be a potential candidate for what I'd like to do..! >=20 > another question from a performance point of view, imagine that you=20 > create a single mirror zpool, something like: > $> zpool create storage mirror loc1 loc2 rem1 rem2 >=20 > (where rem1 and rem2 are iSCSI disks) >=20 > I guess that ZFS will split the read requests accross all devices in > order to maximize performance... which could lead to contrary to what = is > expecpted when iSCSI disks are involved, no? Not necessarily no, if your network card is not a bottleneck. If you only have 1Gbps adapters, forget this solution. You should have at least 10Gbps adapters, and depending on how many = disks you have, you would have to go with 25/40Gbps adapters... > Is there some sysctl params which could prevent this unexpected > behavior? >=20 >>=20 >>>=20 >>> Ben >>>=20 >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >>=20 >> --=20 >> Julien Cigar >> Belgian Biodiversity Platform (http://www.biodiversity.be) >> PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 >> No trees were killed in the creation of this message. >> However, many electrons were terribly inconvenienced. >=20 >=20 >=20 > --=20 > Julien Cigar > Belgian Biodiversity Platform (http://www.biodiversity.be) > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > No trees were killed in the creation of this message. > However, many electrons were terribly inconvenienced. From owner-freebsd-fs@freebsd.org Mon Jul 4 06:05:44 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B72EBB91FC1 for ; Mon, 4 Jul 2016 06:05:44 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x230.google.com (mail-wm0-x230.google.com [IPv6:2a00:1450:400c:c09::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48E1A2189 for ; Mon, 4 Jul 2016 06:05:44 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x230.google.com with SMTP id 187so19509991wmz.1 for ; Sun, 03 Jul 2016 23:05:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=QItGqZLKOiOpEjOND7ggoU7vUSG9BeckU2CR6iaoWVk=; b=MaZ7XppwPD1DG3ANKm5B7329UbwSS0DNU7ynBNx1F1InJ3Y7cqwppPHbQH25V3hzBk /Mm3c3YU5IwbCm2rn4wJfpq55J7xOfV7StTVv/SuEaPHMLCehM7g7YDsP9CqF2SZg+jN 0IXkm78jRndwBOUwbVsNeyUXyPNhmIppSRdm/y9WBTZFyi0FthjiK5pXZ5YMYqyAlODD bOSF66RPHPkOSbWNy4aIgYTvzIPAgrKHG87KhadTIxxLTl6AgVa9jrIibj47p7se3Cnb cKYzb3tSNYiznhOgCDrkHWJ7iNPi2RXfys636fJ0jl5I0y3x9LTT8VTBy9TI9pgwm0BX mOIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=QItGqZLKOiOpEjOND7ggoU7vUSG9BeckU2CR6iaoWVk=; b=Iuebc303HB9a4RHZZMDCGRNQmV8scrTd67KxZVm362t12Fq+QmlzHrDaxZ7/YzzgtE HPciRmbsB/ttVE3Wn3ytRfJFDFgFIC9s5g/Qsb7I31uqQs7PaJsTd9w3NfIedY9DgEFD PCjXxYDnz5S7NRxGPqhVeaFcYnKdnM/VjgephH36wVcVKuiPuPzLsfGtiJyTUl/swfPV m1e4S8bmAQaeeyYPxB2UztNf2eGsBCd8VTJB5jY3n8gvTk7K0hYsfzTOK0aPuSjP1yr5 1C8rhx8r3n5rOPXkntODR2Md/7cogQCQfLkOSiJzmG9OIHLOyrz1WQVQRLIi03G/k2PH ZA4w== X-Gm-Message-State: ALyK8tKU7KVZVLvccgy2+dZrT/zZupP4+3KkpUpPrtw3Dyh2T+HEwfRCM0TDhCyzKmyRxQ== X-Received: by 10.28.158.206 with SMTP id h197mr9322348wme.50.1467612342320; Sun, 03 Jul 2016 23:05:42 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id w15sm2306056wmd.11.2016.07.03.23.05.41 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 03 Jul 2016 23:05:41 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: HAST + ZFS + NFS + CARP From: Ben RUBSON In-Reply-To: <65906F84-CFFC-40E9-8236-56AFB6BE2DE1@ixsystems.com> Date: Mon, 4 Jul 2016 08:05:40 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> <20160703192945.GE41276@mordor.lan> <20160703214723.GF41276@mordor.lan> <65906F84-CFFC-40E9-8236-56AFB6BE2DE1@ixsystems.com> To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 06:05:44 -0000 > On 04 Jul 2016, at 00:24, Jordan Hubbard wrote: >=20 >> On Jul 3, 2016, at 2:47 PM, Julien Cigar = wrote: >>=20 >> I guess that ZFS will split the read requests accross all devices in >> order to maximize performance... which could lead to contrary to what = is >> expecpted when iSCSI disks are involved, no? >> Is there some sysctl params which could prevent this unexpected >> behavior? >=20 > Nope. You will suffer the performance implications of layering a = filesystem that expects =E2=80=9Crotating media or SSDs=E2=80=9D (with = the innate ability to parallelize multiple requests in a way that ADD = performance) on top of a system which is now serializing the requests = across an internet connection to another software layer which may offer = no performance benefits to having multiple LUNs at all. You can try = iSCSI-specific tricks like MPIO to try and increase performance, but ZFS = itself is just going to treat everything it sees as =E2=80=9Ca disk=E2=80=9D= and so physical concepts like mirrors or multiple vdevs for performance = won=E2=80=99t translate across. >=20 > Example question: What=E2=80=99s the point of writing multiple copies = of data across virtual disks in a mirror configuration if the underlying = storage for the virtual disks is already redundant and the I/Os to it = serialize? > Example Answer: There is no point. In fact, it=E2=80=99s a = pessimization to do so. Of course Jordan, in this topic, we (well at least me :) make the = following assumption : one iSCSI target/disk =3D one real physical disk (a SAS disk, a SSD = disk...), from a server having its own JBOD, no RAID adapter or = whatever, just what ZFS likes ! > This is not a lot different than running ZFS on top of RAID = controllers that turn N physical disks into 1 or more virtual disks. = You have to make entirely different performance decisions based on such = scenarios and that=E2=80=99s just the way it is, which is also why we = don=E2=80=99t recommend doing that. Of course you loose all ZFS benefits if you only mirror 2 "disks", a big = one from storage array A, the same from storage array B. No interest. From owner-freebsd-fs@freebsd.org Mon Jul 4 08:31:44 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 02C66B8FB49 for ; Mon, 4 Jul 2016 08:31:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E6FD5207B for ; Mon, 4 Jul 2016 08:31:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u648Vh4Z082253 for ; Mon, 4 Jul 2016 08:31:43 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204764] Filesystem deadlock, process in vodead state Date: Mon, 04 Jul 2016 08:31:43 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: daniel@blodan.se X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 08:31:44 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D204764 --- Comment #27 from Daniel Ylitalo --- Anyone aware if this will be released as a patch for 10.3 through freebsd-update or if one should go patch the diff? :) --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Jul 4 12:25:33 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BDBC4B910A9 for ; Mon, 4 Jul 2016 12:25:33 +0000 (UTC) (envelope-from julien@perdition.city) Received: from relay-b02.edpnet.be (relay-b02.edpnet.be [212.71.1.222]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "edpnet.email", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 670782091 for ; Mon, 4 Jul 2016 12:25:32 +0000 (UTC) (envelope-from julien@perdition.city) X-ASG-Debug-ID: 1467635127-0a7b8d55be1d0ae30001-3nHGF7 Received: from mordor.lan (213.219.165.225.bro01.dyn.edpnet.net [213.219.165.225]) by relay-b02.edpnet.be with ESMTP id pwKqCi0SoSqyoqCO (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 04 Jul 2016 14:25:29 +0200 (CEST) X-Barracuda-Envelope-From: julien@perdition.city X-Barracuda-Effective-Source-IP: 213.219.165.225.bro01.dyn.edpnet.net[213.219.165.225] X-Barracuda-Apparent-Source-IP: 213.219.165.225 Date: Mon, 4 Jul 2016 14:25:27 +0200 From: Julien Cigar To: Ben RUBSON Cc: freebsd-fs@freebsd.org Subject: Re: HAST + ZFS + NFS + CARP Message-ID: <20160704122527.GH41276@mordor.lan> X-ASG-Orig-Subj: Re: HAST + ZFS + NFS + CARP References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="y9PDtDHaFrXNoMPU" Content-Disposition: inline In-Reply-To: <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> User-Agent: Mutt/1.6.1 (2016-04-27) X-Barracuda-Connect: 213.219.165.225.bro01.dyn.edpnet.net[213.219.165.225] X-Barracuda-Start-Time: 1467635128 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://212.71.1.222:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 5175 X-Virus-Scanned: by bsmtpd at edpnet.be X-Barracuda-BRTS-Status: 1 X-Barracuda-Bayes: INNOCENT GLOBAL 0.5000 1.0000 0.0100 X-Barracuda-Spam-Score: 0.51 X-Barracuda-Spam-Status: No, SCORE=0.51 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=6.0 tests=BSF_SC1_TG070 X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.31003 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.50 BSF_SC1_TG070 Custom Rule TG070 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 12:25:33 -0000 --y9PDtDHaFrXNoMPU Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jul 02, 2016 at 05:04:22PM +0200, Ben RUBSON wrote: >=20 > > On 30 Jun 2016, at 20:57, Julien Cigar wrote: > >=20 > > On Thu, Jun 30, 2016 at 11:32:17AM -0500, Chris Watson wrote: > >>=20 > >>=20 > >> Sent from my iPhone 5 > >>=20 > >>>=20 > >>>>=20 > >>>> Yes that's another option, so a zpool with two mirrors (local +=20 > >>>> exported iSCSI) ? > >>>=20 > >>> Yes, you would then have a real time replication solution (as HAST), = compared to ZFS send/receive which is not. > >>> Depends on what you need :) > >>>=20 > >>>>=20 > >>>>> ZFS would then know as soon as a disk is failing. > >>=20 > >> So as an aside, but related, for those watching this from the peanut g= allery and for the benefit of the OP perhaps those that run with this setup= might give some best practices and tips here in this thread on making this= a good reliable setup. I can see someone reading this thread and tossing t= wo crappy Ethernet cards in a box and then complaining it doesn't work well= =2E=20 > >=20 > > It would be more than welcome indeed..! I have the feeling that HAST > > isn't that much used (but maybe I am wrong) and it's difficult to find= =20 > > informations on it's reliability and concrete long-term use cases... > >=20 > > Also the pros vs cons of HAST vs iSCSI >=20 > I made further testing today. >=20 > # serverA, serverB : > kern.iscsi.ping_timeout=3D5 > kern.iscsi.iscsid_timeout=3D5 > kern.iscsi.login_timeout=3D5 > kern.iscsi.fail_on_disconnection=3D1 >=20 > # Preparation : > - serverB : let's make 2 iSCSI targets : rem3, rem4. > - serverB : let's start ctld. > - serverA : let's create a mirror pool made of 4 disks : loc1, loc2, rem3= , rem4. > - serverA : pool is healthy. >=20 > # Test 1 : > - serverA : put a lot of data into the pool ; > - serverB : stop ctld ; > - serverA : put a lot of data into the pool ; > - serverB : start ctld ; > - serverA : make all pool disks online : it works, pool is healthy. >=20 > # Test 2 : > - serverA : put a lot of data into the pool ; > - serverA : export the pool ; > - serverB : import the pool : it does not work, as ctld locks the disks != Good news, nice protection (both servers won't be able to access the same = disks at the same time). > - serverB : stop ctld ; > - serverB : import the pool : it works, 2 disks missing ; > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > - serverB : make all pool disks online : it works, pool is healthy. >=20 > # Test 3 : > - serverA : put a lot of data into the pool ; > - serverB : stop ctld ; > - serverA : put a lot of data into the pool ; > - serverB : import the pool : it works, 2 disks missing ; > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > - serverB : make all pool disks online : it works, pool is healthy, but o= f course data written at step3 is lost. >=20 > # Test 4 : > - serverA : put a lot of data into the pool ; > - serverB : stop ctld ; > - serverA : put a lot of data into the pool ; > - serverA : export the pool ; > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > - serverB : import the pool : it works, pool is healthy, data written at = step3 is here. >=20 > # Test 5 : > - serverA : rsync a huge remote repo into the pool in the background ; > - serverB : stop ctld ; > - serverA : 2 disks missing, but rsync still runs flawlessly ; > - serverB : start ctld ; > - serverA : make all pool disks online : it works, pool is healthy. > - serverB : ifconfig down ; > - serverA : 2 disks missing, but rsync still runs flawlessly ; > - serverB : ifconfig up ; > - serverA : make all pool disks online : it works, pool is healthy. > - serverB : power reset ! > - serverA : 2 disks missing, but rsync still runs flawlessly ; > - serverB : let's wait for server to be up ; > - serverA : make all pool disks online : it works, pool is healthy. >=20 > Quite happy with these tests actually :) another option I had in mind is to export ZVOL through iSCSI: - on serverA: $> zpool create storage mirror /dev/ada1 /dev/ada2 - on serverB: $> zpool create storage mirror /dev/ada1 /dev/ada2 create a 50G dedicated redundant storage for serverC: - on serverA: $> zfs create -V 50G storage/serverc - on serverB: $> zfs create -V 50G storage/serverc iSCSI export /dev/zvol/storage/serverc from serverA and serverB and create a ZFS dataset on serverC: - on serverC: $> zpool create storage mirror /dev/ivol1 /dev/ivol2 (where ivol1 is the volume from serverA and ivol2 the volume from serverB) create, for example, a dataset for the dovecot service - on serverC: $> zfs create -o compress=3Dlz4 storage/dovecot any thoughts? >=20 > Ben >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --y9PDtDHaFrXNoMPU Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCgAGBQJXelW0AAoJELK7NxCiBCPAN8YP/Rl6MQcdQpjUeKAE5n06V/1g aUUuKOsMvYOKwWLNguyPebd9lROMkcGmT9HPCDJx0QmN6UipLAmn/UtcqoUOxvjX cEJZJBp5RpKibGMXjsLyii3epCAjd/xYzNYckXwXYaSHA6FVekZIjSnxyUYJnPuz Fg2pyW8us2D9emY8bqXQ+WOZvrMX+1RM90RDwjzNjeOWeX92poNm7MhnCPytfDre SkvfgoISbmiExDmqEWkHsXBjMvXGtdpZaqtxlHAM/p4ontdyjOsbkfo0qF7Z8jqp NyAenAbEyrjVousHQICL9P9wHOlgQWSJpuassJgPbQpAm+kDrYKRoJdYG91FJZM9 lzZFWFe/HFVS27xQi5uuGt+Fxdue2FwY42mJNcOlCY6iJe12yJDr/sAUYVQJ3nNi KggK9lEr9bwOtl548IpoRxjgE6epVTu3xFbup5IzVKsW9/C6Olr5ufBJid42aqoe 7diqg5sjO41kanNU3gp0IfV9sH31Grp+thBjqzMSFjfVY4VRWb/sOuBkrFxPvNgJ gtrY5oo5Bs0QdEGmjiB+I1L4PgsBArzYwkzIbjL45/hnwlS6RZc1ORi/4CeQ8O5X YaQzPitlmfMxdd8Rv8/PNWHM+iBGsw1BQPWGsBDg/fJGJbvcQbV8KSUpV9QWiwFZ MiZvWynjeqFY5X62fH59 =inJ8 -----END PGP SIGNATURE----- --y9PDtDHaFrXNoMPU-- From owner-freebsd-fs@freebsd.org Mon Jul 4 12:46:01 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51894B9136E for ; Mon, 4 Jul 2016 12:46:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 417262844 for ; Mon, 4 Jul 2016 12:46:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u64Ck0bf043187 for ; Mon, 4 Jul 2016 12:46:01 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203864] ZFS deadlock between zfs send, zfs rename and ctrl-C Date: Mon, 04 Jul 2016 12:46:01 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: dogfood, needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: peixoto.cassiano@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 12:46:01 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203864 --- Comment #12 from Cassiano Peixoto --- (In reply to roel from comment #11) Hi Roel, Thanks to send me. Do you know some way to simulate this issue? i tried to = just make my second server down and run zrep, but didn't work. Thanks. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Jul 4 12:55:27 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0ABE5B915D1 for ; Mon, 4 Jul 2016 12:55:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EF0062BD8 for ; Mon, 4 Jul 2016 12:55:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u64CtQqW062337 for ; Mon, 4 Jul 2016 12:55:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 209580] ZFS and geli broken with INVARIANTS enabled Date: Mon, 04 Jul 2016 12:55:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: shawn.webb@hardenedbsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 12:55:27 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209580 --- Comment #3 from Shawn Webb --- I'm just checking in. Any updates on this? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Jul 4 12:56:08 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E672CB91618 for ; Mon, 4 Jul 2016 12:56:08 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A83692C5F for ; Mon, 4 Jul 2016 12:56:07 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 8E85828452; Mon, 4 Jul 2016 14:56:04 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id DA82128429; Mon, 4 Jul 2016 14:56:03 +0200 (CEST) Message-ID: <577A5CE3.8060100@quip.cz> Date: Mon, 04 Jul 2016 14:56:03 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: Julien Cigar , Ben RUBSON CC: freebsd-fs@freebsd.org Subject: Re: HAST + ZFS + NFS + CARP References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> <20160704122527.GH41276@mordor.lan> In-Reply-To: <20160704122527.GH41276@mordor.lan> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 12:56:09 -0000 Julien Cigar wrote on 07/04/2016 14:25: > another option I had in mind is to export ZVOL through iSCSI: > - on serverA: $> zpool create storage mirror /dev/ada1 /dev/ada2 > - on serverB: $> zpool create storage mirror /dev/ada1 /dev/ada2 > > create a 50G dedicated redundant storage for serverC: > - on serverA: $> zfs create -V 50G storage/serverc > - on serverB: $> zfs create -V 50G storage/serverc > > iSCSI export /dev/zvol/storage/serverc from serverA and serverB and > create a ZFS dataset on serverC: > - on serverC: $> zpool create storage mirror /dev/ivol1 /dev/ivol2 > > (where ivol1 is the volume from serverA and ivol2 the volume from > serverB) > > create, for example, a dataset for the dovecot service > - on serverC: $> zfs create -o compress=lz4 storage/dovecot I think it will be painfully slow. ZFS on top of ZFS throught iSCSI... too much layering, too much delays. Miroslav Lachman From owner-freebsd-fs@freebsd.org Mon Jul 4 12:57:13 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B7BF7B91665 for ; Mon, 4 Jul 2016 12:57:13 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22b.google.com (mail-wm0-x22b.google.com [IPv6:2a00:1450:400c:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5C0BE2D02 for ; Mon, 4 Jul 2016 12:57:13 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22b.google.com with SMTP id z126so41024925wme.0 for ; Mon, 04 Jul 2016 05:57:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=gZUY3Xqnd6tMBScsGpijAwCqyW0+gS1z4HO/3iYGlCA=; b=ats1/eHdEWpJVIrnKoCZ/THzonH6kYV2j5eepfrHgHDFdBIf71lTseDKQ5bE1H88qV hZJ0ZcGC/Ejac3IARpL2+PMEy8nhNkVw95BSHiELWoP1w7ktQ8fdVlJ3Vcu8pcclPVuB THAg+wnbjC+gWHru4XBv34QVE5OlxWJt18RryWI8Zzr9mhCpyZ8pkEV2ZSPzCFKC/lRc pb1Bwo0bjPgoGIMkVZH2aezuwRnRzNNDYy/gHGIORhS8fyU2iaLduEZ65EomQIDyVwem c2rAWdS61lsetd3qX6IuMxOpC9uKAm2TmphC5LHVywA3teBDSM1nr9djI4H2LNc8TSsT 2Ojg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=gZUY3Xqnd6tMBScsGpijAwCqyW0+gS1z4HO/3iYGlCA=; b=G78kqfPeHIM90nZx4MizNBkhIGO/XlG6OyImIM6GRl6gq55ozwqdFc4VEU+BZNrZfL w+ryqq1ME/HRnIMuS8eto5ebfCX+y8O7vok3G9eC1xtDiV3HsgRXxPydUHl8wDCYx49e UN3XciN3xZ9IkjRDMkIkJ9VsW/91TUucG8syBCV931HLlSu868LjAfqYhRYhs110jN/6 p+gJVytzXsev4iiNd0DWW6Fmpe6GPcSCPHKD+sEjLwLBUktCkfpmebVQ1cIFWgBpm0d+ Dc47CgQqnDbgHpGbmlgD3lzCaSsidniK3GRY7FGYy2hd8Gr3NUY78c+dfRWcI8rVDpyW JYtg== X-Gm-Message-State: ALyK8tLfekLvlev3ecZTMCzZZuMumOqrq7mx3LjZjgWQOOQdeS9CwbQhk6gu4In4w8v8EQ== X-Received: by 10.194.20.132 with SMTP id n4mr5751676wje.12.1467637031657; Mon, 04 Jul 2016 05:57:11 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id o5sm1157638wjy.21.2016.07.04.05.57.10 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 04 Jul 2016 05:57:10 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: HAST + ZFS + NFS + CARP From: Ben RUBSON In-Reply-To: <577A5CE3.8060100@quip.cz> Date: Mon, 4 Jul 2016 14:57:10 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> <20160704122527.GH41276@mordor.lan> <577A5CE3.8060100@quip.cz> To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 12:57:13 -0000 > On 04 Jul 2016, at 14:56, Miroslav Lachman <000.fbsd@quip.cz> wrote: >=20 > Julien Cigar wrote on 07/04/2016 14:25: >=20 >> another option I had in mind is to export ZVOL through iSCSI: >> - on serverA: $> zpool create storage mirror /dev/ada1 /dev/ada2 >> - on serverB: $> zpool create storage mirror /dev/ada1 /dev/ada2 >>=20 >> create a 50G dedicated redundant storage for serverC: >> - on serverA: $> zfs create -V 50G storage/serverc >> - on serverB: $> zfs create -V 50G storage/serverc >>=20 >> iSCSI export /dev/zvol/storage/serverc from serverA and serverB and >> create a ZFS dataset on serverC: >> - on serverC: $> zpool create storage mirror /dev/ivol1 /dev/ivol2 >>=20 >> (where ivol1 is the volume from serverA and ivol2 the volume from >> serverB) >>=20 >> create, for example, a dataset for the dovecot service >> - on serverC: $> zfs create -o compress=3Dlz4 storage/dovecot >=20 > I think it will be painfully slow. ZFS on top of ZFS throught iSCSI... = too much layering, too much delays. And serverC is a SPOF. From owner-freebsd-fs@freebsd.org Mon Jul 4 13:03:45 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8BE3BB917E5 for ; Mon, 4 Jul 2016 13:03:45 +0000 (UTC) (envelope-from jg@internetx.com) Received: from mx1.internetx.com (mx1.internetx.com [62.116.129.39]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1BAF02050 for ; Mon, 4 Jul 2016 13:03:44 +0000 (UTC) (envelope-from jg@internetx.com) Received: from localhost (localhost [127.0.0.1]) by mx1.internetx.com (Postfix) with ESMTP id D730045FC0EA; Mon, 4 Jul 2016 15:03:35 +0200 (CEST) X-Virus-Scanned: InterNetX GmbH amavisd-new at ix-mailer.internetx.de Received: from mx1.internetx.com ([62.116.129.39]) by localhost (ix-mailer.internetx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aMeFDcp9n0mu; Mon, 4 Jul 2016 15:03:30 +0200 (CEST) Received: from ox-groupware-node01.internetx.de (ox.internetx.com [85.236.36.83]) by mx1.internetx.com (Postfix) with ESMTP id 3A82A45FC0E8; Mon, 4 Jul 2016 15:03:30 +0200 (CEST) Received: from ox-groupware-node01.internetx.de (localhost [127.0.0.1]) by ox-groupware-node01.internetx.de (Postfix) with ESMTP id 266E3A1213E; Mon, 4 Jul 2016 15:03:30 +0200 (CEST) Date: Mon, 4 Jul 2016 15:03:30 +0200 (CEST) From: InterNetX - Juergen Gotteswinter To: Miroslav Lachman <000.fbsd@quip.cz>, Ben RUBSON , Julien Cigar Cc: freebsd-fs@freebsd.org Message-ID: <1740049005.1349.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> In-Reply-To: <577A5CE3.8060100@quip.cz> References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> <20160704122527.GH41276@mordor.lan> <577A5CE3.8060100@quip.cz> Subject: Re: HAST + ZFS + NFS + CARP MIME-Version: 1.0 X-Priority: 3 Importance: Medium X-Mailer: Open-Xchange Mailer v7.8.0-Rev27 Organization: InterNetX GmbH X-Originating-Client: com.openexchange.ox.gui.dhtml Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 13:03:45 -0000 i see so much pain upcoming when $something goes wrong, even the smallest piece will end up in drama zfs has been designed as a pragmatic stupid simpel solid solution, why whould one rape this concept with such complexity :( > Miroslav Lachman <000.fbsd@quip.cz> hat am 4. Juli 2016 um 14:56 geschrieben: > > > Julien Cigar wrote on 07/04/2016 14:25: > > > another option I had in mind is to export ZVOL through iSCSI: > > - on serverA: $> zpool create storage mirror /dev/ada1 /dev/ada2 > > - on serverB: $> zpool create storage mirror /dev/ada1 /dev/ada2 > > > > create a 50G dedicated redundant storage for serverC: > > - on serverA: $> zfs create -V 50G storage/serverc > > - on serverB: $> zfs create -V 50G storage/serverc > > > > iSCSI export /dev/zvol/storage/serverc from serverA and serverB and > > create a ZFS dataset on serverC: > > - on serverC: $> zpool create storage mirror /dev/ivol1 /dev/ivol2 > > > > (where ivol1 is the volume from serverA and ivol2 the volume from > > serverB) > > > > create, for example, a dataset for the dovecot service > > - on serverC: $> zfs create -o compress=lz4 storage/dovecot > > I think it will be painfully slow. ZFS on top of ZFS throught iSCSI... > too much layering, too much delays. > > Miroslav Lachman > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Mon Jul 4 17:55:42 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5CDE0B91222 for ; Mon, 4 Jul 2016 17:55:42 +0000 (UTC) (envelope-from jkh@ixsystems.com) Received: from barracuda.ixsystems.com (barracuda.ixsystems.com [12.229.62.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ixsystems.com", Issuer "Go Daddy Secure Certificate Authority - G2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 307762BC7 for ; Mon, 4 Jul 2016 17:55:42 +0000 (UTC) (envelope-from jkh@ixsystems.com) X-ASG-Debug-ID: 1467654941-08ca0410fe51b40001-3nHGF7 Received: from zimbra.ixsystems.com ([10.246.0.20]) by barracuda.ixsystems.com with ESMTP id gdXKAcl8qjjLkiLM (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 04 Jul 2016 10:55:41 -0700 (PDT) X-Barracuda-Envelope-From: jkh@ixsystems.com X-Barracuda-RBL-Trusted-Forwarder: 10.246.0.20 X-ASG-Whitelist: Client Received: from localhost (localhost [127.0.0.1]) by zimbra.ixsystems.com (Postfix) with ESMTP id 8B29FCA4024; Mon, 4 Jul 2016 10:55:41 -0700 (PDT) Received: from zimbra.ixsystems.com ([127.0.0.1]) by localhost (zimbra.ixsystems.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 5Ra626h0xOLI; Mon, 4 Jul 2016 10:55:41 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.ixsystems.com (Postfix) with ESMTP id 2DA1FCA414F; Mon, 4 Jul 2016 10:55:41 -0700 (PDT) X-Virus-Scanned: amavisd-new at ixsystems.com Received: from zimbra.ixsystems.com ([127.0.0.1]) by localhost (zimbra.ixsystems.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id NAscgVdF_ydd; Mon, 4 Jul 2016 10:55:41 -0700 (PDT) Received: from [172.20.0.34] (vpn.ixsystems.com [10.249.0.2]) by zimbra.ixsystems.com (Postfix) with ESMTPSA id DD06DCA4024; Mon, 4 Jul 2016 10:55:40 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: HAST + ZFS + NFS + CARP From: Jordan Hubbard X-ASG-Orig-Subj: Re: HAST + ZFS + NFS + CARP In-Reply-To: Date: Mon, 4 Jul 2016 10:55:40 -0700 Cc: freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <61283600-A41A-4A8A-92F9-7FAFF54DD175@ixsystems.com> References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> <20160703192945.GE41276@mordor.lan> <20160703214723.GF41276@mordor.lan> <65906F84-CFFC-40E9-8236-56AFB6BE2DE1@ixsystems.com> To: Ben RUBSON X-Mailer: Apple Mail (2.3124) X-Barracuda-Connect: UNKNOWN[10.246.0.20] X-Barracuda-Start-Time: 1467654941 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://10.246.0.26:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at ixsystems.com X-Barracuda-BRTS-Status: 1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 17:55:42 -0000 > On Jul 3, 2016, at 11:05 PM, Ben RUBSON wrote: >=20 > Of course Jordan, in this topic, we (well at least me :) make the = following assumption : > one iSCSI target/disk =3D one real physical disk (a SAS disk, a SSD = disk...), from a server having its own JBOD, no RAID adapter or = whatever, just what ZFS likes ! I certainly wouldn=E2=80=99t make that assumption. Once you allow iSCSI = to be the back-end in any solution, end-users will avail themselves of = the flexibility to also export arbitrary or synthetic devices (like = zvols / RAID devices) as =E2=80=9Cdisks=E2=80=9D. You can=E2=80=99t = stop them from doing so, so you might as well incorporate that scenario = into your design. Even if you could somehow enforce the 1:1 mapping of = LUN to disk, iSCSI itself is still going to impose a serialization / = performance / reporting (iSCSI LUNs don=E2=80=99t report SMART status) = penalty that removes a lot of the advantages of having direct physical = access to the media, so one might also ask what you=E2=80=99re gaining = by imposing those restrictions. - Jordan From owner-freebsd-fs@freebsd.org Mon Jul 4 18:06:37 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 035BAB91558 for ; Mon, 4 Jul 2016 18:06:37 +0000 (UTC) (envelope-from jkh@ixsystems.com) Received: from barracuda.ixsystems.com (barracuda.ixsystems.com [12.229.62.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ixsystems.com", Issuer "Go Daddy Secure Certificate Authority - G2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D93712219 for ; Mon, 4 Jul 2016 18:06:36 +0000 (UTC) (envelope-from jkh@ixsystems.com) X-ASG-Debug-ID: 1467655595-08ca0410fd51d70001-3nHGF7 Received: from zimbra.ixsystems.com ([10.246.0.20]) by barracuda.ixsystems.com with ESMTP id FXssCJIzgJKF8ofn (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 04 Jul 2016 11:06:35 -0700 (PDT) X-Barracuda-Envelope-From: jkh@ixsystems.com X-Barracuda-RBL-Trusted-Forwarder: 10.246.0.20 X-ASG-Whitelist: Client Received: from localhost (localhost [127.0.0.1]) by zimbra.ixsystems.com (Postfix) with ESMTP id 7674ACA41D4; Mon, 4 Jul 2016 11:06:35 -0700 (PDT) Received: from zimbra.ixsystems.com ([127.0.0.1]) by localhost (zimbra.ixsystems.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id MiABdm9fTWbL; Mon, 4 Jul 2016 11:06:35 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.ixsystems.com (Postfix) with ESMTP id 22BF3CA41E0; Mon, 4 Jul 2016 11:06:35 -0700 (PDT) X-Virus-Scanned: amavisd-new at ixsystems.com Received: from zimbra.ixsystems.com ([127.0.0.1]) by localhost (zimbra.ixsystems.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id I3335Gtf2c9R; Mon, 4 Jul 2016 11:06:35 -0700 (PDT) Received: from [172.20.0.34] (vpn.ixsystems.com [10.249.0.2]) by zimbra.ixsystems.com (Postfix) with ESMTPSA id CC94DCA41D4; Mon, 4 Jul 2016 11:06:34 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: HAST + ZFS + NFS + CARP From: Jordan Hubbard X-ASG-Orig-Subj: Re: HAST + ZFS + NFS + CARP In-Reply-To: Date: Mon, 4 Jul 2016 11:06:34 -0700 Cc: freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <73DC3B93-60FF-4B20-AF80-5712F77818EF@ixsystems.com> References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> To: Ben RUBSON X-Mailer: Apple Mail (2.3124) X-Barracuda-Connect: UNKNOWN[10.246.0.20] X-Barracuda-Start-Time: 1467655595 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://10.246.0.26:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at ixsystems.com X-Barracuda-BRTS-Status: 1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 18:06:37 -0000 > On Jul 1, 2016, at 11:23 AM, Ben RUBSON wrote: >=20 > Would you say that giving an iSCSI disk to ZFS hides some details of = the raw disk to ZFS ? Yes, of course. > I though that iSCSI would have been a totally "transparent" layer, = transferring all ZFS requests to the raw disk, giving back the answers, = hiding anything. Not really, no. There are other ways of talking to a disk or SSD = device, such as getting S.M.A.R.T. data to see when/if a drive is = failing. Drives also return checksum errors that may be masked by the = iSCSI target. Finally, there is SCSI-2 and there is SCSI-3 (where = things like persistent reservations are implemented). None of these = things are necessarily implemented by iSCSI. - Jordan From owner-freebsd-fs@freebsd.org Mon Jul 4 18:36:49 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9A974B91E57 for ; Mon, 4 Jul 2016 18:36:49 +0000 (UTC) (envelope-from julien@perdition.city) Received: from relay-b02.edpnet.be (relay-b02.edpnet.be [212.71.1.222]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "edpnet.email", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47B722F45 for ; Mon, 4 Jul 2016 18:36:49 +0000 (UTC) (envelope-from julien@perdition.city) X-ASG-Debug-ID: 1467657403-0a7b8d6119236010001-3nHGF7 Received: from mordor.lan (213.219.165.225.bro01.dyn.edpnet.net [213.219.165.225]) by relay-b02.edpnet.be with ESMTP id Ok6G5uEUrEaLdbB9 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 04 Jul 2016 20:36:45 +0200 (CEST) X-Barracuda-Envelope-From: julien@perdition.city X-Barracuda-Effective-Source-IP: 213.219.165.225.bro01.dyn.edpnet.net[213.219.165.225] X-Barracuda-Apparent-Source-IP: 213.219.165.225 Date: Mon, 4 Jul 2016 20:36:43 +0200 From: Julien Cigar To: Jordan Hubbard Cc: Ben RUBSON , freebsd-fs@freebsd.org Subject: Re: HAST + ZFS + NFS + CARP Message-ID: <20160704183643.GI41276@mordor.lan> X-ASG-Orig-Subj: Re: HAST + ZFS + NFS + CARP References: <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> <20160703192945.GE41276@mordor.lan> <20160703214723.GF41276@mordor.lan> <65906F84-CFFC-40E9-8236-56AFB6BE2DE1@ixsystems.com> <61283600-A41A-4A8A-92F9-7FAFF54DD175@ixsystems.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="81JctsDUVPekGcy+" Content-Disposition: inline In-Reply-To: <61283600-A41A-4A8A-92F9-7FAFF54DD175@ixsystems.com> User-Agent: Mutt/1.6.1 (2016-04-27) X-Barracuda-Connect: 213.219.165.225.bro01.dyn.edpnet.net[213.219.165.225] X-Barracuda-Start-Time: 1467657403 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://212.71.1.222:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 2557 X-Virus-Scanned: by bsmtpd at edpnet.be X-Barracuda-BRTS-Status: 1 X-Barracuda-Bayes: INNOCENT GLOBAL 0.5000 1.0000 0.0100 X-Barracuda-Spam-Score: 0.01 X-Barracuda-Spam-Status: No, SCORE=0.01 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=6.0 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.31010 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 18:36:49 -0000 --81JctsDUVPekGcy+ Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jul 04, 2016 at 10:55:40AM -0700, Jordan Hubbard wrote: >=20 > > On Jul 3, 2016, at 11:05 PM, Ben RUBSON wrote: > >=20 > > Of course Jordan, in this topic, we (well at least me :) make the follo= wing assumption : > > one iSCSI target/disk =3D one real physical disk (a SAS disk, a SSD dis= k...), from a server having its own JBOD, no RAID adapter or whatever, just= what ZFS likes ! >=20 > I certainly wouldn=E2=80=99t make that assumption. Once you allow iSCSI = to be the back-end in any solution, end-users will avail themselves of the = flexibility to also export arbitrary or synthetic devices (like zvols / RAI= D devices) as =E2=80=9Cdisks=E2=80=9D. You can=E2=80=99t stop them from do= ing so, so you might as well incorporate that scenario into your design. E= ven if you could somehow enforce the 1:1 mapping of LUN to disk, iSCSI itse= lf is still going to impose a serialization / performance / reporting (iSCS= I LUNs don=E2=80=99t report SMART status) penalty that removes a lot of the= advantages of having direct physical access to the media, so one might als= o ask what you=E2=80=99re gaining by imposing those restrictions. I think the discussion evolved a bit since I started this thread, the original purpose was to build a low-cost redundant storage for a small infrastructure, no more no less. The context is the following: I work in a small company, partially financed by public funds, we started small, evolved a bit to a point that some redundancy is required for $services.=20 Unfortunately I'm alone to take care of the infrastructure (and it's=20 only 50% of my time) and we don't have that much money :(=20 That's why I was just thinking of two machines with a simple HBA card,=20 2x4To, a zpool mirror on those 4 disks (2 local, 2 iSCSI exported), a=20 NFS share on top, and an easy failover procedure.. I understand that iSCSI hides some details, but as far as I know it's the "lowest" level that you can provide to ZFS when the disks are not local, no ? Anyway, thanks for your feedback, it's greatly appreciated! :) Julien >=20 > - Jordan >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --81JctsDUVPekGcy+ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCgAGBQJXeqy3AAoJELK7NxCiBCPAu6MP/Rp6+bsCOYMeyXeMTEdRaR9/ s+LGJSFv1Oh9wsOBY5NwDb7Zf5uSsGOvWig3JkGKk0OdqXwawmziyoJ7LUDRVy73 NKeb+VfZ0cCcPPCMQhsw1ReQHuaBvUO3ZOZfVfj5BCdPwGbXvcJILKsnNH0vW6vV 7PsyZ8O43ieHvJeFhTPw/0Tq+1xrgNhsWpEY1Tb53ChlVHK++ua3Gqwr9SQjM6Lz fyTM0sh2v/kxeS5Zat3aZnqsWhaEohtHzstYi7oeun9p0/A4pWkfvHfWq+DkcS9A 9Uc7H9fXy0FeJexNetFVHXWgeZtlZFO9V94kbolpn6Op7ojY0wyZvt/LOnRRNH9C 2tk2ep9/5oiEORciEcVB0mHlOcveiXV0G1d8yE3oEHkXeQhUJR9F0R0EEvbtoT3E Pl8MI22BiD8c1kEeS4NtY2vYymc4+ZgKZXx5CeZ10mfKuFAf5y6YtdnMNnP5xvft 4iLJg7xuJFEduOAutbfqE8nuV+6uY1jnHVLO8I4wbkm67JI0H86xXO2j58AEqIvE 5hX2vnhOlwhpVg4zzw5R1bEODBFpdQbCAMqasVO/0l0FCfbHw9REFaUF5LoXK3Df OuSXlzlzZRWS7KT3U1mgt65EnN5LxWElLeEI+sBm23wodoD9Up84LpU+5yjWPZ9q AYvFA5gj1nLF05HRsVMW =0oAt -----END PGP SIGNATURE----- --81JctsDUVPekGcy+-- From owner-freebsd-fs@freebsd.org Mon Jul 4 18:57:00 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8FE35B9123C for ; Mon, 4 Jul 2016 18:57:00 +0000 (UTC) (envelope-from jkh@ixsystems.com) Received: from barracuda.ixsystems.com (barracuda.ixsystems.com [12.229.62.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ixsystems.com", Issuer "Go Daddy Secure Certificate Authority - G2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7160326C1 for ; Mon, 4 Jul 2016 18:57:00 +0000 (UTC) (envelope-from jkh@ixsystems.com) X-ASG-Debug-ID: 1467658619-08ca0410fe52540001-3nHGF7 Received: from zimbra.ixsystems.com ([10.246.0.20]) by barracuda.ixsystems.com with ESMTP id jjIzBrmqsynMWHBv (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 04 Jul 2016 11:56:59 -0700 (PDT) X-Barracuda-Envelope-From: jkh@ixsystems.com X-Barracuda-RBL-Trusted-Forwarder: 10.246.0.20 X-ASG-Whitelist: Client Received: from localhost (localhost [127.0.0.1]) by zimbra.ixsystems.com (Postfix) with ESMTP id 3B324CA4281; Mon, 4 Jul 2016 11:56:59 -0700 (PDT) Received: from zimbra.ixsystems.com ([127.0.0.1]) by localhost (zimbra.ixsystems.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id IN90iGIvEJaT; Mon, 4 Jul 2016 11:56:58 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.ixsystems.com (Postfix) with ESMTP id 770F2CA142E; Mon, 4 Jul 2016 11:56:58 -0700 (PDT) X-Virus-Scanned: amavisd-new at ixsystems.com Received: from zimbra.ixsystems.com ([127.0.0.1]) by localhost (zimbra.ixsystems.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id wW2gZQa8p6rk; Mon, 4 Jul 2016 11:56:58 -0700 (PDT) Received: from [172.20.0.34] (vpn.ixsystems.com [10.249.0.2]) by zimbra.ixsystems.com (Postfix) with ESMTPSA id EEF81CA0CC0; Mon, 4 Jul 2016 11:56:57 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: HAST + ZFS + NFS + CARP From: Jordan Hubbard X-ASG-Orig-Subj: Re: HAST + ZFS + NFS + CARP In-Reply-To: <20160704183643.GI41276@mordor.lan> Date: Mon, 4 Jul 2016 11:56:57 -0700 Cc: Ben RUBSON , freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> <20160703192945.GE41276@mordor.lan> <20160703214723.GF41276@mordor.lan> <65906F84-CFFC-40E9-8236-56AFB6BE2DE1@ixsystems.com> <61283600-A41A-4A8A-92F9-7FAFF54DD175@ixsystems.com> <20160704183643.GI41276@mordor.lan> To: Julien Cigar X-Mailer: Apple Mail (2.3124) X-Barracuda-Connect: UNKNOWN[10.246.0.20] X-Barracuda-Start-Time: 1467658619 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://10.246.0.26:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at ixsystems.com X-Barracuda-BRTS-Status: 1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 18:57:00 -0000 > On Jul 4, 2016, at 11:36 AM, Julien Cigar = wrote: >=20 > I think the discussion evolved a bit since I started this thread, the > original purpose was to build a low-cost redundant storage for a small > infrastructure, no more no less. >=20 > The context is the following: I work in a small company, partially > financed by public funds, we started small, evolved a bit to a point > that some redundancy is required for $services.=20 > Unfortunately I'm alone to take care of the infrastructure (and it's=20= > only 50% of my time) and we don't have that much money :(=20 Sure, I get that part also, but let=E2=80=99s put the entire = conversation into context: 1. You=E2=80=99re looking for a solution to provide some redundant = storage in a very specific scenario. 2. We=E2=80=99re talking on a public mailing list with a bunch of folks, = so the conversation is also naturally going to go from the specific to = the general - e.g. =E2=80=9CIs there anything of broader applicability = to be learned / used here?=E2=80=9D I=E2=80=99m speaking more to the = larger audience who is probably wondering if there=E2=80=99s a more = general solution here using the same =E2=80=9Cmoving parts=E2=80=9D. To get specific again, I am not sure I would do what you are = contemplating given your circumstances since it=E2=80=99s not the = cheapest / simplest solution. The cheapest / simplest solution would be = to create 2 small ZFS servers and simply do zfs snapshot replication = between them at periodic intervals, so you have a backup copy of the = data for maximum safety as well as a physically separate server in case = one goes down hard. Disk storage is the cheap part now, particularly if = you have data redundancy and can therefore use inexpensive disks, and = ZFS replication is certainly =E2=80=9Cgood enough=E2=80=9D for disaster = recovery. As others have said, adding additional layers will only = increase the overall fragility of the solution, and =E2=80=9Cfragile=E2=80= =9D is kind of the last thing you need when you=E2=80=99re frantically = trying to deal with a server that has gone down for what could be any = number of reasons. I, for example, use a pair of FreeNAS Minis at home to store all my = media and they work fine at minimal cost. I use one as the primary = server that talks to all of the VMWare / Plex / iTunes server = applications (and serves as a backup device for all my iDevices) and it = replicates the entire pool to another secondary server that can be = pushed into service as the primary if the first one loses a power supply = / catches fire / loses more than 1 drive at a time / etc. Since I have = a backup, I can also just use RAIDZ1 for the 4x4Tb drive configuration = on the primary and get a good storage / redundancy ratio (I can lose a = single drive without data loss but am also not wasting a lot of storage = on parity). Just my two cents. There are a lot of different ways to do this, and = like all things involving computers (especially PCs), the simplest way = is usually the best. - Jordan From owner-freebsd-fs@freebsd.org Mon Jul 4 19:31:40 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D7C3AB91C58 for ; Mon, 4 Jul 2016 19:31:40 +0000 (UTC) (envelope-from julien@perdition.city) Received: from relay-b03.edpnet.be (relay-b03.edpnet.be [212.71.1.220]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "edpnet.email", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 847F629EF for ; Mon, 4 Jul 2016 19:31:40 +0000 (UTC) (envelope-from julien@perdition.city) X-ASG-Debug-ID: 1467660692-0a88181ce58d0490001-3nHGF7 Received: from mordor.lan (213.219.165.225.bro01.dyn.edpnet.net [213.219.165.225]) by relay-b03.edpnet.be with ESMTP id Y0nG9NvMVrKnJfqV (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 04 Jul 2016 21:31:33 +0200 (CEST) X-Barracuda-Envelope-From: julien@perdition.city X-Barracuda-Effective-Source-IP: 213.219.165.225.bro01.dyn.edpnet.net[213.219.165.225] X-Barracuda-Apparent-Source-IP: 213.219.165.225 Date: Mon, 4 Jul 2016 21:31:32 +0200 From: Julien Cigar To: Jordan Hubbard Cc: Ben RUBSON , freebsd-fs@freebsd.org Subject: Re: HAST + ZFS + NFS + CARP Message-ID: <20160704193131.GJ41276@mordor.lan> X-ASG-Orig-Subj: Re: HAST + ZFS + NFS + CARP References: <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> <20160703192945.GE41276@mordor.lan> <20160703214723.GF41276@mordor.lan> <65906F84-CFFC-40E9-8236-56AFB6BE2DE1@ixsystems.com> <61283600-A41A-4A8A-92F9-7FAFF54DD175@ixsystems.com> <20160704183643.GI41276@mordor.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="3MMMIZFJzhAsRj/+" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.6.1 (2016-04-27) X-Barracuda-Connect: 213.219.165.225.bro01.dyn.edpnet.net[213.219.165.225] X-Barracuda-Start-Time: 1467660692 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://212.71.1.220:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 3577 X-Virus-Scanned: by bsmtpd at edpnet.be X-Barracuda-BRTS-Status: 1 X-Barracuda-Bayes: INNOCENT GLOBAL 0.5000 1.0000 0.0000 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=6.0 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.31010 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 19:31:40 -0000 --3MMMIZFJzhAsRj/+ Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jul 04, 2016 at 11:56:57AM -0700, Jordan Hubbard wrote: >=20 > > On Jul 4, 2016, at 11:36 AM, Julien Cigar wrote: > >=20 > > I think the discussion evolved a bit since I started this thread, the > > original purpose was to build a low-cost redundant storage for a small > > infrastructure, no more no less. > >=20 > > The context is the following: I work in a small company, partially > > financed by public funds, we started small, evolved a bit to a point > > that some redundancy is required for $services.=20 > > Unfortunately I'm alone to take care of the infrastructure (and it's=20 > > only 50% of my time) and we don't have that much money :(=20 >=20 > Sure, I get that part also, but let=E2=80=99s put the entire conversation= into context: >=20 > 1. You=E2=80=99re looking for a solution to provide some redundant storag= e in a very specific scenario. >=20 > 2. We=E2=80=99re talking on a public mailing list with a bunch of folks, = so the conversation is also naturally going to go from the specific to the = general - e.g. =E2=80=9CIs there anything of broader applicability to be le= arned / used here?=E2=80=9D I=E2=80=99m speaking more to the larger audien= ce who is probably wondering if there=E2=80=99s a more general solution her= e using the same =E2=80=9Cmoving parts=E2=80=9D. of course..! It has been an interesting discussion, learned some things, and it's always enjoyable to get different point of view. >=20 > To get specific again, I am not sure I would do what you are contemplatin= g given your circumstances since it=E2=80=99s not the cheapest / simplest s= olution. The cheapest / simplest solution would be to create 2 small ZFS s= ervers and simply do zfs snapshot replication between them at periodic inte= rvals, so you have a backup copy of the data for maximum safety as well as = a physically separate server in case one goes down hard. Disk storage is t= he cheap part now, particularly if you have data redundancy and can therefo= re use inexpensive disks, and ZFS replication is certainly =E2=80=9Cgood en= ough=E2=80=9D for disaster recovery. As others have said, adding additiona= l layers will only increase the overall fragility of the solution, and =E2= =80=9Cfragile=E2=80=9D is kind of the last thing you need when you=E2=80=99= re frantically trying to deal with a server that has gone down for what cou= ld be any number of reasons. >=20 > I, for example, use a pair of FreeNAS Minis at home to store all my media= and they work fine at minimal cost. I use one as the primary server that = talks to all of the VMWare / Plex / iTunes server applications (and serves = as a backup device for all my iDevices) and it replicates the entire pool t= o another secondary server that can be pushed into service as the primary i= f the first one loses a power supply / catches fire / loses more than 1 dri= ve at a time / etc. Since I have a backup, I can also just use RAIDZ1 for = the 4x4Tb drive configuration on the primary and get a good storage / redun= dancy ratio (I can lose a single drive without data loss but am also not wa= sting a lot of storage on parity). You're right, I'll definitively reconsider the zfs send / zfs receive approach. >=20 > Just my two cents. There are a lot of different ways to do this, and lik= e all things involving computers (especially PCs), the simplest way is usua= lly the best. >=20 Thanks! Julien > - Jordan >=20 --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --3MMMIZFJzhAsRj/+ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCgAGBQJXermQAAoJELK7NxCiBCPAxloP/0EHtudE2MmOGNtgttmTGNGN KLLGYcaoDZI4sn2t/q+48oBNfTDPdkHnsXDhGgVRXcQ4yXTuBg9IzcqezwNB+lGb FaC5ckFLqBlXzXUo2Y4lia6T45MtH4QrZn9GPH6O4QfvIQp0ulmCEtVVIUKw/fuA tanEGTTwc6qMhdzd1Uopml2HZJ74SzGmavFf/N4eP3Gnzz/p9a6SQjCH7TAtvxH7 ck6AM3teS79eGdb2BmU6Ehs9A10LZefnleRfMLi2V7RDNNXP2oI4ohI00dTRz1Cf GrcmPB4oRlGEonbMBytYZVOTdrwCGsFyrXTnT7xy4XHCg/9ndUZDcnnDTD8syEBL CFz8l/uZB++l0lAs8a+RyarTNRSMvTznrS039IaOHhD23M/zGXLGVe8EwQlpkwom I/3UUQeQI1291Xnq12PPJUKE2SK8gZ9eJWswO97eXJow7ky1L2bIc3HFUkpkDu9y pPbrMtzslTdfCb4w2SHlE1yJn0/Mo/FyKMuPKbHP5uBDvVoc5PpH912Tcg544Ose ChriAsDJ1Fy23pg52wj/W5zXCzfMjTKLmokNwV3xH4c8wbpFo7jm7IR9lA8BoFME dNSSnLPF3mPFAc4TUM0hS980cPAYE5ovoKMwsOWPR0YEhwtFGZp5xUdpMdkrpnAA SbW49On9C/KPLhfK+NFp =TMxe -----END PGP SIGNATURE----- --3MMMIZFJzhAsRj/+-- From owner-freebsd-fs@freebsd.org Tue Jul 5 10:37:37 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C19F1B21CEE for ; Tue, 5 Jul 2016 10:37:37 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x229.google.com (mail-wm0-x229.google.com [IPv6:2a00:1450:400c:c09::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B59A1CB3 for ; Tue, 5 Jul 2016 10:37:37 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x229.google.com with SMTP id z126so70008416wme.0 for ; Tue, 05 Jul 2016 03:37:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=LhEKewYLOfRMLOcpuOxDwuj3eGAsP3Ej/sLMOgieULQ=; b=LdKNcTPh7oaol0Wm4uAhu/zxdFDfos5O7Cyytc9x1sq/I+5ko6fSW2dJUuOidWXrSp fE8dV5sPBHgFXTSE3phn1bnTBTn1OAwXczdfyRsqpFqGs9Sh0n+RXna2hwMnszeqC55C HZagSnny2lScp503YvaHelLdpPIPuQEymNDDgtdryw1HPMrUOZZXAmFEGTfq8sU0jCb6 77/sWm3u6/90Gr525z+Whfi5k/ZkdKrpYPenlD19z5IGKtqnViZrypOOA9+qK6SJpmqQ E7IOIgumpNcjnbI+NX1IbVcLJyh+JzcER5AnduAVZY/J7l1bKr8ciwLQ/1gENCzLxOy4 5+Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=LhEKewYLOfRMLOcpuOxDwuj3eGAsP3Ej/sLMOgieULQ=; b=mvhl6WyizFy1ShQAQ7ahyRKCxCa87SUfAe7LN/IOWlQYBuhipAGg5Q7s4bXzkwbnPu K6X6JYWfoLmUTAUZXD+hS6xkmqNfMhxz8Q2SP69md0l6hmj2MV2J8+R4EIOmIGr38kaY vObndA2ronqnDV0wH61Oca+fKgWfR8I8jYNVc5aNOCFaAGNbg/0z/gL9/PTN3ya+sDMM MYz8cstZnQQ+MY0Kbwvz1llHO081NmjqZhxaJLoc9h5VCkEhZjdHqJMbdB4fUajBFZCK +9sAvBCCfXdhKEfnm6mGGDEK5aCugqOjxPZvz9tZ4r/kotaaQ+AwPjc4sb8k84u6kQw8 36Vw== X-Gm-Message-State: ALyK8tLveFJ6+Uym45N41uEiZ0trBmRJxYGrceb4yV4rlx62WUkbYhonaC5cOnAmpUvb1w== X-Received: by 10.194.112.161 with SMTP id ir1mr14676570wjb.118.1467715055351; Tue, 05 Jul 2016 03:37:35 -0700 (PDT) Received: from [192.168.1.16] (210.236.26.109.rev.sfr.net. [109.26.236.210]) by smtp.gmail.com with ESMTPSA id bh7sm1930508wjb.22.2016.07.05.03.37.34 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 05 Jul 2016 03:37:34 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: HAST + ZFS + NFS + CARP From: Ben RUBSON In-Reply-To: <73DC3B93-60FF-4B20-AF80-5712F77818EF@ixsystems.com> Date: Tue, 5 Jul 2016 12:37:33 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <73DC3B93-60FF-4B20-AF80-5712F77818EF@ixsystems.com> To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2016 10:37:37 -0000 > On 04 Jul 2016, at 20:06, Jordan Hubbard wrote: >=20 >=20 >> On Jul 1, 2016, at 11:23 AM, Ben RUBSON wrote: >>=20 >> Would you say that giving an iSCSI disk to ZFS hides some details of = the raw disk to ZFS ? >=20 > Yes, of course. >=20 >> I though that iSCSI would have been a totally "transparent" layer, = transferring all ZFS requests to the raw disk, giving back the answers, = hiding anything. >=20 > Not really, no. There are other ways of talking to a disk or SSD = device, such as getting S.M.A.R.T. data to see when/if a drive is = failing. Yep but SMART is not used by ZFS itself, according to a dev on #openzfs. There is however a feature request here : = https://github.com/zfsonlinux/zfs/issues/2777 I don't know whether FreeBSD iSCSI target implementation is SMART = pass-thru or not (I don't think so, my tests some months ago did not = work). However SMART data of iSCSI disks can easily be checked on the target = server itself (so not on the initiator, I agree) using smartmontools. > Drives also return checksum errors that may be masked by the iSCSI = target. Should be caught by smartmontools running on target, right ? > Finally, there is SCSI-2 and there is SCSI-3 (where things like = persistent reservations are implemented). None of these things are = necessarily implemented by iSCSI. At least FreeBSD implements persistent reservations : https://www.freebsd.org/cgi/man.cgi?query=3Dctl However I'm not sure ZFS itself uses such feature, which are more = relevant for clusters. Does it ? Ben From owner-freebsd-fs@freebsd.org Tue Jul 5 12:21:34 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1FE82B7151B for ; Tue, 5 Jul 2016 12:21:34 +0000 (UTC) (envelope-from michelle@sorbs.net) Received: from hades.sorbs.net (mail.sorbs.net [67.231.146.200]) by mx1.freebsd.org (Postfix) with ESMTP id 140AB1640 for ; Tue, 5 Jul 2016 12:21:33 +0000 (UTC) (envelope-from michelle@sorbs.net) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from isux.com (firewall.isux.com [213.165.190.213]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0O9U00A81DC4HF00@hades.sorbs.net> for freebsd-fs@freebsd.org; Tue, 05 Jul 2016 05:28:54 -0700 (PDT) To: "freebsd-fs@freebsd.org" From: Michelle Sullivan Subject: resilver slowdown.... Message-id: <577BA643.6040206@sorbs.net> Date: Tue, 05 Jul 2016 14:21:23 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:43.0) Gecko/20100101 Firefox/43.0 SeaMonkey/2.40 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2016 12:21:34 -0000 After retuning for priority to the resilver, I see massive (very good) performance.... then at some random time later (or so it seemed) the performance would drop off very quickly always at least to half the performance, sometimes as much as a quarter of the speed... At first I though it was memory based, then observing system with 19G (of 32G) free figured it must be based on the stuff on the disk got some 16T of very small files, and another 16T of video files so figured it was probably going from large files to small files etc... Nope, after a reboot and resetting the tuning params to the same each time after the reboot I'd be seeing an immediate and sustained performance... Watching it more it seems this is directly related: [root@colossus ~]# sysctl -a | grep \\.arc_ vfs.zfs.arc_meta_limit: 4294967296 vfs.zfs.arc_meta_used: 5529107352 vfs.zfs.arc_min: 8589934592 vfs.zfs.arc_max: 17179869184 Specifically when the vfs.zfs.arc_meta_used reaches and then exceeds vfs.zfs.arc_meta_limit.. This is the average output of 'gstat -I 5s' when the limit is exceeded: dT: 5.001s w: 5.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 0 49 49 2016 192.2 0 0 0.0 82.7| mfid0 0 50 50 1981 206.5 0 0 0.0 82.4| mfid1 0 49 49 1988 194.4 0 0 0.0 80.7| mfid2 0 51 51 1984 218.5 0 0 0.0 83.8| mfid3 0 50 50 1993 215.5 0 0 0.0 82.5| mfid4 0 51 51 2067 204.5 0 0 0.0 83.1| mfid5 0 54 54 1983 181.0 0 0 0.0 77.2| mfid6 0 49 49 1982 207.3 0 0 0.0 83.2| mfid7 0 49 49 1983 204.0 0 0 0.0 76.4| mfid8 0 51 51 2130 213.5 0 0 0.0 84.0| mfid9 0 51 51 2040 213.2 0 0 0.0 84.6| mfid10 74 142 0 0 0.0 142 2362 988.5 100.2| mfid11 0 52 52 1988 145.1 0 0 0.0 79.3| mfid12 0 50 50 1978 166.8 0 0 0.0 90.1| mfid13 0 101 0 0 0.0 101 2104 173.6 91.6| mfid14 0 0 0 0 0.0 0 0 3.2 0.1| ada0 0 0 0 0 0.0 0 0 3.1 0.1| ada1 0 0 0 0 0.0 0 0 0.0 0.0| raid/r0 0 0 0 0 0.0 0 0 0.0 0.0| raid/r0p1 0 0 0 0 0.0 0 0 0.0 0.0| raid/r0p2 0 0 0 0 0.0 0 0 0.0 0.0| raid/r0p3 0 0 0 0 0.0 0 0 0.0 0.0| label/Boot 0 0 0 0 0.0 0 0 0.0 0.0| gptid/a9e5606c-c587-11e3-9aac-ac220bcb8f6c 0 0 0 0 0.0 0 0 0.0 0.0| label/Swap 0 0 0 0 0.0 0 0 0.0 0.0| label/RootFS Before the limit is reached... (ie immediately after a reboot) and the following tuning is done: [root@colossus ~]# sysctl vfs.zfs.resilver_min_time_ms=5000; sysctl vfs.zfs.resilver_delay=0 ; sysctl vfs.zfs.vdev.max_pending=10000; sysctl vfs.zfs.top_maxinflight=3200 ; sysctl vfs.zfs.scan_idle=1 vfs.zfs.resilver_min_time_ms: 3000 -> 5000 vfs.zfs.resilver_delay: 2 -> 0 vfs.zfs.vdev.max_pending: 10 -> 10000 vfs.zfs.top_maxinflight: 32 -> 3200 vfs.zfs.scan_idle: 50 -> 1 The following is the average output of the same gstat: dT: 5.001s w: 5.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 0 115 115 6396 304.1 0 0 0.0 100.5| mfid0 0 117 117 6532 298.3 0 0 0.0 100.6| mfid1 0 112 112 6379 235.1 0 0 0.0 104.9| mfid2 0 114 114 6349 302.4 0 0 0.0 97.7| mfid3 0 117 117 6529 300.0 0 0 0.0 98.1| mfid4 25 112 112 6070 266.4 0 0 0.0 101.5| mfid5 39 106 106 5773 245.7 0 0 0.0 100.7| mfid6 25 110 110 6037 266.1 0 0 0.0 103.2| mfid7 35 111 111 6001 250.6 0 0 0.0 104.2| mfid8 35 110 110 5925 255.6 0 0 0.0 99.3| mfid9 11 116 116 6270 259.2 0 0 0.0 105.3| mfid10 207 374 0 0 0.0 374 4694 576.2 103.5| mfid11 8 115 115 6287 255.3 0 0 0.0 103.0| mfid12 1 113 113 6359 202.3 0 0 0.0 107.6| mfid13 95 268 0 0 0.0 268 4972 207.5 105.6| mfid14 0 0 0 0 0.0 0 0 3.4 0.1| ada0 0 0 0 0 0.0 0 0 3.5 0.1| ada1 0 0 0 0 0.0 0 0 0.0 0.0| raid/r0 0 0 0 0 0.0 0 0 0.0 0.0| raid/r0p1 0 0 0 0 0.0 0 0 0.0 0.0| raid/r0p2 0 0 0 0 0.0 0 0 0.0 0.0| raid/r0p3 0 0 0 0 0.0 0 0 0.0 0.0| label/Boot 0 0 0 0 0.0 0 0 0.0 0.0| gptid/a9e5606c-c587-11e3-9aac-ac220bcb8f6c 0 0 0 0 0.0 0 0 0.0 0.0| label/Swap 0 0 0 0 0.0 0 0 0.0 0.0| label/RootFS Is this a bug? 9.3-p33 on LSI 9260-16i-512mb set in 15 3Tb SATA3 RAID 0 arrays, no read-ahead, write-back, cached. Regards, -- Michelle Sullivan http://www.mhix.org/ From owner-freebsd-fs@freebsd.org Tue Jul 5 19:07:27 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E82A0B20E58 for ; Tue, 5 Jul 2016 19:07:27 +0000 (UTC) (envelope-from michelle@sorbs.net) Received: from hades.sorbs.net (mail.sorbs.net [67.231.146.200]) by mx1.freebsd.org (Postfix) with ESMTP id DC3CB1A33 for ; Tue, 5 Jul 2016 19:07:27 +0000 (UTC) (envelope-from michelle@sorbs.net) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from isux.com (firewall.isux.com [213.165.190.213]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0O9U00AALW4UHF00@hades.sorbs.net> for freebsd-fs@freebsd.org; Tue, 05 Jul 2016 12:14:56 -0700 (PDT) Subject: Re: resilver slowdown.... To: "freebsd-fs@freebsd.org" References: <577BA643.6040206@sorbs.net> From: Michelle Sullivan Message-id: <577C056C.7070108@sorbs.net> Date: Tue, 05 Jul 2016 21:07:24 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:43.0) Gecko/20100101 Firefox/43.0 SeaMonkey/2.40 In-reply-to: <577BA643.6040206@sorbs.net> X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2016 19:07:28 -0000 Michelle Sullivan wrote: > After retuning for priority to the resilver, I see massive (very good) > performance.... then at some random time later (or so it seemed) the > performance would drop off very quickly always at least to half the > performance, sometimes as much as a quarter of the speed... > > > At first I though it was memory based, then observing system with 19G > (of 32G) free figured it must be based on the stuff on the disk got > some 16T of very small files, and another 16T of video files so > figured it was probably going from large files to small files etc... > Nope, after a reboot and resetting the tuning params to the same each > time after the reboot I'd be seeing an immediate and sustained > performance... Watching it more it seems this is directly related: > > [root@colossus ~]# sysctl -a | grep \\.arc_ > vfs.zfs.arc_meta_limit: 4294967296 > vfs.zfs.arc_meta_used: 5529107352 > vfs.zfs.arc_min: 8589934592 > vfs.zfs.arc_max: 17179869184 Yup think it is related to the meta_limit and meta_used... Resilvering at 203M/s (according to zpool status) [root@colossus ~]# sysctl -a | grep \\.arc_ vfs.zfs.arc_meta_limit: 4294967296 vfs.zfs.arc_meta_used: 2187549528 vfs.zfs.arc_min: 8589934592 vfs.zfs.arc_max: 17179869184 Noticed it had slowed again.. around 64M/s ... [root@colossus ~]# sysctl -a | grep \\.arc_ vfs.zfs.arc_meta_limit: 4294967296 vfs.zfs.arc_meta_used: 4295111952 vfs.zfs.arc_min: 8589934592 vfs.zfs.arc_max: 17179869184 Rebooted and back to >120M/s (can't get a reading atm - too soon after the reboot) -- Michelle Sullivan http://www.mhix.org/ From owner-freebsd-fs@freebsd.org Wed Jul 6 15:01:23 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26765B75A0B for ; Wed, 6 Jul 2016 15:01:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 15F801F88 for ; Wed, 6 Jul 2016 15:01:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u66F1JhT053425 for ; Wed, 6 Jul 2016 15:01:22 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 194513] zfs recv hangs in state kmem arena Date: Wed, 06 Jul 2016 15:01:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: shared+bugs.freebsd.org@twingly.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2016 15:01:23 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194513 --- Comment #19 from Twingly --- Some additional information: I left 4 hanged zfs receive overnight and in t= he morning they had completed. My shell reported two of them running for 10h 3= 5m 26s and 10h 38m 50s. Note that this was for newly created datasets without = any data on them, so shouldn't they have completed instantly? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Jul 8 13:44:00 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7EC0DB84336 for ; Fri, 8 Jul 2016 13:44:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6DCE41D6E for ; Fri, 8 Jul 2016 13:44:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u68DhwxX028478 for ; Fri, 8 Jul 2016 13:44:00 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 209158] node / npm triggering zfs rename deadlock Date: Fri, 08 Jul 2016 13:43:58 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: needs-qa, patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: koobs@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? mfc-stable11? X-Bugzilla-Changed-Fields: keywords bug_file_loc flagtypes.name Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 13:44:00 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209158 Kubilay Kocak changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |needs-qa, patch URL| |https://reviews.freebsd.org | |/D6533 Flags| |mfc-stable10?, | |mfc-stable11? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Jul 8 14:03:50 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4F2A0B8494A for ; Fri, 8 Jul 2016 14:03:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3505219BC for ; Fri, 8 Jul 2016 14:03:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u68E3mkk007163 for ; Fri, 8 Jul 2016 14:03:50 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203864] ZFS deadlock between zfs send, zfs rename and ctrl-C Date: Fri, 08 Jul 2016 14:03:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: dogfood, needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 14:03:50 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203864 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mm@FreeBSD.org, | |pjd@FreeBSD.org Status|New |Open --- Comment #13 from Andriy Gapon --- I think that I've been able to reproduce this problem or, at least, somethi= ng that looks very much like it. I did the standard procstat debugging and I noticed something that did not appear in any of the previous reports: 6 100572 zfskern txg_thread_enter mi_switch+0x167 sleepq_switch+0xe7 sleepq_wait+0x43 _sx_xlock_hard+0x49d _sx_xlock+0xc5 zvol_rename_minors+0x104 dsl_dataset_rename_snapshot_sync_impl+0x308 dsl_dataset_rename_snapshot_sync+0xc1 dsl_sync_task_sync+0xef dsl_pool_sync+0x45b spa_sync+0x7c7 txg_sync_thread+0x383 fork_exit+0x84 fork_trampoline+0xe 1226 100746 zfs - mi_switch+0x167 sleepq_switch+0xe7 sleepq_wait+0x43 _cv_wait+0x1e4 txg_wait_synced+0x13b dsl_sync_task+0x205 dsl_dataset_user_release_impl+0x1cf dsl_dataset_user_release_onexit+0x86 zfs_onexit_destroy+0x56 zfsdev_close+0= x88 devfs_destroy_cdevpriv+0x8b devfs_close_f+0x65 _fdrop+0x1a closef+0x200 closefp+0xa3 amd64_syscall+0x2db Xfast_syscall+0xfb 1228 100579 zfs - mi_switch+0x167 sleepq_switch+0xe7 sleepq_wait+0x43 _cv_wait+0x1e4 txg_wait_synced+0x13b dsl_sync_task+0x205 dsl_dataset_rename_snapshot+0x3a zfs_ioc_rename+0x157 zfsdev_ioctl+0x635 devfs_ioctl_f+0x156 kern_ioctl+0x246 sys_ioctl+0x171 amd64_syscall+0x2db Xfast_syscall+0xfb Thread 100746 is it. zfsdev_close() holds spa_namespace_lock and then calls dsl_sync_task() -> txg_wait_synced(). On the other hand the sync thread (100572) gets stuck on spa_namespace_lock in a call to zvol_rename_minors(). My opinion is that the sync thread must never try to take spa_namespace_loc= k.=20 The problem seems to be introduced quite a while ago in base r219317. Some later commits like base r272474 also followed the same pattern. The proble= m is certainly FreeBSD-specific as illumos handles ZVOL names in a very different manner. Also, the problem is rather deep-rooted and at the moment I do not see any = easy way to fix without breaking ZVOL name tracking. P.S. A bit of information from ddb: db> p spa_namespace_lock ffffffff822b1ee0 db> show lock 0xffffffff822b1ee0 class: sx name: spa_namespace_lock state: XLOCK: 0xfffff8001da60500 (tid 100746, pid 1226, "zfs") waiters: exclusive db> thread 100746 [ thread pid 1226 tid 100746 ] sched_switch+0x48a: movl %gs:0x34,%eax db> bt Tracing pid 1226 tid 100746 td 0xfffff8001da60500 sched_switch() at sched_switch+0x48a/frame 0xfffffe004def4590 mi_switch() at mi_switch+0x167/frame 0xfffffe004def45c0 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe004def4600 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe004def4630 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe004def4690 txg_wait_synced() at txg_wait_synced+0x13b/frame 0xfffffe004def46d0 dsl_sync_task() at dsl_sync_task+0x205/frame 0xfffffe004def4790 dsl_dataset_user_release_impl() at dsl_dataset_user_release_impl+0x1cf/frame 0xfffffe004def4910 dsl_dataset_user_release_onexit() at dsl_dataset_user_release_onexit+0x86/f= rame 0xfffffe004def4940 zfs_onexit_destroy() at zfs_onexit_destroy+0x56/frame 0xfffffe004def4970 zfsdev_close() at zfsdev_close+0x88/frame 0xfffffe004def4990 devfs_destroy_cdevpriv() at devfs_destroy_cdevpriv+0x8b/frame 0xfffffe004def49b0 devfs_close_f() at devfs_close_f+0x65/frame 0xfffffe004def49e0 _fdrop() at _fdrop+0x1a/frame 0xfffffe004def4a00 closef() at closef+0x200/frame 0xfffffe004def4a90 closefp() at closefp+0xa3/frame 0xfffffe004def4ae0 amd64_syscall() at amd64_syscall+0x2db/frame 0xfffffe004def4bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004def4bf0 --- syscall (6, FreeBSD ELF64, sys_close), rip =3D 0x8013f996a, rsp =3D 0x7fffffffd438, rbp =3D 0x7fffffffd450 --- --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Jul 8 14:32:32 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EA4B1B84EA6 for ; Fri, 8 Jul 2016 14:32:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D9F6618B8 for ; Fri, 8 Jul 2016 14:32:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u68EWWBv065948 for ; Fri, 8 Jul 2016 14:32:32 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203864] ZFS deadlock between zfs send, zfs rename and ctrl-C Date: Fri, 08 Jul 2016 14:32:32 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: dogfood, needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: smh@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 14:32:33 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203864 --- Comment #14 from Steven Hartland --- You're conclusion seems valid there Andriy but as you say there doesn't see= m to be an easy way to fix it. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Jul 9 05:58:44 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27EBCB84F59 for ; Sat, 9 Jul 2016 05:58:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0612317C1 for ; Sat, 9 Jul 2016 05:58:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u695wghS056523 for ; Sat, 9 Jul 2016 05:58:43 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203864] ZFS deadlock between zfs send, zfs rename and ctrl-C Date: Sat, 09 Jul 2016 05:58:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: dogfood, needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Jul 2016 05:58:44 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203864 --- Comment #15 from Andriy Gapon --- (In reply to Steven Hartland from comment #14) One possible approach is to move all the "minors" manipulations to ZFS ioctl handlers, the only place where they were called originally, out of the sync thread. This sounds rather straightforward, but could turn out to be a bit laboriou= s. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Jul 9 06:02:45 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95A4DB85171 for ; Sat, 9 Jul 2016 06:02:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 858E21AE7 for ; Sat, 9 Jul 2016 06:02:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u6962iC9003577 for ; Sat, 9 Jul 2016 06:02:45 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203864] ZFS deadlock between zfs send, zfs rename and ctrl-C Date: Sat, 09 Jul 2016 06:02:44 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: dogfood, needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Jul 2016 06:02:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203864 --- Comment #16 from Andriy Gapon --- (In reply to Steven Hartland from comment #14) Another possibility is to apply all zvol changes asynchronously. Either by queuing them or by having a dedicated thread scanning for them. The former should be almost trivial to implement. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Jul 9 11:08:53 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C565B83A0B for ; Sat, 9 Jul 2016 11:08:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6C299198B for ; Sat, 9 Jul 2016 11:08:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u69B8q80037411 for ; Sat, 9 Jul 2016 11:08:53 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203864] ZFS deadlock between zfs send, zfs rename and ctrl-C Date: Sat, 09 Jul 2016 11:08:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: dogfood, needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Jul 2016 11:08:53 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203864 --- Comment #17 from Andriy Gapon --- Another observation is that the current code seems to be broken in a couple= of places as well. 1. dsl_dataset_rename_snapshot_sync_impl() uses ddrsa_fsname to construct t= he old and new snapshot names, but this would be wrong in the case of a recurs= ive snpashot rename (zfs rename -r x@snap1 x@snap2) as ddrsa_fsname always poin= ts to the name of the top level filesystem. 2. dsl_dataset_promote_sync(): as far as I can see, oldname is never popula= ted, so the code can do arbitrary renames. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Jul 9 21:01:45 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1ED5B76D8A for ; Sat, 9 Jul 2016 21:01:45 +0000 (UTC) (envelope-from mailinglists35@gmail.com) Received: from mail-oi0-x22b.google.com (mail-oi0-x22b.google.com [IPv6:2607:f8b0:4003:c06::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AC3A71ED7 for ; Sat, 9 Jul 2016 21:01:45 +0000 (UTC) (envelope-from mailinglists35@gmail.com) Received: by mail-oi0-x22b.google.com with SMTP id s66so102407268oif.1 for ; Sat, 09 Jul 2016 14:01:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=sqmMRUxiWhtyOCprksQqQOvn5q3cyCsmxP7HnrNseEg=; b=HoK1DJuxrqhzhlXQgMgQySGOZFOW5XXUGcvRXAgTtr2Q2x4eZIQS4Kszksa7b1FhGH XRnZOMi1xfF20dHDVBaK+uSZ5crk8FD12el8cChDetW24c40ilcxs7UpmOE2KJzWLyiy cKs56G5Ex7g0fAVrmjSGG0di3toOSKUz1w6k4PRSF2xu+kvsmVNYBEPBUC9DxTRvkrL+ KfeD7MOwmJtvOWOinwTRSldFo7lYFYBCnk5K01kDUFL0Lr88Ja1fKX8hhmOHVkhkfY7Q 8YwU3qgYkk0Da7So8Dmhq+Bd7Mf809hMtQVUonFhjaopJFK1RGNsWrey8gC9sQk20RDv E9pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=sqmMRUxiWhtyOCprksQqQOvn5q3cyCsmxP7HnrNseEg=; b=QhQ1e31X2rhDH/pbqDRZ/DBPkTu8BAVEd55ZNT4sDpS0R1usds4OkH5xlNCAxNu68+ zJQ/tE72jq50Y+wvOZY/5ioAydixKQ1V2AoRyQtgCK6e+xpfVYHeDKsKTjxeBG95Whwy A8DgDeN5xv9A/Ij6KP9mdWvS3O/m7Wc27/9dVAaKV9jqrgqaxfAG4Zbd7twoNOrngN8r 39SMWoTaV4yO/iht+GJ/wjanAuekLDsHCViqpTnfD5etHVg7Txuy6FHA/k8WmcwfUp+3 1MGMsXvmyxJ51B9c6z+4pGar/NYkB8/GaH9TAvIhgbwR+TqWcQ7O6fRzHWdwNQIvAqAe JtOA== X-Gm-Message-State: ALyK8tKCtvD/4Pdo5AnMvqx+lJcEltr/BNgP42VPdn074x8zy+RtfafvmZ0M1jhX8wzDdeXnPnYR9ALq0j4JNw== X-Received: by 10.202.82.23 with SMTP id g23mr4819376oib.128.1468098104534; Sat, 09 Jul 2016 14:01:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.202.183.196 with HTTP; Sat, 9 Jul 2016 14:01:25 -0700 (PDT) From: Mailing Lists Date: Sun, 10 Jul 2016 00:01:25 +0300 Message-ID: Subject: help request for testing zfs issue To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Jul 2016 21:01:46 -0000 Hi, I'm trying to learn if a zfs issue can be reproduced on freebsd. I myself have reproduced it on 10.3 and 11 beta inside a linux KVM vm, and also on linux, but don't have a separate physical fbsd installation at hand. I'd really be grateful to someone who can take some of their time to verify this. Be aware that I expect the test machine to be rebooted if the test is reproductible (or at least their zfs pools to become unavailable) so this is not for people who have their rootfs on zfs unless they want to take the risk of getting to a hang. FWIW, the test machines have less than 4Gb of ram (the test vm was tested with 1 and 3Gb) so maybe you want to limit the available ram at boot time if you have more. These are the steps that hang my pool under linux and under the fbsd vm: mount -t tmpfs -o size=2662400k tmpfs /mnt/test/ cd /mnt/test for l in `seq 0 9`; do dd bs=1M count=256 if=/dev/zero of=loop$l; done zpool create -o failmode=continue -O logbias=throughput -O atime=off -o autoreplace=on -O primarycache=metadata -O secondarycache=metadata -O compression=lz4 -m /srv/big big raidz3 /mnt/test/loop* dd if=/dev/zero of=/srv/big/file bs=4M #interrupt with ctrl-c after some Gb written ls -lh /srv/big/ du -csh /srv/big/ rm /srv/big/file zfs set compression=off big while true; do dd if=/dev/zero of=/srv/big/file bs=4M #on freebsd11 dd never complains about no free space left and just hangs and had to add count=421 rm -f /srv/big/file done Some time after many iterations (as many as 370 on an ubuntu linux machine), the pool should list write errors on it but no errors on vdev or each disk. On linux the kernel prints a message but on freebsd there is no message and you have to manually check for pool status. Please cc me on replies or directly comment on the ZoL issue tracker here https://github.com/zfsonlinux/zfs/issues/4832 Thank you in advance