Date: Wed, 10 Aug 2016 15:10:40 +0200 From: Julien Cigar <julien@perdition.city> To: Ben RUBSON <ben.rubson@gmail.com> Cc: freebsd-fs@freebsd.org Subject: Re: HAST + ZFS + NFS + CARP Message-ID: <20160810131040.GH70364@mordor.lan> In-Reply-To: <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <AD42D8FD-D07B-454E-B79D-028C1EC57381@gmail.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] On Sat, Jul 02, 2016 at 05:04:22PM +0200, Ben RUBSON wrote: > > > On 30 Jun 2016, at 20:57, Julien Cigar <julien@perdition.city> wrote: > > > > On Thu, Jun 30, 2016 at 11:32:17AM -0500, Chris Watson wrote: > >> > >> > >> Sent from my iPhone 5 > >> > >>> > >>>> > >>>> Yes that's another option, so a zpool with two mirrors (local + > >>>> exported iSCSI) ? > >>> > >>> Yes, you would then have a real time replication solution (as HAST), compared to ZFS send/receive which is not. > >>> Depends on what you need :) > >>> > >>>> > >>>>> ZFS would then know as soon as a disk is failing. > >> > >> So as an aside, but related, for those watching this from the peanut gallery and for the benefit of the OP perhaps those that run with this setup might give some best practices and tips here in this thread on making this a good reliable setup. I can see someone reading this thread and tossing two crappy Ethernet cards in a box and then complaining it doesn't work well. > > > > It would be more than welcome indeed..! I have the feeling that HAST > > isn't that much used (but maybe I am wrong) and it's difficult to find > > informations on it's reliability and concrete long-term use cases... > > > > Also the pros vs cons of HAST vs iSCSI > > I made further testing today. > > # serverA, serverB : > kern.iscsi.ping_timeout=5 > kern.iscsi.iscsid_timeout=5 > kern.iscsi.login_timeout=5 > kern.iscsi.fail_on_disconnection=1 > > # Preparation : > - serverB : let's make 2 iSCSI targets : rem3, rem4. > - serverB : let's start ctld. > - serverA : let's create a mirror pool made of 4 disks : loc1, loc2, rem3, rem4. > - serverA : pool is healthy. > > # Test 1 : > - serverA : put a lot of data into the pool ; > - serverB : stop ctld ; > - serverA : put a lot of data into the pool ; > - serverB : start ctld ; > - serverA : make all pool disks online : it works, pool is healthy. > > # Test 2 : > - serverA : put a lot of data into the pool ; > - serverA : export the pool ; > - serverB : import the pool : it does not work, as ctld locks the disks ! Good news, nice protection (both servers won't be able to access the same disks at the same time). > - serverB : stop ctld ; > - serverB : import the pool : it works, 2 disks missing ; > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > - serverB : make all pool disks online : it works, pool is healthy. > > # Test 3 : > - serverA : put a lot of data into the pool ; > - serverB : stop ctld ; > - serverA : put a lot of data into the pool ; > - serverB : import the pool : it works, 2 disks missing ; > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > - serverB : make all pool disks online : it works, pool is healthy, but of course data written at step3 is lost. > > # Test 4 : > - serverA : put a lot of data into the pool ; > - serverB : stop ctld ; > - serverA : put a lot of data into the pool ; > - serverA : export the pool ; > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > - serverB : import the pool : it works, pool is healthy, data written at step3 is here. > > # Test 5 : > - serverA : rsync a huge remote repo into the pool in the background ; > - serverB : stop ctld ; > - serverA : 2 disks missing, but rsync still runs flawlessly ; > - serverB : start ctld ; > - serverA : make all pool disks online : it works, pool is healthy. > - serverB : ifconfig <replication_interface> down ; > - serverA : 2 disks missing, but rsync still runs flawlessly ; > - serverB : ifconfig <replication_interface> up ; > - serverA : make all pool disks online : it works, pool is healthy. > - serverB : power reset ! > - serverA : 2 disks missing, but rsync still runs flawlessly ; > - serverB : let's wait for server to be up ; > - serverA : make all pool disks online : it works, pool is healthy. > > Quite happy with these tests actually :) Hello, So, after testing ZFS replication with zrep (which works more or less perfectly) I'm busy to experiment a ZFS + iSCSI solution with two small HP DL20 and 2 disks in each. Machines are partitionned the same (https://gist.github.com/silenius/d3fdcd52ab35957f37527af892615ca7) with a zfs root (https://gist.github.com/silenius/f347e90ab187495cdea6e3baf64b881b) On filer2.prod.lan I have exported the two dedicated partitions (/dev/da0p4 and /dev/da1p4) as an iSCSI target (https://gist.github.com/silenius/8efda8334cb16cd779efff027ff5f3bd) which are available on filer1.prod.lan as /dev/da3 and /dev/da4 (https://gist.github.com/silenius/f6746bc02ae1a5fb7e472e5f5334238b) Then on filer1.prod.lan I made a zpool mirror over those 4 disks (https://gist.github.com/silenius/eecd61ad07385e16b41b05e6d2373a9a) Interfaces are configured as the following: https://gist.github.com/silenius/4af55df446f82319eaf072049bc9a287 with "bge1" being the dedicated interface for iSCSI traffic, and "bge0" the "main" interface through which $clients access the filer (it has a floating IP 192.168.10.15). (I haven't made any network optimizations yet) Primary results are encouraging too, although I haven't tested under heavy write yet. I made more or less what Ben did above, trying to corrupt the pool and ... without success :) I also checked manually with: $> md5 -qs "$(find -s DIR -type f -print0|xargs -0 md5 -q)" to check the integrity of the DIR I copied. I tried also a basic failover scenario with https://gist.github.com/silenius/b81e577f0f0a37bf7773ef15f7d05b5d which seems to work atm. To avoid a split-brain scenario I think it is also very important that the pool isn't automatically mounted at boot (so setting cachefile=none) Comments ? :) Julien > > Ben > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" -- Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCgAGBQJXqyfNAAoJELK7NxCiBCPAn60QAIBd0aHdlnnJEUXV/QxmtEoH 79RadAUJov0C2feea/SPBWaPXAqc/k5Q8BTLNABsSBu0LFcKzDDzVTbQ8l+JhvIn MkcMRICuiFMRYQvE3LNLthZAQrzTguJkcYYshTFvYmk6qSYpCWwPQvtlfw64a9bC Eclk7889otGlRRL3Bi44MshoGuCsFlh9jrNrKlSjQJxNZfO49UysejZxALIKSnw0 lp4J/ByT/AfcSNMjwBYxYPZ08jUiq1Fjo7CYQJuvQlBll/GxirRxTypQPxe7jGEf Ij9eKp/gyTNgOrn/i7TZj8LmLKtsYM4XOKjaMYnrA0yQ49+Ez331Ub9Fta20NyzU fPn0qusttxMy9nc9GZN4QabV5LI36p85cQjiibF22euuyB0jf+EwE6kqPdYGSYeH +5Nrl0hDln62RfXEihwJu0oqNta8/uFlCoEVBvZkZBf2rGJLc7yi+TqHq5jODv37 H/PFBpIYz+t0z8EzA8uZYLgQ3hATEz+z9+PaVaxYqGDbMehy+4o51GQ7O/aJjVoL bs3LxJzkiElNx+32lmWrq2gcdpn5ZZQTQQr0hV7Uzw/VoWQcz39C5gCEIKuT8us4 6OQ4Slgbrnb8Vx3Na0H1tGaH9T8+Nthn1GQpJGlSISH5e9FRMvsPzCg2vubZdhlm jZv2Dt8MqA9Bttrml/BG =lU8A -----END PGP SIGNATURE-----help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160810131040.GH70364>