From owner-freebsd-fs@freebsd.org Mon Jul 4 12:25:33 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BDBC4B910A9 for ; Mon, 4 Jul 2016 12:25:33 +0000 (UTC) (envelope-from julien@perdition.city) Received: from relay-b02.edpnet.be (relay-b02.edpnet.be [212.71.1.222]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "edpnet.email", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 670782091 for ; Mon, 4 Jul 2016 12:25:32 +0000 (UTC) (envelope-from julien@perdition.city) X-ASG-Debug-ID: 1467635127-0a7b8d55be1d0ae30001-3nHGF7 Received: from mordor.lan (213.219.165.225.bro01.dyn.edpnet.net [213.219.165.225]) by relay-b02.edpnet.be with ESMTP id pwKqCi0SoSqyoqCO (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 04 Jul 2016 14:25:29 +0200 (CEST) X-Barracuda-Envelope-From: julien@perdition.city X-Barracuda-Effective-Source-IP: 213.219.165.225.bro01.dyn.edpnet.net[213.219.165.225] X-Barracuda-Apparent-Source-IP: 213.219.165.225 Date: Mon, 4 Jul 2016 14:25:27 +0200 From: Julien Cigar To: Ben RUBSON Cc: freebsd-fs@freebsd.org Subject: Re: HAST + ZFS + NFS + CARP Message-ID: <20160704122527.GH41276@mordor.lan> X-ASG-Orig-Subj: Re: HAST + ZFS + NFS + CARP References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="y9PDtDHaFrXNoMPU" Content-Disposition: inline In-Reply-To: <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> User-Agent: Mutt/1.6.1 (2016-04-27) X-Barracuda-Connect: 213.219.165.225.bro01.dyn.edpnet.net[213.219.165.225] X-Barracuda-Start-Time: 1467635128 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://212.71.1.222:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 5175 X-Virus-Scanned: by bsmtpd at edpnet.be X-Barracuda-BRTS-Status: 1 X-Barracuda-Bayes: INNOCENT GLOBAL 0.5000 1.0000 0.0100 X-Barracuda-Spam-Score: 0.51 X-Barracuda-Spam-Status: No, SCORE=0.51 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=6.0 tests=BSF_SC1_TG070 X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.31003 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.50 BSF_SC1_TG070 Custom Rule TG070 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 12:25:33 -0000 --y9PDtDHaFrXNoMPU Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jul 02, 2016 at 05:04:22PM +0200, Ben RUBSON wrote: >=20 > > On 30 Jun 2016, at 20:57, Julien Cigar wrote: > >=20 > > On Thu, Jun 30, 2016 at 11:32:17AM -0500, Chris Watson wrote: > >>=20 > >>=20 > >> Sent from my iPhone 5 > >>=20 > >>>=20 > >>>>=20 > >>>> Yes that's another option, so a zpool with two mirrors (local +=20 > >>>> exported iSCSI) ? > >>>=20 > >>> Yes, you would then have a real time replication solution (as HAST), = compared to ZFS send/receive which is not. > >>> Depends on what you need :) > >>>=20 > >>>>=20 > >>>>> ZFS would then know as soon as a disk is failing. > >>=20 > >> So as an aside, but related, for those watching this from the peanut g= allery and for the benefit of the OP perhaps those that run with this setup= might give some best practices and tips here in this thread on making this= a good reliable setup. I can see someone reading this thread and tossing t= wo crappy Ethernet cards in a box and then complaining it doesn't work well= =2E=20 > >=20 > > It would be more than welcome indeed..! I have the feeling that HAST > > isn't that much used (but maybe I am wrong) and it's difficult to find= =20 > > informations on it's reliability and concrete long-term use cases... > >=20 > > Also the pros vs cons of HAST vs iSCSI >=20 > I made further testing today. >=20 > # serverA, serverB : > kern.iscsi.ping_timeout=3D5 > kern.iscsi.iscsid_timeout=3D5 > kern.iscsi.login_timeout=3D5 > kern.iscsi.fail_on_disconnection=3D1 >=20 > # Preparation : > - serverB : let's make 2 iSCSI targets : rem3, rem4. > - serverB : let's start ctld. > - serverA : let's create a mirror pool made of 4 disks : loc1, loc2, rem3= , rem4. > - serverA : pool is healthy. >=20 > # Test 1 : > - serverA : put a lot of data into the pool ; > - serverB : stop ctld ; > - serverA : put a lot of data into the pool ; > - serverB : start ctld ; > - serverA : make all pool disks online : it works, pool is healthy. >=20 > # Test 2 : > - serverA : put a lot of data into the pool ; > - serverA : export the pool ; > - serverB : import the pool : it does not work, as ctld locks the disks != Good news, nice protection (both servers won't be able to access the same = disks at the same time). > - serverB : stop ctld ; > - serverB : import the pool : it works, 2 disks missing ; > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > - serverB : make all pool disks online : it works, pool is healthy. >=20 > # Test 3 : > - serverA : put a lot of data into the pool ; > - serverB : stop ctld ; > - serverA : put a lot of data into the pool ; > - serverB : import the pool : it works, 2 disks missing ; > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > - serverB : make all pool disks online : it works, pool is healthy, but o= f course data written at step3 is lost. >=20 > # Test 4 : > - serverA : put a lot of data into the pool ; > - serverB : stop ctld ; > - serverA : put a lot of data into the pool ; > - serverA : export the pool ; > - serverA : let's make 2 iSCSI targets : rem1, rem2 ; > - serverB : import the pool : it works, pool is healthy, data written at = step3 is here. >=20 > # Test 5 : > - serverA : rsync a huge remote repo into the pool in the background ; > - serverB : stop ctld ; > - serverA : 2 disks missing, but rsync still runs flawlessly ; > - serverB : start ctld ; > - serverA : make all pool disks online : it works, pool is healthy. > - serverB : ifconfig down ; > - serverA : 2 disks missing, but rsync still runs flawlessly ; > - serverB : ifconfig up ; > - serverA : make all pool disks online : it works, pool is healthy. > - serverB : power reset ! > - serverA : 2 disks missing, but rsync still runs flawlessly ; > - serverB : let's wait for server to be up ; > - serverA : make all pool disks online : it works, pool is healthy. >=20 > Quite happy with these tests actually :) another option I had in mind is to export ZVOL through iSCSI: - on serverA: $> zpool create storage mirror /dev/ada1 /dev/ada2 - on serverB: $> zpool create storage mirror /dev/ada1 /dev/ada2 create a 50G dedicated redundant storage for serverC: - on serverA: $> zfs create -V 50G storage/serverc - on serverB: $> zfs create -V 50G storage/serverc iSCSI export /dev/zvol/storage/serverc from serverA and serverB and create a ZFS dataset on serverC: - on serverC: $> zpool create storage mirror /dev/ivol1 /dev/ivol2 (where ivol1 is the volume from serverA and ivol2 the volume from serverB) create, for example, a dataset for the dovecot service - on serverC: $> zfs create -o compress=3Dlz4 storage/dovecot any thoughts? >=20 > Ben >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --y9PDtDHaFrXNoMPU Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCgAGBQJXelW0AAoJELK7NxCiBCPAN8YP/Rl6MQcdQpjUeKAE5n06V/1g aUUuKOsMvYOKwWLNguyPebd9lROMkcGmT9HPCDJx0QmN6UipLAmn/UtcqoUOxvjX cEJZJBp5RpKibGMXjsLyii3epCAjd/xYzNYckXwXYaSHA6FVekZIjSnxyUYJnPuz Fg2pyW8us2D9emY8bqXQ+WOZvrMX+1RM90RDwjzNjeOWeX92poNm7MhnCPytfDre SkvfgoISbmiExDmqEWkHsXBjMvXGtdpZaqtxlHAM/p4ontdyjOsbkfo0qF7Z8jqp NyAenAbEyrjVousHQICL9P9wHOlgQWSJpuassJgPbQpAm+kDrYKRoJdYG91FJZM9 lzZFWFe/HFVS27xQi5uuGt+Fxdue2FwY42mJNcOlCY6iJe12yJDr/sAUYVQJ3nNi KggK9lEr9bwOtl548IpoRxjgE6epVTu3xFbup5IzVKsW9/C6Olr5ufBJid42aqoe 7diqg5sjO41kanNU3gp0IfV9sH31Grp+thBjqzMSFjfVY4VRWb/sOuBkrFxPvNgJ gtrY5oo5Bs0QdEGmjiB+I1L4PgsBArzYwkzIbjL45/hnwlS6RZc1ORi/4CeQ8O5X YaQzPitlmfMxdd8Rv8/PNWHM+iBGsw1BQPWGsBDg/fJGJbvcQbV8KSUpV9QWiwFZ MiZvWynjeqFY5X62fH59 =inJ8 -----END PGP SIGNATURE----- --y9PDtDHaFrXNoMPU--