From owner-freebsd-stable@freebsd.org Fri Jun 3 10:40:17 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D298FB68839 for ; Fri, 3 Jun 2016 10:40:17 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C0F4F120E for ; Fri, 3 Jun 2016 10:40:17 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: by mailman.ysv.freebsd.org (Postfix) id BC9C6B68838; Fri, 3 Jun 2016 10:40:17 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BC442B68837 for ; Fri, 3 Jun 2016 10:40:17 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from constantine.ingresso.co.uk (constantine.ingresso.co.uk [IPv6:2a02:b90:3002:e550::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48098120C for ; Fri, 3 Jun 2016 10:40:17 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from dilbert.london-internal.ingresso.co.uk ([10.64.50.6] helo=dilbert.ingresso.co.uk) by constantine.ingresso.co.uk with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.86_2 (FreeBSD)) (envelope-from ) id 1b8mWR-000Ee0-5j for stable@freebsd.org; Fri, 03 Jun 2016 10:40:11 +0000 Received: from petefrench by dilbert.ingresso.co.uk with local (Exim 4.87 (FreeBSD)) (envelope-from ) id 1b8mWR-0000uA-3H for stable@freebsd.org; Fri, 03 Jun 2016 11:40:11 +0100 To: stable@freebsd.org Subject: Re: HAST, zfs and local mirroring In-Reply-To: <57514843.6010003@norma.perm.ru> Message-Id: From: Pete French Date: Fri, 03 Jun 2016 11:40:11 +0100 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2016 10:40:17 -0000 > You message lead me to a though that I could use iSCSI to replicate the > zfs pool from node1 to both iSCSI-provided disk on a node2 in a 4-way > mirror, right ? Are there any obvious obstacles to this, that I don't > see, considering the bandwith will be enough ? I have spent a long time doing this kind of thing - always needing a pair of machines, 2 discs in each and 4-way mirroring. over the last decade weve used ggated/ggatec + gmirrors, iscsi + gmirror, iscsi + zfs and finally hast + zfs. Out of all that I have to say that hnast has been a godsend - its is by far the best of all the solutions. so my recommentation would be to hast the drives into two hast resources and zpool on top of that. yes, you have two hast devices, but why is that an issue ? what we also do is to add separate ether cards into the machines just dedicated to the hast traffic between them, so that is not a bottlneck. the reason hwy hast is so good compared to all the other network block evel solutions we have tried is that it behaves ell in the case when things fail. Theres nothing wrong with either ggatec or iscsi in the normal case, but in the past I had a lot of issues with the mirroring layer not understanding properly when the unerlying devices vanish due to crashes or other unexpected events. As the purpose of all of this is, preseumably, fault tolerance, then thats unfortunate. this may have chnaged, I havent tried these technologies out in a while, but with hast you dont ever see a device simply vanish rom underneath your zpool, and the whole thing continues to work reliably and smoothly until the remote node comes back up, whereupon it resyncs perfectly. really, go with hast + zpool unless you have a good reason not to... -pete.