From owner-freebsd-fs@freebsd.org Sat Jul 2 15:04:25 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AB3F7B8FE24 for ; Sat, 2 Jul 2016 15:04:25 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x233.google.com (mail-wm0-x233.google.com [IPv6:2a00:1450:400c:c09::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3E2462635 for ; Sat, 2 Jul 2016 15:04:25 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x233.google.com with SMTP id r201so63279709wme.1 for ; Sat, 02 Jul 2016 08:04:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=IyE2mf+k+cmA39kJ6mLs7DmkSGvdavwBuYNk3P26Joo=; b=fK++Sh+omSOQIBQxoZugVl6XsQMFR/CD11R2ImB3srOmF4tzC63cAP4ubyaAqeCMww auAHGBKoAl3q8eX6yqDtLxZCBdRFGCMLCZoWbRKvtLvuvDUmcr7TVFoofN1DOQcxYv0T DvUyTJCKOhTlUOUNUgyzluUyubB4SFfwWhdaQDqILP/y/3z0hWZOEQ/6KBfeu5hDElxS hvWF4FcBwcs4SKqJUGgSqjcBG4lcUguDCp4KlN/2NHXyHok/4jRuZaab/mMv/fJyImGU HGITJ4UXcER3+sDmqdRIoIS1s9kVyHj8LJd4vrVUz9Vj64GKbLEPYO+N1rdzk1bh7nmE m+vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=IyE2mf+k+cmA39kJ6mLs7DmkSGvdavwBuYNk3P26Joo=; b=GRmbd2nscSu5Fqd76eT1ThDhqIsr6/wuVxczORLoLLyp15mGNRPktZvLRVhyJRcmyC 7HwMYNs1bP8v5svpvK34CtLKjwBLHJWfcKXer8kXDp+dhbZeGD9F/ixYdgLmGLqVvPjP 2qfKUycJHwK/8vTFk17Wg0UZ3oB5hyeMz1rLMlpMIU4WmSvXOSv5W9pRwy36xcqFuF2E IXuMDB4WpiKqTIz3nqHmmK7mttZBNqM2ULWreDf78IVU3B22+gS24hPbqLj8fKlUPmx9 8AQEXBvlOlcQZsOSx4ArAXbZx/wQ0A/lah+whQi1CWB2f847eRBR7Z3piNe8FiBNyfUB xtOg== X-Gm-Message-State: ALyK8tJu2BtR2V0YyJOFru2xLeID3l/ooD86s8aDvxLCLUQxdCHq+JM6xjem3+XN8MTogA== X-Received: by 10.28.107.23 with SMTP id g23mr3380231wmc.101.1467471863555; Sat, 02 Jul 2016 08:04:23 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id n26sm3267499wmi.3.2016.07.02.08.04.22 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 02 Jul 2016 08:04:22 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: HAST + ZFS + NFS + CARP From: Ben RUBSON In-Reply-To: <20160630185701.GD5695@mordor.lan> Date: Sat, 2 Jul 2016 17:04:22 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jul 2016 15:04:25 -0000 > On 30 Jun 2016, at 20:57, Julien Cigar wrote: >=20 > On Thu, Jun 30, 2016 at 11:32:17AM -0500, Chris Watson wrote: >>=20 >>=20 >> Sent from my iPhone 5 >>=20 >>>=20 >>>>=20 >>>> Yes that's another option, so a zpool with two mirrors (local +=20 >>>> exported iSCSI) ? >>>=20 >>> Yes, you would then have a real time replication solution (as HAST), = compared to ZFS send/receive which is not. >>> Depends on what you need :) >>>=20 >>>>=20 >>>>> ZFS would then know as soon as a disk is failing. >>=20 >> So as an aside, but related, for those watching this from the peanut = gallery and for the benefit of the OP perhaps those that run with this = setup might give some best practices and tips here in this thread on = making this a good reliable setup. I can see someone reading this thread = and tossing two crappy Ethernet cards in a box and then complaining it = doesn't work well.=20 >=20 > It would be more than welcome indeed..! I have the feeling that HAST > isn't that much used (but maybe I am wrong) and it's difficult to find=20= > informations on it's reliability and concrete long-term use cases... >=20 > Also the pros vs cons of HAST vs iSCSI I made further testing today. # serverA, serverB : kern.iscsi.ping_timeout=3D5 kern.iscsi.iscsid_timeout=3D5 kern.iscsi.login_timeout=3D5 kern.iscsi.fail_on_disconnection=3D1 # Preparation : - serverB : let's make 2 iSCSI targets : rem3, rem4. - serverB : let's start ctld. - serverA : let's create a mirror pool made of 4 disks : loc1, loc2, = rem3, rem4. - serverA : pool is healthy. # Test 1 : - serverA : put a lot of data into the pool ; - serverB : stop ctld ; - serverA : put a lot of data into the pool ; - serverB : start ctld ; - serverA : make all pool disks online : it works, pool is healthy. # Test 2 : - serverA : put a lot of data into the pool ; - serverA : export the pool ; - serverB : import the pool : it does not work, as ctld locks the disks = ! Good news, nice protection (both servers won't be able to access the = same disks at the same time). - serverB : stop ctld ; - serverB : import the pool : it works, 2 disks missing ; - serverA : let's make 2 iSCSI targets : rem1, rem2 ; - serverB : make all pool disks online : it works, pool is healthy. # Test 3 : - serverA : put a lot of data into the pool ; - serverB : stop ctld ; - serverA : put a lot of data into the pool ; - serverB : import the pool : it works, 2 disks missing ; - serverA : let's make 2 iSCSI targets : rem1, rem2 ; - serverB : make all pool disks online : it works, pool is healthy, but = of course data written at step3 is lost. # Test 4 : - serverA : put a lot of data into the pool ; - serverB : stop ctld ; - serverA : put a lot of data into the pool ; - serverA : export the pool ; - serverA : let's make 2 iSCSI targets : rem1, rem2 ; - serverB : import the pool : it works, pool is healthy, data written at = step3 is here. # Test 5 : - serverA : rsync a huge remote repo into the pool in the background ; - serverB : stop ctld ; - serverA : 2 disks missing, but rsync still runs flawlessly ; - serverB : start ctld ; - serverA : make all pool disks online : it works, pool is healthy. - serverB : ifconfig down ; - serverA : 2 disks missing, but rsync still runs flawlessly ; - serverB : ifconfig up ; - serverA : make all pool disks online : it works, pool is healthy. - serverB : power reset ! - serverA : 2 disks missing, but rsync still runs flawlessly ; - serverB : let's wait for server to be up ; - serverA : make all pool disks online : it works, pool is healthy. Quite happy with these tests actually :) Ben