From owner-freebsd-fs@freebsd.org Mon Jul 4 05:56:32 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2943BB91E3F for ; Mon, 4 Jul 2016 05:56:32 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22e.google.com (mail-wm0-x22e.google.com [IPv6:2a00:1450:400c:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ADDB42E9F for ; Mon, 4 Jul 2016 05:56:31 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22e.google.com with SMTP id f126so89614848wma.1 for ; Sun, 03 Jul 2016 22:56:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=taqJckWj2YiqIN870q/Hv6OXvldTcsXXo436yNIjIs8=; b=0an3ErElBAlnKicuF1RsQkzNDddhUqIlw3DRGOY7pt5VEM4fDILbocAr08W6/HsbUD KAZeiLA9WLB2TERjZt5RfrTwSQ22bKGG9ijaIhqRFR8QXp5zoqLn9rNPD4x/4IhHYzFQ mnaYuPXelW0pT9sltCiKDK6QyCvwTcf6UV5PlTTmpdhYWHsxXeSTYik9ahSL4qoT7D07 LG4Aysee/1BkRjFeb2rqzNCHSklTkv5TfXR4f/V2I+apu2lX61EgwU9gaYle8n30L2MF eXajfxX6N7ddNUFIeKnnyhguy46kPuRE1CZLLFv13iquCKL4PZTB5Y5XQudLH/zKgUO9 hZwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=taqJckWj2YiqIN870q/Hv6OXvldTcsXXo436yNIjIs8=; b=YFHAOuJRZ6VQiW56ahCJkZd0+fbfZKCb04OXRXkujIlHmU/dbqNtD7pEoEpcvSJFQ7 C7p8K9YZK/ABcaq30izs7Np/dUt5CJqJVyAbnLPEkCGML4EpyQ2+q0RMrh8N76P3T5Za 6oYIvtmIaVzXRZ61D/10PgQvmSCSDGX3BBFLgXVFKuL4p6ulSZLk1EnNfOCBeUa8gAlr x7NPwTTu7J5Juw3BrjC8223dSkzyoOaFGLJr7IhEQMqru7acgiAhDw9WjH+7YRyYSzkf r5FdeFowllqqlRdN0KqJIFL58SfPFV/sx8CnttVMFkQQF0vX1QrWy4IQdo8sLv4Z797l RpGQ== X-Gm-Message-State: ALyK8tJMur8D2rdCXTLzFZ13C4XPqs6xQ7PX/dLQjPbq4bcdLHtnvSBiqif5itujie1INQ== X-Received: by 10.28.144.80 with SMTP id s77mr9064046wmd.41.1467611789412; Sun, 03 Jul 2016 22:56:29 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id dv8sm3583321wjd.49.2016.07.03.22.56.28 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 03 Jul 2016 22:56:28 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: HAST + ZFS + NFS + CARP From: Ben RUBSON In-Reply-To: <20160703214723.GF41276@mordor.lan> Date: Mon, 4 Jul 2016 07:56:27 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20160630144546.GB99997@mordor.lan> <71b8da1e-acb2-9d4e-5d11-20695aa5274a@internetx.com> <20160630153747.GB5695@mordor.lan> <63C07474-BDD5-42AA-BF4A-85A0E04D3CC2@gmail.com> <678321AB-A9F7-4890-A8C7-E20DFDC69137@gmail.com> <20160630185701.GD5695@mordor.lan> <6035AB85-8E62-4F0A-9FA8-125B31A7A387@gmail.com> <20160703192945.GE41276@mordor.lan> <20160703214723.GF41276@mordor.lan> To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2016 05:56:32 -0000 > On 03 Jul 2016, at 23:47, Julien Cigar wrote: >=20 > On Sun, Jul 03, 2016 at 09:29:46PM +0200, Julien Cigar wrote: >> On Sat, Jul 02, 2016 at 05:04:22PM +0200, Ben RUBSON wrote: >>>=20 >>>> On 30 Jun 2016, at 20:57, Julien Cigar = wrote: >>>>=20 >>>> On Thu, Jun 30, 2016 at 11:32:17AM -0500, Chris Watson wrote: >>>>>=20 >>>>>=20 >>>>> Sent from my iPhone 5 >>>>>=20 >>>>>>=20 >>>>>>>=20 >>>>>>> Yes that's another option, so a zpool with two mirrors (local +=20= >>>>>>> exported iSCSI) ? >>>>>>=20 >>>>>> Yes, you would then have a real time replication solution (as = HAST), compared to ZFS send/receive which is not. >>>>>> Depends on what you need :) >>>>>>=20 >>>>>>>=20 >>>>>>>> ZFS would then know as soon as a disk is failing. >>>>>=20 >>>>> So as an aside, but related, for those watching this from the = peanut gallery and for the benefit of the OP perhaps those that run with = this setup might give some best practices and tips here in this thread = on making this a good reliable setup. I can see someone reading this = thread and tossing two crappy Ethernet cards in a box and then = complaining it doesn't work well.=20 >>>>=20 >>>> It would be more than welcome indeed..! I have the feeling that = HAST >>>> isn't that much used (but maybe I am wrong) and it's difficult to = find=20 >>>> informations on it's reliability and concrete long-term use = cases... >>>>=20 >>>> Also the pros vs cons of HAST vs iSCSI >>>=20 >>> I made further testing today. >>>=20 >>> # serverA, serverB : >>> kern.iscsi.ping_timeout=3D5 >>> kern.iscsi.iscsid_timeout=3D5 >>> kern.iscsi.login_timeout=3D5 >>> kern.iscsi.fail_on_disconnection=3D1 >>>=20 >>> # Preparation : >>> - serverB : let's make 2 iSCSI targets : rem3, rem4. >>> - serverB : let's start ctld. >>> - serverA : let's create a mirror pool made of 4 disks : loc1, loc2, = rem3, rem4. >>> - serverA : pool is healthy. >>>=20 >>> # Test 1 : >>> - serverA : put a lot of data into the pool ; >>> - serverB : stop ctld ; >>> - serverA : put a lot of data into the pool ; >>> - serverB : start ctld ; >>> - serverA : make all pool disks online : it works, pool is healthy. >>>=20 >>> # Test 2 : >>> - serverA : put a lot of data into the pool ; >>> - serverA : export the pool ; >>> - serverB : import the pool : it does not work, as ctld locks the = disks ! Good news, nice protection (both servers won't be able to access = the same disks at the same time). >>> - serverB : stop ctld ; >>> - serverB : import the pool : it works, 2 disks missing ; >>> - serverA : let's make 2 iSCSI targets : rem1, rem2 ; >>> - serverB : make all pool disks online : it works, pool is healthy. >>>=20 >>> # Test 3 : >>> - serverA : put a lot of data into the pool ; >>> - serverB : stop ctld ; >>> - serverA : put a lot of data into the pool ; >>> - serverB : import the pool : it works, 2 disks missing ; >>> - serverA : let's make 2 iSCSI targets : rem1, rem2 ; >>> - serverB : make all pool disks online : it works, pool is healthy, = but of course data written at step3 is lost. >>>=20 >>> # Test 4 : >>> - serverA : put a lot of data into the pool ; >>> - serverB : stop ctld ; >>> - serverA : put a lot of data into the pool ; >>> - serverA : export the pool ; >>> - serverA : let's make 2 iSCSI targets : rem1, rem2 ; >>> - serverB : import the pool : it works, pool is healthy, data = written at step3 is here. >>>=20 >>> # Test 5 : >>> - serverA : rsync a huge remote repo into the pool in the background = ; >>> - serverB : stop ctld ; >>> - serverA : 2 disks missing, but rsync still runs flawlessly ; >>> - serverB : start ctld ; >>> - serverA : make all pool disks online : it works, pool is healthy. >>> - serverB : ifconfig down ; >>> - serverA : 2 disks missing, but rsync still runs flawlessly ; >>> - serverB : ifconfig up ; >>> - serverA : make all pool disks online : it works, pool is healthy. >>> - serverB : power reset ! >>> - serverA : 2 disks missing, but rsync still runs flawlessly ; >>> - serverB : let's wait for server to be up ; >>> - serverA : make all pool disks online : it works, pool is healthy. >>>=20 >>> Quite happy with these tests actually :) >>=20 >> Thank you very much for thoses quick tests! I'll start my own ones >> tomorrow, but based on your preliminary it *seems* that ZFS + iSCSI >> combinaison could be a potential candidate for what I'd like to do..! >=20 > another question from a performance point of view, imagine that you=20 > create a single mirror zpool, something like: > $> zpool create storage mirror loc1 loc2 rem1 rem2 >=20 > (where rem1 and rem2 are iSCSI disks) >=20 > I guess that ZFS will split the read requests accross all devices in > order to maximize performance... which could lead to contrary to what = is > expecpted when iSCSI disks are involved, no? Not necessarily no, if your network card is not a bottleneck. If you only have 1Gbps adapters, forget this solution. You should have at least 10Gbps adapters, and depending on how many = disks you have, you would have to go with 25/40Gbps adapters... > Is there some sysctl params which could prevent this unexpected > behavior? >=20 >>=20 >>>=20 >>> Ben >>>=20 >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >>=20 >> --=20 >> Julien Cigar >> Belgian Biodiversity Platform (http://www.biodiversity.be) >> PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 >> No trees were killed in the creation of this message. >> However, many electrons were terribly inconvenienced. >=20 >=20 >=20 > --=20 > Julien Cigar > Belgian Biodiversity Platform (http://www.biodiversity.be) > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > No trees were killed in the creation of this message. > However, many electrons were terribly inconvenienced.