From owner-freebsd-fs@freebsd.org Thu Aug 18 10:50:16 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 97AC9BBE52A for ; Thu, 18 Aug 2016 10:50:16 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4EEAC1A8C for ; Thu, 18 Aug 2016 10:50:15 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.36] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id 2AC549DD3B3; Thu, 18 Aug 2016 12:50:07 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: HAST + ZFS + NFS + CARP From: Borja Marcos In-Reply-To: <6b866b6e-1ab3-bcc5-151b-653e401742bd@kateley.com> Date: Thu, 18 Aug 2016 12:50:06 +0200 Cc: Chris Watson , freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <354253C2-E42E-4B9C-9931-9135A5A7DFD9@sarenet.es> References: <61283600-A41A-4A8A-92F9-7FAFF54DD175@ixsystems.com> <20160704183643.GI41276@mordor.lan> <20160704193131.GJ41276@mordor.lan> <20160811091016.GI70364@mordor.lan> <1AA52221-9B04-4CF6-97A3-D2C2B330B7F9@sarenet.es> <472bc879-977f-8c4c-c91a-84cc61efcd86@internetx.com> <20160817085413.GE22506@mordor.lan> <465bdec5-45b7-8a1d-d580-329ab6d4881b@internetx.com> <20160817095222.GG22506@mordor.lan> <52d5b687-1351-9ec5-7b67-bfa0be1c8415@kateley.com> <92F4BE3D-E4C1-4E5C-B631-D8F124988A83@gmail.com> <6b866b6e-1ab3-bcc5-151b-653e401742bd@kateley.com> To: linda@kateley.com X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2016 10:50:16 -0000 > On 17 Aug 2016, at 20:03, Linda Kateley wrote: >=20 > You do risk losing data if you batch zfs send. It is very hard to run = that real time. You have to take the snap then send the snap. Most = people run in cron, even if it's not in cron, you would want one to = finish before you started the next. If you lose the sending host before = the receive is complete you won't have a full copy. With zfs though you = will probably still have the data on the sending host, however long it = takes to bring it back up. RSF-1 runs in the zfs stack and send the = writes to the second system. It's kind of pricey, but actually much less = expensive than commercial alternatives. Doing somewhat critical stuff off cron is not usually a good idea. I do = ZFS replication with a custom program which makes sure of some important = stuff: - Using holds to avoid an accidental snapshot deletion to require a full = send/receive.=20 - Avoiding starting a new send/receive on a dataset in case the previous = one didn=E2=80=99t finish for whatever reason (the main problem with = cron) - Offering the possibility of some random variation on the replication = period so that, in case several happen to start simultaneously, you = don=E2=80=99t have a periodically overloaded system. - Avoiding mounting the replicas so that the receive won=E2=80=99t need = a rollback, which would be potentially risky. - Supports one-to-many replicas, with different periodicity for each = destination if required. I am sorry I can=E2=80=99t share it (company property) but the program = is rather silly anyway. The important work was the decision to have the = previous features, and a design decision to avoid destructive and portentially = error-prone operations such as rollbacks.=20 Most applications that require real time replication are databases, and = they usually include a clustering option which can be much simpler to = manage (and more robust in this case) than filesystem replication. For other cases, often you can design around the loss of a small amount = of data. I understand that in some cases you have no other option, but the benefits of asynchronous send/receive are so many, especially if = you are on a tight budget, it=E2=80=99s well worth to try to make the = most of it. Borja.