From owner-freebsd-fs@FreeBSD.ORG Mon Jul 8 11:35:38 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A8C0CCC4 for ; Mon, 8 Jul 2013 11:35:38 +0000 (UTC) (envelope-from markus.gebert@hostpoint.ch) Received: from mail.adm.hostpoint.ch (mail.adm.hostpoint.ch [IPv6:2a00:d70:0:a::e0]) by mx1.freebsd.org (Postfix) with ESMTP id 6AF0710FB for ; Mon, 8 Jul 2013 11:35:38 +0000 (UTC) Received: from [2001:1620:2013:1:8863:9957:fb76:e10e] (port=50969) by mail.adm.hostpoint.ch with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1Uw93Q-000Acw-Ll; Mon, 08 Jul 2013 12:52:24 +0200 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: EBS snapshot backups from a FreeBSD zfs file system: zpool freeze? From: Markus Gebert In-Reply-To: <87zjty11gn.wl%berend@pobox.com> Date: Mon, 8 Jul 2013 12:51:42 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <41CC5720-B1EA-4841-8BA5-893F4A628EAD@hostpoint.ch> References: <87li5o5tz2.wl%berend@pobox.com> <87ehbg5raq.wl%berend@pobox.com> <20130703055047.GA54853@icarus.home.lan> <6488DECC-2455-4E92-B432-C39490D18484@dragondata.com> <14A2336A-969C-4A13-9EFA-C0C42A12039F@hostpoint.ch> <87zjty11gn.wl%berend@pobox.com> To: Berend de Boer X-Mailer: Apple Mail (2.1508) Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jul 2013 11:35:38 -0000 On 07.07.2013, at 23:53, Berend de Boer wrote: >>>>>> "Markus" =3D=3D Markus Gebert = writes: >=20 > Markus> But taking a zfs snapshot is an atomic operation. Why not > Markus> use that? For example: >=20 > Markus> 1. snapshot the zfs at the same point in time you'd issue > Markus> that ioctl on Linux 2. take the EBS snapshot at any time > Markus> 3. clone the EBS snapshot to the new/other VM 4. zfs > Markus> import the pool there 5. zfs rollback the filesystem to > Markus> the snapshot taken in step 1 (or clone it and use that) >=20 > OK, various tests later: this does not really work. If you create the > snapshot, and make a backup, the snapshot does not show up on the > backup (whatever the reason, perhaps the disks so inconsistent zfs has > to rollback). I was under the impression that metadata operations are always = synchronous in zfs, so since 'zfs snapshot' is such an operation it = should be on disk as soon as the command completes. But I've nerver = actually confirmed this. So, it could be that zfs snapshots don't get = commited to disk immediately after all, or EBS confirmed write/flush = commands that we're not commited to whatever is considered stable = storage in that cloud. I don't know how well-behaved EBS and its = snapshots are when ich comes to flush commands, write order etc. So all = just speculation, stopping here... > But the biggest issue is that if writing is going on and you make the > EBS snapshot, you can't really mount it. Maybe zfs mounts after hours, > but I just gave if it didn't mount it after 1.5 hours. By 'mount' do you mean the import of the pool? Did you use -F on import? = In any case, this sounds too long. Was the system doing IO? > Another interesting thing I've seen was a completely empty drive after > mount! That's a bit unspecific. What's empty? Disk full of zeros? Partition = full of zeros? Pool without file system on it? Pool with empty = filesystems? > So clearly EBS snapshots on a mounted multi-drive pool don't work. I think with a lot of writes and transactions, you can't realiably avoid = a scenario where zfs can't find a valid or good enough mutual = transaction accross all disks. By avoiding writes while doing the EBS = snapshots, you could more likely end up with something you can actually = import. If you can't avoid the writes, you're out of luck. What you = really need in that case is the ability to snapshot all EBS disks as = group. Anyway, with the tools at hand (no IO "freeze" like Linux, no EBS = snapshots for groups of disks), I don't think you can acomplish what you = originally intended. Markus