Date: Tue, 15 Jul 2008 10:52:41 -0700 (PDT) From: Matthew Dillon <dillon@apollo.backplane.com> To: freebsd-stable@freebsd.org, sven@dmv.com Subject: Re: Multi-machine mirroring choices Message-ID: <200807151752.m6FHqf7E007803@apollo.backplane.com> References: <200807151523.m6FFNIRJ044047@lurza.secnetix.de> <487CC919.2010203@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
:Oliver Fromme wrote: : :> Yet another way would be to use DragoFly's "Hammer" file :> system which is part of DragonFly BSD 2.0 which will be :> released in a few days. It supports remote mirroring, :> i.e. mirror source and mirror target can run on different :> machines. Of course it is still very new and experimental :> (however, ZFS is marked experimental, too), so you probably :> don't want to use it on critical production machines. : :Let's not get carried away here :) : :Kris Heh. I think its safe to say that a *NATIVE* uninterrupted and fully cache coherent fail-over feature is not something any of us in BSDland have yet. It's a damn difficult problem that is frankly best solved above the filesytem layer, but with filesystem support for bulk mirroring operations. HAMMER's native mirroring was the last major feature to go into it before the upcoming release, so it will definitely be more experimental then the rest of HAMMER. This is mainly because it implements a full blown queue-less incremental snapshot and mirroring algorithm, single-master-to-multi-slave. It does it at a very low level, by optimally scanning HAMMER's B-Tree. In other words, the kitchen sink. The B-Tree propagates the highest transaction id up to the root to support incremental mirroring and that's the bit that is highly experimental and not well tested yet. It's fairly complex because even destroyed B-Tree records and collapses must propagate a transaction id up the tree (so the mirroring code knows what it needs to send to the other end to do comparative deletions on the target). (transaction ids are bundled together in larger flushes so the actual B-Tree overhead is minimal). The rest of HAMMER is shaping up very well for the release. It's phenominal when it comes to storing backups. Post-release I'll be moving more of our production systems to HAMMER. The only sticky issue we have is filesystem-full handling, but it is more a matter of fine-tuning then anything else. -- Someone mentioned atime and mtime. For something like ZFS or HAMMER, these fields represent a real problem (atime more then mtime). I'm kinda interested in knowing, does ZFS do block replacement for atime updates? For HAMMER I don't roll new B-Tree records for atime or mtime updates. I update the fields in-place in the current version of the inode and all snapshot accesses will lock them (in getattr) to ctime in order to guarantee a consistent result. That way (tar | md5) can be used to validate snapshot integrity. At the moment, in this first release, the mirroring code does not propagate atime or mtime. I plan to do it, though. Even though I don't roll new B-Tree records for atime/mtime updates I can still propagate a new transaction id up the B-Tree to make the changes visible to the mirroring code. I'll definitely be doing that for mtime and will have the option to do it for atime as well. But atime still represents a big expense in actual mirroring bandwidth. If someone reads a million files on the master then a million inode records (sans file contents) would end up in the mirroring stream just for the atime update. Ick. -Matt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200807151752.m6FHqf7E007803>