Date: Mon, 6 Oct 2008 10:59:40 -0700 From: Jeremy Chadwick <koitsu@FreeBSD.org> To: Evren Yurtesen <yurtesen@ispro.net.tr> Cc: questions@freebsd.org Subject: Re: continuous backup solution for freebsd? Message-ID: <20081006175940.GB26368@icarus.home.lan> In-Reply-To: <48EA45D2.70502@ispro.net.tr> References: <48E9E146.9040308@ispro.net.tr> <20081006112030.GA18670@icarus.home.lan> <48EA2270.2080405@ispro.net.tr> <20081006152430.GA23608@icarus.home.lan> <48EA45D2.70502@ispro.net.tr>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Oct 06, 2008 at 08:07:30PM +0300, Evren Yurtesen wrote: > First of all, I am not an r1soft advocate, but they seem to be making a > software which is popular and affordable and interested in giving > FreeBSD support... r1soft is not the issue here, the problem is that > there is no way to do near continuous backups on FreeBSD servers. > > Jeremy Chadwick wrote: > >> That said, I'd like to know exactly how "low-level" R1Soft's software >> truly is. dump(8), AFAIK, is "block-level" -- and that's a userland >> program. Does R1Soft's software *truly* require kernel-land? I have >> more to say on that issue (not against R1Soft, but speaking with regards >> to the current state of FreeBSD's developer count) if it truly does. > > I think you might not have understood the concept of near continuous > backups. The R1Soft backup monitors the filesystem operations and backs > up written blocks. So it has to know what is written and when to be able > to back it up. The dump command simply reads/writes the blocks. It cant > only read changed blocks. It has to read the whole thing (inefficient). It depends on how it's implemented, but in general, yes, I guess this would advocate reliance on GEOM, which would be kernel. The thing is, the GEOM gate class could be extended to handle this situation -- it's a class intended for filesystem replication in real-time, over a network. That said, I shall unleash with the comments I had originally planned on including, but removed them since I felt it might be too hasty of me. The sad reality with FreeBSD is that we do not have enough clueful folks who are familiar with the kernel innards. Those who are clueful are very busy (with other things, and with real life), and often do not have the time to give direct/constant focus on a single item for long periods of time. I have a mental list of those who are absolutely incredible FreeBSD developers (and I will name one of them: pjd@, who should be given tens of thousands of dollars, IMHO, for his work on bringing ZFS to FreeBSD), but the list is small compared to how many *users* we have. The learning curve for getting familiar with pieces of the FreeBSD kernel is astoundingly large. I myself have tried it on a couple of occasions, but lack of concise and up-to-date documentation makes it very difficult to accomplish. (I'm familiar with very old operating systems, such as MS-DOS, ProDOS, and GS/OS on the Apple IIGS -- FreeBSD is far from those). Books are also not of much help, as I've been told that the existing book which covers FreeBSD engineering models is "long outdated" and that "many pieces now are completely different". A complete and total moral killer right there. The book is for FreeBSD 5.2, by the way. We cannot rely on the FreeBSD Documentation folks to write the necessary docs either, because they do not have the knowledge of the kernel to write such. As someone who's written software, I can assure you that the only way to get good documentation for low-level pieces is to write the documentation in parallel to the code; otherwise, you end up with lots of "after-the-fact" reverse engineering efforts, which takes tons of time, and requires a lot of communication between the code author(s) and the documenters. We're talking thousands of hours here. Requiring the user/developer to reverse-engineer hundreds of thousands of lines of C code is not reasonable/plausible; hardly anyone is willing to do that for free. This is why Linux often has the upper hand: they have multiple eyes and individuals fully familiar with different pieces of the kernel. If one or two people go on hiatus or disappear (death, life, whatever), the existing kernel piece does not sit in limbo for years -- there are other people to pick up the responsibilities. Much of the FreeBSD kernel and device layer does not have this degree of freedom; much is single-person, single-maintainer, single-point-of-failure. Then there's commercial company support -- by that I mean, actual hardware vendors that support the OS. FreeBSD has some of this, but most are very small companies (few employees), with limited funds, or have very *very* limited/specific focus; there are a couple big ones, but they are few and far between. Linux has hundreds, and many of the vendors are *very* large. In fact, the support is so large that freelance Linux developers are able to get things like development PCI boards for new NICs from the vendors directly; FreeBSD? Rare. What this means is that the commercial world takes Linux seriously, while FreeBSD not so. Sorry, but that's reality. It amazes me how "easy" someone can pick up programming something in kernel-land for Linux, while for FreeBSD it just doesn't happen on a regular basis. When I see it happen, it's bizarre -- suddenly out of no where comes this one fellow (we'll call him Bob), appearing on a mailing list with a bunch of patches. Heard of him before? Nope, but here he is, and somehow he engineered all of this. What's his background? I don't know, maybe some old guy who lives in a cave and has been studying BSD code in the steam tunnels; who knows. It's like they literally come out of the woodwork, while I don't see this sort of behaviour with Linux. With Linux, it's often "Hi I work for <large vendor>, we're adding support for Linux, I need some help with regards to the following kernel piece..." and they've got responses from 20 people in 24 hours. What I'm saying is that Linux has the upper hand here. More eyes, more people, more developers, larger community, larger vendor support, and much **much** faster turn-around time on fixes/bugs. We can sit here and argue about those facts all we want (it's the equivalent of doing burn-outs in an AMC Pacer in a parking lot -- wasted time, zero gain), but nothing changes the facts. >>> Continuous backups as well as bare-metal-restore seem to be a key >>> feature for many hosters. >> >> Regarding continuous backups: the GEOM gate class could be used for >> this. Meaning, I think it could be used as an alternate to R1Soft's >> software. > > The GEOM gate allows mirroring to a remote machine, am I not right? That > would be more or less same as same as using RAID. The continuous backup > (or near continuous) means that you can restore the filesystem to a > point like 15 minutes ago, or 1 hour ago. What you're talking about sounds like filesystem snapshots, with an *immense* amount of granularity. Enterprise-level filers have this capability (I'm talking Network Appliance), and UFS2 with softupdates have it (called snapshots; but please be aware that there are *HUGE* problems with it, and it should not be relied upon for this kind of functionality) -- but nothing that can be restored within *minutes*. Even Netapp filers do not have that kind of granularity -- the amount of disk it would require would be astounding. Netapp filers often do snapshot generation hourly or nightly (it's configurable how often); minutes is unheard of. ZFS also has snapshot capability, but does not have real-time filesystem mirroring capabilities over a network (keyword: real-time). > Besides, I hear geom might > have network delay problems and it is much more complicated setup to > build two machines in mirror configuration just for backup purposes as > well as you cant restore to a point in the past. Well, GEOM gate is the only thing I know of which replicates filesystems over a network in real-time. >> Regarding bare-metal restoration I'm not aware of how to do that under >> FreeBSD, Linux, or even Solaris "with ease". In most cases, companies >> develop their own PXE-booting environments which wipe the disks and >> reinstall + restore data as they see fit. There is no "standard". > > OK. Actually there is more than one solution which can do > bare-metal-restores for FreeBSD also. However those solutions at best > rely on nightly backups of the filesystems. With R1Soft, you can restore > the system to only few minutes before the total meltdown. > > Unrelated to bare metal restore, with normal backups you are not taking > backups of files which are created/deleted often. For example this can > be customer mails or if a hacker hacks the box and removes his trails. > Even sometimes customers upload some file and remove from their computer > the same they and then accidentally remove from the server. With R1Soft > backup the data would go into the backup server right away and you an > restore every single file independent of when it was put or removed. Right. We're definitely talking about snapshots, at least in concept. The fact that you're able to restore data within *minutes* is pretty impressive. I'm curious what sort of disk requirements are needed though (I guess it depends on how often changes happen on the filesystem). >>> FreeBSD is loosing users because of this issue. >> >> Why does the "number of FreeBSD users" matter? Quantity does not >> necessarily represent quality. > > Thats a perfectly fine statement. But a quality product would be nothing > without users. As well as this problem effects the quality. Consider a > system which has sensitive data which shouldnt get lost, with continuous > data protecton you can restore such failed system to only few minutes > before the failure point. Doing this is currently impossible with > FreeBSD. Best we can do is to return to previous snapshot taken (which > might be a day old). This is an important design criteria since > restoring the lost data might be time consuming and expensive. Thge > issue is not even r1soft, they are just the most popular company giving > such solution, only if there was at least one backup solution which > could provide near continuous data protection... Part of the "design issue" here is that there's two concepts being merged into one thing: snapshots and backup restoration. I for one have never correlated snapshots and backup restorations (bare-metal recovery). I consider them completely separate things, and handled *very* differently. I have a feeling that no one's done this on FreeBSD because the amount of effort required is quite large. Someone did mention HAMMER on DragonflyBSD, but I have no knowledge of it or what it provides -- that said, Matt (Dillon)'s stuff is usually very, very good. > In addition to this, near continuous backups create less load on boxes > with a lot of reads but little writes. Standart backups have to scan all > the files to detect which files were changed. It depends on how the filesystem is done. For example, with UFS2+SU snapshots, snapshot generation can take literally hours: completely unreasonable. While with ZFS, snapshot generation usually takes 2-3 seconds -- even on massive changes (e.g. take a snapshot, then rm a 600MB ISO image, then compare present vs. snapshot -- the diff is something like 40KBytes). >> I'm sorry for sounding anti-FreeBSD, but the reality is that people >> should use whatever solutions work best for them -- if that's using >> Windows, Solaris, or Linux, great! Remember that open-source is about >> choice: and choice means supporting the possibility that someone chooses >> something else. Blind one-sided advocacy is very damaging to the >> open-source model and concept. > > I agree, and please dont shoot the messenger :) I just have a bunch of > customers who would use FreeBSD but not using only because of this > problem. In addition to that I myself would like to use near continuous > backups as well. Understood. I now realise the full importance of what it is you've described, and what R1Soft has developed. Thank you for taking the time to educate me -- I appreciate it! > I was just trying to inform the FreeBSD community here so if somebody > can have some time to divert to giving the right advices to r1soft then > we all could benefit from it. It doesnt even have to be free even, with > a reasonable price they can probably hire somebody to work for building > the basics of this feature. > > So the real question is, is there anybody who is willing and have the > experience to help on this issue? The response you're going to get is: "why don't you state how much you're willing to pay for this feature, so that anyone who IS interested can decide if that amount of money is worth the time required?" My response is different: this sort of thing should definitely be pawned off onto the FreeBSD Foundation. IMHO, this is the sort of thing the Foundation *should* be handling. There is money there, and this sounds like a project which could benefit FreeBSD as a whole. It's possible that R1Soft, if paid, would take up such a challenge, assuming some key folks (like Kirk McKusick; not volunteering him, just saying he has experience with filesystems) could help with the development process and learning curve. I can't speak for the Foundation, but it really sounds like that's the way to go with this. I don't think you'll get any responses from interested parties on freebsd-questions. :-) -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20081006175940.GB26368>