From owner-freebsd-questions@FreeBSD.ORG Tue Jun 22 01:24:58 2004 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BE4C116A4CE for ; Tue, 22 Jun 2004 01:24:58 +0000 (GMT) Received: from internet.potentialtech.com (h-66-167-251-6.phlapafg.covad.net [66.167.251.6]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6334E43D1D for ; Tue, 22 Jun 2004 01:24:58 +0000 (GMT) (envelope-from wmoran@potentialtech.com) Received: from working.potentialtech.com (pa-plum1c-102.pit.adelphia.net [24.53.179.102]) by internet.potentialtech.com (Postfix) with ESMTP id 48CC669A39; Mon, 21 Jun 2004 21:24:56 -0400 (EDT) Date: Mon, 21 Jun 2004 21:24:55 -0400 From: Bill Moran To: Chuck Swiger Message-Id: <20040621212455.44310df1.wmoran@potentialtech.com> In-Reply-To: <40D785A6.1040408@mac.com> References: <20040621132006.2b1a296f.wmoran@potentialtech.com> <20040621172520.3544d6fe.wmoran@potentialtech.com> <20040621214348.GB63857@happy-idiot-talk.infracaninophile.co.uk> <20040621175626.3e762448.wmoran@potentialtech.com> <40D76DA3.9090809@mac.com> <20040621203245.1f0e7444.wmoran@potentialtech.com> <40D785A6.1040408@mac.com> Organization: Potential Technologies X-Mailer: Sylpheed version 0.9.10 (GTK+ 1.2.10; i386-portbld-freebsd4.9) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit cc: questions@freebsd.org Subject: [OT] Re: What's the best possible email failover solution X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jun 2004 01:24:58 -0000 Chuck Swiger wrote: > Bill Moran wrote: > > Chuck Swiger wrote: > [ ... ] > >> The latter uses one-message-per-file, and ought to work *much* better both in > >> terms of performance and stability, and in terms of playing nice with the way > >> rsync wants to back things up. > > > > Doesn't really matter. Fact is, the mail directories are something on the order > > of 3G. No matter how efficiently I store them, rsync is not going to be able > > to back them up fast enough to hit the level of redundancy I'm shooting for. > > You may well be right, as you aren't really talking about performing backups, > you're talking about creating a fully redundant storage which is kept > up-to-date in realtime. > > > Although Maildirs might work a little better, since I wouldn't have to stop > > the IMAP server during backup. > > That, and the granularity of one-message-per-file fits perfectly with rsync's > file-driven model. I don't know about that. Last I checked, many files resulted in rsync taking a lot of time, and a lot of memory to build a file list. I can't say whether the end result is more efficient or not, as I've never tested it, but rsync does intelligently move portions of files, instead of the whole file, when changes occur. > > It takes about 30 minutes to rsync the system to the backup server right now. > > That's perfectly acceptable for nightly backup purposes. This is a 1.5Ghz > > with 256M RAM and 80G ATA 100 HDDs. If the system runs rysnc continuously > > 24/7, I still have 30mins old data. > > Oh, yes. Just don't forget that if you do eliminate this time gap, you still > ought to have another system actually taking backups. Any change the system > encounters will be replicated to the redundant mail storage system in real > time, including bad changes. Hehe ... I've been trying to explain that to some of my less intelligent clients for a while now ... "Yes, when you do something wrong, it backs that up as well ..." Actually had a client ask me once why the backup didn't know the difference between something it should be backing up, and something that shouldn't be backed up. I told him if I knew how to do that, I'd be a lot richer! > >>[ I don't think that stuffing email into a database is a particularly good > >>idea since that means keeping large blobs of non-relational data floating > >>around, something that the filesystem can do a better job of handling... ] > > > > It's a good idea if I want real-time redundancy. I see where you're coming > > from, and it's true that a RDBMS isn't the best way to store emails. But, > > when you look at the features available, it's the best way for this > > circumstance. With something like Slony, I'd have real-time redundancy > > with (I'm expecting) only a minor performance drop. Although I can't be > > sure until I can put something together to test. Reliability is much more > > important than performance in this case. Who cares if their email takes > > and extra 60 seconds to deliver, as long as it doesn't get lost! If the > > email arrives fast, it's useless if the server fails and the email is > > lost because the SMTP server told the delivering server that it had > > arrived and then crashed before it could be backed up. > > I suspect that the relatively heavy weight of database transactions compared > with filesystem access is going to slow things down a fair amount, too, > particularly when running against a replicated DB. But reliability over > performance is a fine choice to make. :-) > > Using RAID improves fault-tolerance, but you still end up with a > single-point-of-failure at the system level; using database replication gives > you higher availability, which seems to be what you mean when you talk about > "reliability". Perhaps SAN or NAS concepts might be worth considering, as you > can set up a fully-redundant fibre channel configuration where the storage is > shared between two or more systems, thus with no single-point-of-failure. Problem there is cost ... that hardware is a bit pricey, compared to PC hardware, anyway. -- Bill Moran Potential Technologies http://www.potentialtech.com