Date: Tue, 22 Jun 2004 08:47:26 -0400 From: Bill Moran <wmoran@potentialtech.com> To: Chuck Swiger <cswiger@mac.com> Cc: questions@freebsd.org Subject: Re: [OT] Re: What's the best possible email failover solution Message-ID: <20040622084726.524bfa39.wmoran@potentialtech.com> In-Reply-To: <40D79FF9.20308@mac.com> References: <20040621132006.2b1a296f.wmoran@potentialtech.com> <a22ff294040621115173bad2e0@mail.gmail.com> <20040621172520.3544d6fe.wmoran@potentialtech.com> <20040621214348.GB63857@happy-idiot-talk.infracaninophile.co.uk> <20040621175626.3e762448.wmoran@potentialtech.com> <40D76DA3.9090809@mac.com> <20040621204111.6e684d45.wmoran@potentialtech.com> <40D79FF9.20308@mac.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Chuck Swiger <cswiger@mac.com> wrote: > Bill Moran wrote: > > Chuck Swiger <cswiger@mac.com> wrote: > >>[ I don't think that stuffing email into a database is a particularly good > >>idea since that means keeping large blobs of non-relational data floating > >>around, something that the filesystem can do a better job of handling... ] > [ ... ] > > During my research of the IMAP protocol, I determined that _the_best_ > > way to store email for high-performance would be to put them in a > > database. This is because IMAP doesn't see email as a big blob of > > text like POP does. It sees the headers as one thing, and the > > different MIME parts of the email each as a seperate thing that can > > be fetched independently of the other MIME parts. This is a pretty > > good layout for a one -> many relationship in a database. Fact is, > > every current IMAP server that I'm aware of has to break emails > > apart on the fly in order to server IMAP. > > There's nothing wrong with applying database concepts to email, and it sounds > like you want things which take advantage of database replication and > transaction management and so forth in order to gain reliability, so perhaps > you will find a DB better suited for your requirements than my comments above > suggest. > > I don't mind being wrong when the result works better for someone. However, > please remember that I know you are an optimist if you think I am a pessimist. > > :-) > > > Now, I could be wrong on this count, as I never wrote the mailserver, > > so my theory could ultimately be proven wrong, but I guess I just > > don't agree with the statement that SQL is a bad way to store email > > until someone has actually proven it. > > My concern has less to do with the suitability of using a database to store > mail as it has to do with database transactions becoming a potential > bottleneck on the system as a whole. Agreed ... and (now that I'm thinking about it) that's why I decided to write a mail server. IIRC, I started researching IMAP, and started wondering how well it would work with a database back end, so I decided to write one to see how _well_ it worked ... then reality intervened and I didn't get to have any fun ... <snip> > In the case of storing email in a DB, while you can break up a mail message > into headers plus seperate MIME components, are you really going to want to > decompose each and every mail message in a 3GB mail volume like that? > Although if you throw enough RAM at a DB so that the entire thing fits into > main memory, that can produce some spectacular results, and is almost doable > for this specific case. > > Anyway, consider each time someone reads a message from the DB, you'd have to > do two or three database transactions per message, maybe more, compared with > read()ing or mmap()ing a single file in an IMAPD and doing strnstr()s for MIME > boundary seperators in C. Remember that hitting the DB involves multiprocess > IPC and adds a lot of latency compared to what a filesystem-based IMAP daemon > does. Possibly. But most IMAP clients cache a lot of stuff, so it'll only grab most things once. The other advantages is it would scale like nobody's business. Since the data is in postgres, you could use multiple backends (replicated with Slony) and have the IMAP daemons contact different back ends if the load got heavy. With a little work, the system could failover silently as well. Like I said, we'll never know till someone tries it. It looks like Dovecot is going to try it eventually, but it seems like they have other priorities at this time. -- Bill Moran Potential Technologies http://www.potentialtech.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040622084726.524bfa39.wmoran>