Date: Mon, 21 Jun 2004 22:56:57 -0400 From: Chuck Swiger <cswiger@mac.com> To: Bill Moran <wmoran@potentialtech.com> Cc: questions@freebsd.org Subject: Re: [OT] Re: What's the best possible email failover solution Message-ID: <40D79FF9.20308@mac.com> In-Reply-To: <20040621204111.6e684d45.wmoran@potentialtech.com> References: <20040621132006.2b1a296f.wmoran@potentialtech.com> <a22ff294040621115173bad2e0@mail.gmail.com> <20040621172520.3544d6fe.wmoran@potentialtech.com> <20040621214348.GB63857@happy-idiot-talk.infracaninophile.co.uk> <20040621175626.3e762448.wmoran@potentialtech.com> <40D76DA3.9090809@mac.com> <20040621204111.6e684d45.wmoran@potentialtech.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Bill Moran wrote: > Chuck Swiger <cswiger@mac.com> wrote: >>[ I don't think that stuffing email into a database is a particularly good >>idea since that means keeping large blobs of non-relational data floating >>around, something that the filesystem can do a better job of handling... ] [ ... ] > During my research of the IMAP protocol, I determined that _the_best_ > way to store email for high-performance would be to put them in a > database. This is because IMAP doesn't see email as a big blob of > text like POP does. It sees the headers as one thing, and the > different MIME parts of the email each as a seperate thing that can > be fetched independently of the other MIME parts. This is a pretty > good layout for a one -> many relationship in a database. Fact is, > every current IMAP server that I'm aware of has to break emails > apart on the fly in order to server IMAP. There's nothing wrong with applying database concepts to email, and it sounds like you want things which take advantage of database replication and transaction management and so forth in order to gain reliability, so perhaps you will find a DB better suited for your requirements than my comments above suggest. I don't mind being wrong when the result works better for someone. However, please remember that I know you are an optimist if you think I am a pessimist. :-) > Now, I could be wrong on this count, as I never wrote the mailserver, > so my theory could ultimately be proven wrong, but I guess I just > don't agree with the statement that SQL is a bad way to store email > until someone has actually proven it. My concern has less to do with the suitability of using a database to store mail as it has to do with database transactions becoming a potential bottleneck on the system as a whole. I've spent a great deal of time in my day job dealing with dynamic websites, which mostly means ones driven by content generated by a database. In my experience, you want to provide static content as efficiently as possible, and reserve database transactions for persisting changes to state and answering relational queries. The most relevant comparison is one involving a site where people can search for images by keyword, which someone was also storing in the database. The idea works fine under light to moderate load, but it turns out that keeping just the "relational" part of the image data (name, keywords, etc) and a filesystem reference, and generating a link using that path for Apache to serve directly scales much better. --- In the case of storing email in a DB, while you can break up a mail message into headers plus seperate MIME components, are you really going to want to decompose each and every mail message in a 3GB mail volume like that? Although if you throw enough RAM at a DB so that the entire thing fits into main memory, that can produce some spectacular results, and is almost doable for this specific case. Anyway, consider each time someone reads a message from the DB, you'd have to do two or three database transactions per message, maybe more, compared with read()ing or mmap()ing a single file in an IMAPD and doing strnstr()s for MIME boundary seperators in C. Remember that hitting the DB involves multiprocess IPC and adds a lot of latency compared to what a filesystem-based IMAP daemon does. -- -Chuck
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?40D79FF9.20308>