From owner-freebsd-isp Sat Mar 22 5:16: 8 2003 Delivered-To: freebsd-isp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8F51137B401 for ; Sat, 22 Mar 2003 05:16:05 -0800 (PST) Received: from Shenton.org (23.ebbed1.client.atlantech.net [209.190.235.35]) by mx1.FreeBSD.org (Postfix) with SMTP id 107AC43F93 for ; Sat, 22 Mar 2003 05:16:04 -0800 (PST) (envelope-from chris@Shenton.Org) Received: (qmail 4846 invoked by uid 1000); 22 Mar 2003 13:16:02 -0000 To: Len Conrad Cc: freebsd-isp@freebsd.org Subject: Re: Maximum recommended user limits on mail server References: <5.2.0.9.0.20030319184158.02da2ba8@mail.go2france.com> From: Chris Shenton Date: 22 Mar 2003 08:16:02 -0500 In-Reply-To: <5.2.0.9.0.20030319184158.02da2ba8@mail.go2france.com> Message-ID: <87znnn5yp9.fsf@Pectopah.shenton.org> Lines: 90 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.2 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-isp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Len Conrad writes: > >A person I know is looking to have a mail server with 10K-15K > >accounts and 50 virtual domains. Shouldn't be a big deal so long as your architecture's sane. > You've got some good pointers on the mailbox servers, but nobody > talked about an multi-box architecture, which is the key to > scaleability and reliability. Absolutely agreed. If you've got a bunch of users, you can't just take the mailserver(s) down to upgrade some HW or SW. Design a system which has no single points of failure. I've learned this at a couple places the hard way and am doing it the "right" way on a project I'm working on. We'll be using a redundant pair of load balancers with a few 1U servers behind them offering SMTP, APOP (historical), IMAP, WebMail and an LDAP replica. Behind those will be a big NFS box (NetApp if I get the budget) and an LDAP master. With this, we'll be able to take down any of the data-less 1U servers if we need to, no interruption of service. To make this work, a shared store for mailboxes is key -- in this case NFS. There are a couple of catches tho. NFS does not always provide reliable locking so multiple writers can corrupt a file. I've also seen that UNIX mbox-style single-file-for-all-mail will lead to disk i/o starvation if mboxes get above 20MB or so -- check the qpopper list for lots of war stories. A solution to both problems is to use a mailstore format that does not put all of a user's messages in a single file. DJB's "Maildir" format does this -- each message gets its own file. No file locking issues, and stuff that's expensive on mbox (e.g., delete a message) is trivial and fast with Maildir. (There are other techniques if your mail servers do not share mailstore; use a directory to route incoming mail to the appropriate box for each users, and likewise proxy POP/IMAP connections. qmail-ldap can do this. You don't get the redundancy of the shared store approach, however). I'm using qmail + vpopmail + courier-imap + sqwebmail at one ISP. It does virtual domains very nicely and the mailstore is Maildir format. This small site uses a single box so has no redundancy, but with an NFS backend it could be fault-tolerant. I'm building another using qmail-ldap (and courier-imap and sqwebmail) and rather enjoying the flexibility of qmail using LDAP to lookup user address variations, mailbox location, quotas, active/disabled status, etc. I also find it rather fast. In some benchmarking I was doing yesterday, I ran 1000 sequential sessions each injecting 3 messages and saw a rate of 22 msg/sec -- on a P2-500 with 256MB RAM and an old IDE disk; this was fully delivered to local Maildir mailboxes, not sitting in a queue trying to be delivered. I was running this on FreeBSD-4.5 or so. I need to do benchmarks on POP/IMAP retrieval, since typically you'll have more reading (users) than writing (delivery to users). If you're religiously set on sendmail, you can use procmail or maildrop as a local mail delivery agent to deliver mail to Maildir format mailboxes. I've done this at one site, and used a maildir-aware POP daemon to serve users. Works OK but sendmail isn't nearly as fast as qmail. I understand Postfix can deliver to Maildir but have no experience with it. > 1. dedicated box as SMTP gateway. > 2. dedicated AV box > 3. mailbox server, muchly liberated because no anti-mail abuse, no DNS > query delay, no retrying of delayed mail, no AV scanning, and 50+% of > the inbound mail rejected at the envelope by 1. I may partition services on my 1U servers to act like the above. As long as I offer the services on each box, I can steer traffic to any I desire based on load balancer rules. (I might have more POP/IMAP services than SMTP, for example). If you don't have a balancer, you can leverage DNS MX for SMTP delivery and DNS round-robin for POP/IMAP access (but you'll have to manually tweak DNS if you take a machine out of POP/IMAP service). I haven't yet investigated anti-spam/anti-virus stuff except to look at Postini and Brightmail services -- pricey but can be cost-justified at corporate sites. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message