From owner-freebsd-questions@FreeBSD.ORG Tue May 23 12:14:52 2006 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 39E5D16A44B for ; Tue, 23 May 2006 12:14:52 +0000 (UTC) (envelope-from dave.list@pixelhammer.com) Received: from ecluster4.tls.net (ecluster4.tls.net [65.196.224.134]) by mx1.FreeBSD.org (Postfix) with SMTP id 02C3443D58 for ; Tue, 23 May 2006 12:14:50 +0000 (GMT) (envelope-from dave.list@pixelhammer.com) Received: (qmail 99641 invoked by uid 89); 23 May 2006 12:14:49 -0000 Received: from 204-8-13-26.bb.hrtc.net (HELO ?192.168.0.103?) (ldg%tls.net@204.8.13.26) by auth-ecluster4.tls.net with SMTP; 23 May 2006 12:14:49 -0000 Message-ID: <4472FC9F.90201@pixelhammer.com> Date: Tue, 23 May 2006 08:14:23 -0400 From: DAve User-Agent: Thunderbird 1.5.0.2 (Windows/20060308) MIME-Version: 1.0 To: FreeBSD Questions Mailing List References: <4471ABF0.3090804@ispro.net.tr> <6.0.0.22.2.20060522102107.0274be28@mail.computinginnovations.com> <4471ECAA.3030406@daleco.biz> <20060522231641.7d63db65@vixen42.vulpes> <4472BB57.7020001@ispro.net.tr> <76921773-B1C7-4500-8FE7-78B815961860@shire.net> <4472C73C.9040501@ispro.net.tr> In-Reply-To: <4472C73C.9040501@ispro.net.tr> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: e-mail server farm question X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 May 2006 12:14:54 -0000 Evren Yurtesen wrote: > Chad Leigh -- Shire.Net LLC wrote: > >> >> On May 23, 2006, at 1:35 AM, Evren Yurtesen wrote: >> >>> Vulpes Velox wrote: >>> >>>> On Mon, 22 May 2006 11:54:02 -0500 >>>> Kevin Kinsey wrote: >>>> >>>>>> At 07:17 AM 5/22/2006, Evren Yurtesen wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I was wondering how does services like yahoo mail is storing >>>>>>> e-mails. Somehow the smtp server should know where to deliver >>>>>>> the mail inside the system and webmail should know from which >>>>>>> server to read it from. >>>>>>> >>>>>>> Does anybody have any practical ideas about how it is done? >>>>>>> >>>>> >>>>> Derek Ragona wrote: >>>>> > If you are using sendmail, as most FreeBSD users are, you can >>>>> > check the sendmail.org site for information on mail handling. >>>>> > There are a number of methods that depend on your setup. >>>>> > >>>>> >>>>> Well, it's pretty obvious that they aren't using a stock >>>>> SendMail: >>>>> >>>>> # telnet mx2.mail.yahoo.com 25 >>>>> Trying 67.28.113.72... >>>>> Connected to mx2.mail.yahoo.com. >>>>> Escape character is '^]'. >>>>> 220 mta309.mail.re4.yahoo.com ESMTP YSmtp service ready >>>>> >>>>> Short of finding an article written by someone 'in the know', >>>>> or an answer for someone like that, we can only guess. I'd >>>>> probably start with guessing a big DB on a large SAN; >>>>> which pretty much negates the "which server to read from" >>>>> question (up to a point). Everything else is pretty >>>>> academic. SMTP, IMAP, POP. >>>> >>>> Maildir makes it easy to distribute it across multiple machines as >>>> well. >>> >>> >>> What do you mean exactly? distributing 1 user's mails into seperate >>> machines? I didnt understand how Maildir helps to this actually. >>> >> >> I am not sure anyone was talking about distributing 1 person's mail >> across separate machines. The discussion seemed to be how to handle >> large amounts of mail spread out across machines, which maildir helps >> with as you can have one or more file servers and lots of consumers >> (imap/pop) and deliverers (mta) accessing those maildirs on your file >> servers. Combine with a backend database of some sort (we use an >> ldap db that includes the path for a specific accounts mail) and voilá. >> >> Chad >> > > Ah sorry, I didnt think it that way for a moment. I thought you meant > Maildir > stores mails in seperate files compred to mbox format used by sendmail > so...anyhow > my mistake :) But it is possible to make changes to sendmail so that it > will store to different folders also. > > I think the conclusion is a database, multiple smtp servers querying > database > to see where to forward received e-mails, multiple pop3/imap servers > querying > database to see from where to read the e-mails and multiple storage > machines. > This way it can scale to an unlimited size. > > So it requires a lot of coding :) > Not really, we have a large system which required little coding except for script tools. We have two FreeBSD servers as public gateways running Sendmail + Milter-Ahead + MailScanner. These 'gateways' scan mail for viruses and then forward the delivery on to three toasters which are FreeBSD servers running qmail + vpopmail + Spamd + SquirrelMail. The toasters each mount an NFS share from a Sun Enterprise to store the mail. Vpopmail answers the validation reqests from Milter-Ahead and gets all it's storage/authentication information such as Maildir delivery, forwarding, SpamAssassin settings, etc from a common MySQL DB. All very stock and the system can grow as large as the mail store server allows. When it is incapable, we will replace it with a larger machine. Just because I know you will ask, the mail store server is raid5, dual power supply, dual nic. The gateways, toasters, and mail store all communicate via a private network which is 1gb. It has proven very robust during the past two years. DAve -- This message was checked by forty monkeys and found to not contain any SPAM whatsoever. Your monkeys may vary