From owner-freebsd-isp Thu Mar 11 22:56:56 1999 Delivered-To: freebsd-isp@freebsd.org Received: from vnode.vmunix.com (vnode.vmunix.com [209.112.4.20]) by hub.freebsd.org (Postfix) with ESMTP id 82F5F14F8C; Thu, 11 Mar 1999 22:56:53 -0800 (PST) (envelope-from mark@vnode.vmunix.com) Received: (from mark@localhost) by vnode.vmunix.com (8.8.8/8.8.8) id BAA07089; Fri, 12 Mar 1999 01:56:34 -0500 (EST) (envelope-from mark) Message-ID: <19990312015634.B6746@vmunix.com> Date: Fri, 12 Mar 1999 01:56:34 -0500 From: Mark Mayo To: Gary Palmer Cc: freebsd-isp@FreeBSD.ORG Subject: Re: Mail server setup References: <19990311194422.A4381@vmunix.com> <89008.921212284@gjp.erols.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.93.1i In-Reply-To: <89008.921212284@gjp.erols.com>; from Gary Palmer on Thu, Mar 11, 1999 at 11:18:04PM -0500 X-Operating-System: FreeBSD 2.2.7-STABLE i386 Sender: owner-freebsd-isp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Thu, Mar 11, 1999 at 11:18:04PM -0500, Gary Palmer wrote: > Mark Mayo wrote in message ID > <19990311194422.A4381@vmunix.com>: > > done it..). That gets you nice horizontal scaling to infinity pretty > > much, but doesn't help you out in terms of availability or serviceability. > > How do you do an OS upgrade on a front end host with no cx impact? What > > happens if a host goes down? What happens if your disk array goes tits up? > > However you do it, you end up with the same problem. Whereever the mail is > stored becomes a SPOF (Single Point Of Failure), unless you go to some sort of > HA scenario, and none of the HA products that I've seen are any where close to > being `mature' enough for my liking. Well that's the real trick, now isn't it.. :) I'm at the point where I'm trying to decide if the HA solutions out there can really do what they claim.. I've done heavy testing on the NetApp clustering stuff in the last month, and to be honest, I've trashed "heads" in just about every way I can imagine, and the clustered failover kicked in and worked 9 times out of 10, which for me is "good enough" for an ISP email environment. > > I'm not saying I like NFS.. but when done correctly (use a real RDBMS for > > most of the POP stuff to eliminate trips to the NFS server for mail checks, > > avoid locking over NFS, failover NFS servers listed for clients, etc.) > > you can build a decent NFS based solution that scales and is very tolerant > > to host / filer crashes. I'd do it with local disk if I could, but I haven't > > been able to come up with a solution based on local disk that I'm satisfied w > > ith.. > > Because of software reasons or hardware reasons? Both. I'm not that impressed with much of the hardware out there, and in my case, the hardware has to work with Sun boxen. And work perfectly. In terms of software, I don't see the flexibility or availability I would like with host based storage. I suppose ultimately you could sum up my thoughts as "I like the idea of network accessable storage - I just wish there was something better than NFS to do it with! Oh well, guess I'll have to use NFS, but do the application so that it actually works..". 8-) > To be perfectly honest, to my > mind, the lack performance in NFS writes it off to my mind instantly. I can > get high performance out of a host based backend with a fraction of the work > of what you propose in the above paragraph. Agreed. My point, however, was that I'm not looking for simply high performance. High performance, to be perfectly honest, is trivial to implement. Balancing high performance, a reasonable price, and high availability in one bundle is where the fun starts. :) As for NFS performance, you'd be surprised what a NetApp F760 with Gigabit ethernet can do. It ain't cheap, but it sure as hell ain't slow. Seeing how this is a FreeBSD list, I'll keep this relevant by mentioning that how I would approach a mail system on FreeBSD differs tremendously from how I would do it with Solaris. Why? 1. Sun UFS is slow as hell. Sun NFS generally works well, and is reasonably quick. 2. FreeBSD NFS is not usable in a high performance mail environment. FreeBSD SoftUpdates is fast as blazing hell. Moving from softupdates on local disk to NFS on a FreeBSD system is a MAJOR, MAJOR performance hit. Moving from Sun's crappy UFS to NFS on a Solaris box in many cases is a performance increase, not decrease. :) > > Of course, that probably has just as much to do with my hatred of Sun's > > ludicrously priced A5000 disk arrays I'm using right now as it does with > > anything else.. ;) That and the fact that I'm happy as pie with my NetApp > > F760 test units, and F740 production filers.. > > Anyone who buys storage direct from Sun needs their head looked at. The A3500 Fortunately I can save face in this scenario and say that I walked into my current situation well after a room full of A5000's were purchased.. :( I've never seen such an over-hyped piece of sh@! for storage. The only real world application I can imagine using it for would be streaming enormous files from array to array, or sucking a huge data set through an E10000 for processing. In other words, there probably are no real world applications that the A5000 is well suited for. :-) Not to mention it often takes Sun *days* to trouble shoot a problem.. It's very encouraging to see companies like DPT offering very functional RAID systems that can work on FreeBSD. Used to be you were stuck with expensive, propriatary solutions from Sun, Auspex, EMC and such if you wanted to play that game. I'm very excited about real disk solutions for FreeBSD. :) Then I can stop using Sun for more things. :) > > Interesting how Sun started making such statements shortly after they > > started pushing SIMS - which doesn't run over NFS reliably. Coincidence? :-) > > SIMS is close to the right architecture IMHO. Its just way too expensive and > large-scale to replace NFS /var/mail in a lot of instances. But thats another > story. I'm only interested in large-scale ISP mail. Solutions for less than 1/4 million mailboxes are numerous, and /var/mail type environments are, as you say, a whole different ball of wax. SIMS does have a good architecure, it's just stupidly priced, like so many of Sun's products. > If you go to the lengths you describe above, and have the RDBMS database, yada > yada yada, then yes, NFS may actually scale and perform. However, thats a lot > of work, which I don't feel is justified when the proxied-to-large-backend > solution works just as well. It's not much work at all, really. You just can't download it in a conveniently packaged .tgz file -- yet. I'm sure in a year or two, this sort of stuff will reach the public domain. At any rate, there are more and more solutions coming out these days. From procmail delivery, to qmail's maildir, to exim's format, NFS safe mailbox formats are finally available. With just a touch more work, you can take this approach to a higher performance, more redundant systems that works better over NFS *and* local disk. That's a good thing. :) -Mark > > Gary > -- > Gary Palmer FreeBSD Core Team Member > FreeBSD: Turning PC's into workstations. See http://www.FreeBSD.ORG/ for info > -- ------------------------------------------------------------------------ Mark Mayo mark@vmunix.com RingZero Comp. http://www.vmunix.com/mark ------------------------------------------------------------------------ "The human brain is like an enormous fish - it is flat and slimy and has gills through which it can see." -- Monty Python To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message