Date: Sun, 16 Feb 2003 03:02:52 +0100 From: Brad Knowles <brad.knowles@skynet.be> To: Terry Lambert <tlambert2@mindspring.com> Cc: Brad Knowles <brad.knowles@skynet.be>, Rahul Siddharthan <rsidd@online.fr>, freebsd-chat@freebsd.org Subject: Re: Email push and pull (was Re: matthew dillon) Message-ID: <a05200f02ba7492f1c777@[10.0.1.4]> In-Reply-To: <3E4DB5DE.D3EA1FE4@mindspring.com> References: <20030211032932.GA1253@papagena.rockefeller.edu> <a05200f2bba6e8fc03a0f@[10.0.1.2]> <3E498175.295FC389@mindspring.com> <a05200f37ba6f50bfc705@[10.0.1.2]> <3E49C2BC.F164F19A@mindspring.com> <a05200f43ba6fe1a9f4d8@[10.0.1.2]> <3E4A81A3.A8626F3D@mindspring.com> <a05200f4cba70710ad3f1@[10.0.1.2]> <3E4B11BA.A060AEFD@mindspring.com> <a05200f5bba7128081b43@[10.0.1.2]> <3E4BC32A.713AB0C4@mindspring.com> <a05200f07ba71ee8ee0b6@[10.0.1.2]> <3E4CB9A5.645EC9C@mindspring.com> <a05200f14ba72aae77b18@[10.0.1.2]> <3E4D7702.8EE51F54@mindspring.com> <a05200f01ba734065f08e@[10.0.1.4]> <3E4DB5DE.D3EA1FE4@mindspring.com>
next in thread | previous in thread | raw e-mail | index | archive | help
At 7:37 PM -0800 2003/02/14, Terry Lambert wrote:
> I like to build systems that are inherently scalable, so that you
> can just throw more resources at them.
The overall architecture that has been delivered should scale to
tens of thousands, hundreds of thousands, or millions of users. It's
the same architecture I delivered at LISA 2000.
The one thing that probably won't scale is the NFS server. But
that's not something I can change.
> You have to specify it on the NFS server system,
Do you know how to do that with NetApp filers?
> or you specify it
> on the NFS client system, so the transactions are not attempted,
Already tried that. However, mount_nfs complained that "noatime"
was not a recognized option.
> if
> you care about the request going over the wire and being ignored,
> rather than just being ignored. Since the issue you are personally
> fighting is write latency for requests you should probably not be
> making (8-)), you ought to work hard on this.
I agree that expensive operations should be avoided if at all
possible, but re-writing the entire OS, the entire NFS server code,
filesystem code, or other significant components of the system is not
an option. We have to deliver in a matter of just a very few weeks
-- we can't afford to take five years to make this happen.
> If your NFS client systems are FreeBSD, then it's a fairly minor
> hack to add the option the the NFS client code. If your NFS server
> is FreeBSD, then turning it off on the exported mount is probably
> sufficient.
>
> If neither is FreeBSD... well, can you switch to FreeBSD?
Regretfully not. Moreover, even if we could, it would only be on
the client. That only solves part of the problem, and perhaps not
the biggest part.
> Engineering.
How much? How many person-decades?
> It's not intended to "hurt them". It's only intended to deal with
> mutiple recipients for a single message. SPAM is almost the only
> type of mail that's externally generated that gets multiple recipient
> targets.
We get almost no spam. We do get some, and some people have
complained. The company took an internal survey and only something
like 10% of the respondents felt that the company should spend any
money to try to curb this problem, and most of the respondents felt
that they'd be perfectly happy using client-side features to solve
this issue.
> The point is not to "hurt" them (if you wanted that, you
> would run RBL or ORBS or SPEWS or ... and not accept connections from
> their servers in the first place), but to mitigate their effect on
> your storage costs.
Spam doesn't come in 44MB messages. The amount of disk storage
required for spam is almost nothing. My primary disk space problem
is friggin' morons in marketing or product research that seem to feel
that they have a God-given right to send the entire contents of their
hard drives via e-mail, or people illegally sending entire video
movies or illegal mp3s via e-mail.
The worst thing of all is when these idiots send this kind of
crap to someone and they are ignorant of where that person is
located. Many of the remote offices are behind very slow links (ISDN
or 56k), and have ancient mail/everything else servers that have
very, very little disk space for /var/spool/mqueue or /var/spool/mail.
When some bloody VP in the US insists on sending an 50MB e-mail
message on a daily basis to a guy in Asia (which is served through
the European hub, which we run), and the mail server the guy in Asia
doesn't even have 50MB free in /var/spool/mqueue on a good day, that
is a much, much bigger headache.
Trust me, disk space problems due to spam are the least of my worries.
> Note that this is the same philosophy you've been
> espousing all along, with quotas: you don't care if it causes a problem
> for your users, only if it causes a problem for you.
No, I do care. However, I have to be practical. Moreover, if
management has taken an internal survey on a subject and almost none
of the respondents support the idea of actually spending money to try
to solve the problem, then I can't really take this as a mandate to
go convince management that they have to delay the project (and other
projects depending on ours freeing up resources that they need).
> Internally, you have a higher connectedness between users, so you
> get much larger than your 1.3 multiplier, and for email lists, it's
> higher still.
They don't really have any mailing lists. I installed the
majordomo server a few months ago, and there has been relatively
little use of it. Part of that is because the current mail system is
so screwed up that it can only be used for one-way announcements, but
even after our replacement is online, I don't expect them to make all
that much use of mailing lists.
> In fact, I would go so far as to say that DJB's idea
> of sending a reference is applicable to email list messages, only
> the messages would be stored on the list server, instead of on the
> sender machine.
In essence, they do this already, and have been doing it for
quite some time. They have a very healthy community of internal
USENET news participants, and a very broad array of internal news
groups that they have set up. So, we don't need to re-invent the
entirety of Internet e-mail for them.
> At the point that you no longer care which machine you send a user
> connection to to retrieve their mail, then you no longer care where
> you send the mail,
No, that's precisely the problem. I *do* care where I send them
to get their mail, or where their mail is sent. But what I want to
do is to hide from them where this location is, because they *don't*
care (or shouldn't have to).
> or if the mail is single instance multiple
> time, a real replica, or a virtual replica (SIS). It takes a small
> amount of additional work.
I think the BS meter just broke.
Sorry, guy. Re-writing the entire mail system just to implement
SIS seems like the dumbest possible idea. Until you've done this for
UW-IMAP, Courier-IMAP, or Cyrus, and you're willing to release that
code back to the public, I don't think you've got a leg to stand on.
> The repeated mailing ("mail bombing") that started this thread is,
> or should be, simple to detect.
Could be. How would you do it? Statistical techniques to
discern how often a particular account normally receives mail based
on the last five minutes, the last hour, day, or comparing today
against a similar time period from yesterday, or last week, or last
month?
The cfengine technique does have a lot going for it, but for a
person who normally receives a lot of e-mail, there'd be a good
chance that it wouldn't detect a "mail bombing" of this sort. You'd
have to go into scanning the content of the message for that -- too
many messages in a given period of time that have essentially the
same body content, or whatever. But what period of time?
> Yes, it's a trivial case, but it's the most common case. You don't
> have to go to a compute-intensive technique to deal with it.
Once you start looking at the message body content, you've
already started down a slippery slope. You've told the user that you
will guarantee that they don't ever get mail-bombed, but as the
incoming messages are more and more diverse, at what point do you
decide that they really are different?
What can you say to the user when you've accidentally thrown away
a legitimate mail message because it seemed to be too similar to a
previous one that they had received?
> You are storing the reference wrong. Use an encapsulated reference,
> not a hard link.
Give me the file system to do that. Then port it to Solaris and
NetApp filer.
> That will permit the metadata operations to occur
> simultaneously, instead of constraining them to occur serially, like
> a link does.
Hell, if it can do all this kind of magic, just give it to me for
FreeBSD. Who needs softupdates, or dirprefs, or dirhash, or any of
that other crap, when you've got TerrysMagicFS?
> You keep saying this, and then you keep arranging the situation
> (order of operations, FS backing store, networ transport protocol,
> etc.) so that it's true, instead of trying to arrange them so it
> isn't.
No, the system is what it is. There are certain things that
cannot be changed. One of them is the NetApp filers for NFS.
Another is Solaris/SPARC. Another is that they absolutely,
positively, do not want *ANY* source code changes made whatsoever to
any of the programs or packages they are using.
They're already looking funny at me for specifying that we should
be using sendmail from sendmail.org as opposed to the vendor-supplied
sendmail which we actually commit the mortal sin of compiling
ourselves, or for actually going in and making source-code level
changes to procmail (which are required if you want to use Maildir
format mailboxes, or a hashed mailbox structure).
I am honestly looking for advice on how I can make this work as
well as I can, and you keep telling me that the only way to do
anything at all is to completely rewrite the entire OS of both the
client and the server.
Granted, I shouldn't be looking for help on non-FreeBSD systems
on a FreeBSD mailing list, but what you're giving me is not very
helpful.
If you want to tell me that I'm screwed and I shouldn't even
bother, that's fine. But don't waste my time and yours by continuing
to harangue me and stress how vitally important it is for me to
completely re-invent all the protocols being used, completely
re-write all the OSes and filesystems being used, etc....
> No, we are not. The transport protocols are the transport protocols,
> and you are constrained to implement to the transport protocols, no
> matter what else you do. But you are not constrained to depend on
> rename-based two phase commits (for example), if your FS or data
> store exports a transaction interface for use by applications: you
> can use that transaction interface instead.
At the ivory tower level, I'm sure that statement makes sense.
But don't tell me to eat cake when there isn't even any bread around.
> I have to say, I've personally delt with "help desk" escalations
> for problems like this, and it's incredibly labor intensive. You
> should always design as if you were going to have to deal with
> 100,000 customers or more, so that you put yourself in a position
> that manual processes will not scale, and then think about the
> problem.
They've got 6000 employees world-wide, and are already the
worlds-largest company in their niche. They were spun off years ago
by Philips, because they felt that this wasn't a core part of their
business. I don't think they're ever going to get particularly
large. And if they do, the architecture will scale, even if the
implementation won't.
On the other hand, I don't have to run or pay for the help desk
operation, and when it comes to quotas, not implementing SIS, not
re-writing the entire filesystem or OS for both the clients and the
server, etc..., well those aren't things that my customer has paid me
to do.
They have set out some pretty specific guidelines as to what they
want, and I have tried to give them the best possible architecture
and best possible implementation that I can, within those guidelines.
> Something simple like recognizing repetitive size/sender/subject
> pairing on the SMTP transit server.
And you would do that how? And you would make use of that information how?
> Ugh. Would you, as a user, bet your comapny on that level of service?
Sure. I've always tried to provide the same level of service
that I would want for myself, and I can be a very demanding customer.
Indeed, I am in the process of setting up my own consulting company,
and I will have a co-lo, where I run my own mail server. I won't be
going quite as far out for myself as I have for them, because my own
personal needs aren't quite so large. But I cannot strive to provide
myself any better level of service than I have done for them.
> 2.8. It's not like OpenLDAP, which needs the transactioning interfaces,
> it's pretty straight-forward code.
From the cyrus-2.0.14 distribution, doc/install-prerequisites.html:
* Berkeley DB, version 3.0.55 or higher. Berkeley DB can be obtained
from Sleepycat. It is strongly recommended that libsasl be compiled
with Berkeley DB support, using the same version of Berkeley DB. (If
you have a Berkeley DB version mismatch, somewhat perplexing crashes
result.)
From the cyrus-2.0.14 distribution, README:
The 2.0 release contains many new features over 1.6.
No further development work will progress on 1.6 so we
encourage you to work with the Cyrus 2.0 code wherever
possible. Support for 1.6 will also be extremely limited.
Sorry, guy. Not an option.
>> Certainly, when it comes to SAMS, all this stuff is pre-compiled
>> and you don't get the option of building Berkeley DB in a different
>> manner, etc....
>
> Yes, you end up having to compile things yourself.
With SAMS, you don't get that option. That's about the only
negative thing that I can say about it.
> If you have control over the clients, you can avoid making update
> requests. If you have no control over either, well, "Bad news, Clem".
I have no control over the clients.
> Though... you *could* allow any of the replicas to accept and queue
> on behalf of the primary, but then deliver only to the primary;
> presumably you'd be able to replace a primary in 7 days.
Doesn't help. The front-end servers will accept and queue on
behalf of the users. I don't need yet another level of queueing on
the back-end servers as well.
--
Brad Knowles, <brad.knowles@skynet.be>
"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
-Benjamin Franklin, Historical Review of Pennsylvania.
GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-chat" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a05200f02ba7492f1c777>