Date: Wed, 28 Nov 2001 04:23:47 -0600 From: Len Conrad <LConrad@Go2France.com> To: freebsd-isp@freebsd.org Subject: Re: Mail server (Sendmail) benchmarking Message-ID: <5.1.0.14.0.20011128030619.04d91bc8@mail.Go2France.com> In-Reply-To: <XFMail.011127161336.mark@work.drapple.com>
next in thread | previous in thread | raw e-mail | index | archive | help
>I'm nearing the deployment phase for a project I'm working on that will
>involve
>the sending of a very large number of relatively small (2k-5k) emails
>(probably
>like 500,000 or more daily).
500K copies of one msg body in different envelopes, or 500K unique
bodies? The difference is huge. My numbers below are for 1 msg body.
Can you string out delivery over 24 hours or does the client want all 500K
to go out with 1 hour? (one postfix admin had to delivery 100K perishable
"financial info" msgs in 1 hour, and used 10 postfix machines in parallel)
What is the source of injecting messages into the MTA? a list manager?
>I'm just curious if anyone has some estimations as to about how
>many emails I can reasonably expect one machine running FreeBSD 4.4-SECURE
>(or whatever that branch is being called now) with Sendmail to be able to
>send in a day.
I manage a FreeBSD4.4 + ecartis 1.0.0 + postfix "joke" co-lo list
server. 900 Mhz, 1 gb RAM, one ATA100 disk. I have ecartis queue chunking
at 100. The average msgs size is 20K. The critical softupdates is not
activated (co-lo monkey is a lazy bugger)
There a several lists sent every day, but two of them are about 5O+ K
members each. When these 2 lists are sent in different hours, delivery is
about 30K/hour. When these 2 lists are sent nearly simultaneously, the
hourly delivery rate flirts with 60K.
Here are the hourly rates for 27 NOV, for the two lists clearly sent at
different hours:
Per-Hour Traffic Summary
time received delivered deferred bounced rejected
--------------------------------------------------------------------
0000-0100 24 25 23 0 0
0100-0200 35 35 32 0 0
0200-0300 441 4860 89 34 1
0300-0400 1906 30527 1632 179 0
0400-0500 1163 17368 1970 163 1
0500-0600 134 351 552 29 0
0600-0700 86 158 347 0 0
0700-0800 81 303 352 2 0
0800-0900 105 139 117 2 0
0900-1000 183 323 144 1 0
1000-1100 2047 38412 1591 137 1
1100-1200 1314 23289 2245 35 0
1200-1300 170 358 1030 2 1
and here are the "msg queue times" for the Top 20 recipient domains:
Host/Domain Summary: Message Delivery (top 20)
sent cnt bytes defers avg dly max dly host/domain
-------- ------- ------- ------- ------- -----------
55623 1291m 186 32.7 s 34.1 m aol.com
17020 415467k 401 1.1 m 1.4 h hotmail.com
11075 261578k 0 21.0 s 3.2 m yahoo.com
3926 92451k 5547 1.1 h 5.7 h webtv.net
3924 47431k 0 4.5 s 15.8 m opt-in4email.com
2966 70305k 6 20.0 s 28.0 m cs.com
2735 63637k 578 3.2 m 2.0 h msn.com
2117 49463k 79 1.6 m 4.8 h home.com
899 21084k 0 11.7 s 36.0 s earthlink.net
837 20329k 3557 3.0 h 20.0 h excite.com
618 14843k 0 12.8 s 21.0 s juno.com
574 13775k 0 13.0 s 2.1 m bellsouth.net
471 11362k 0 13.1 s 33.0 s prodigy.net
410 9460k 0 15.7 s 1.7 m mediaone.net
402 9591k 0 13.5 s 1.9 m netscape.net
369 8606k 0 13.7 s 23.0 s worldnet.att.net
280 6655k 0 12.7 s 26.0 s sympatico.ca
259 5906k 0 10.8 s 18.0 s peoplepc.com
257 6076k 0 18.5 s 1.7 m gateway.net
237 5671k 0 17.6 s 1.5 m mindspring.com
... was a bad day for AOL, which usually has 0 defers and sub-20 second
avg. In contrast, note the typically crappy peformance of MS's MX's.
webtv and excite are so ludicrous, I don't bother to compare them with the
big boys who have the volumes and should have the resources to be the
best. I wonder what OS is run on the MX's of MS and AOL??
The above times are for how long a msg lived on disk in the mailqueue, from
completion of injection to completion of delivery.
I suspect a pretty good chunk of those seconds was queue wait time within
the machine, and not just SMTP-session time, because I've seen other
postfix gateways deliver 100's non-list mail to AOL, eg, with a
sub-6-second average. ie, our poor little single, ATA100 disk is
overwhelmed, but jokes can wait. :))
If you figure an average of 20 secs per delivery, for 500K msgs, that's 10
million delivery-seconds. If you had 1000 SMTP processes delivering in
parallel, that's 10K seconds start to finish, about 3 hours, which is
achievable with postfix.
So your bandwidth calculation is not 3 gb in 24 hours, but in 3 hours.
>The machine(s) I'm looking at will be P3-933Mhz with 1GB of RAM. My main
>question is whether I will need to throw more than one machine at this.
yes, no doubt about. one machine could do it, if you had the time to wait,
but if you client wants spiked delivery, you will need multiple machines,
say 2 to 4 primary machines.
>I've done some calculations, and 500,000 emails at 5k each is 2.5GB/day.
>Dividing that by 86400 (# of seconds/day) I get 28935 bytes/second, but I am
>promised by the client that I'll have the bandwidth I need, so I'm just mostly
>wondering if the machine will be the limiting factor
The machine will be, so go multi-box. of maybe your bandwidth will be, if
you want to deliver all 3 gb in one hour.
DNS: put BIND8 as caching-only on each box as "forward first" with one
external DNS box as a single forwarder. This will build a big, common
cache in the forwarder, which will be shared among the caches in the BINDs
of the MTAs.
disk:
1. run softupdates on the logging and mailqueue filesystems
2. 2 disks: one for logging, one for mailq
3. use fast disks with big caching, 160 Mb/sec SCSI, with 64 or 128 megs
of on-board caching. disk i/o for mailqueueing is the the limiting factor.
4. so, benchmark with md. or, with deep pockets, try SSD, solid state disk,
for the mailqueue disk. :))) Actually, one user on the postfix lists run
SSD to obtain highest possible disk bandwidth.
5. memory: 1 or 1.5 gb should be enough
6. SMTP/D processes. postfix prefers SMTPD processes receiving over SMTP
processes sending. When the message injection source is high-capacity,
SMTPD receiving will hog the disk bandwidth, such that SMTP sending will be
starved of disk access to the mailqueue, and msg delivery rate
suffers. Tough problem, and tedious to balannce by limiting the max number
of SMTPD processes so that some disk bandwidth is left for SMTP processes.
Wietse has recently added a new parameter that couples the receiving and
sending rates, the "in_flow_delay" param, so that when out_flow is sensed
as starved, you can set in_flow_delay to a number of seconds. This pauses
in_flow so out_flow can send, dynamically coupling in_flow with out_flow to
better share total disk bandwidth. if you weren't paying attention: disk
bandwidth is the limiting factor.
7. the key to delivery is 100's of SMTP processes (about 1 mb/process)
sending in parallel. So increase SMTP max from default of 50 to
500. STMPD of 50 should be enough to overwhelm the disk i/o. and run 2 or
3 primary outbound machines in parallel.
8. with 500K msgs, probably 5 or 6% will be dead addresses, so make sure
you use a list manager with automatic bounce unsubscribe
management. postfix helps here too, as it has a "fallback_relay" which
allows you a two-tier delivery scheme, where certain classes of msgs
undeliverable by the primary outbound relay will be dumped on the
fallback_relay machine, freeing the primary relay's deferred queue and disk
i/o for good deliverable addreses.
>forum, I can ask it in -questions but I thought the people in -isp would
>have more similar experience to what I'm asking.
best place is to ask in the users list for the MTA you plan to use.
Len
http://MenAndMice.com/DNS-training
http://BIND8NT.MEIway.com : ISC BIND 8.2.4 for NT4 & W2K
http://IMGate.MEIway.com : Build free, hi-perf, anti-abuse mail gateways
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5.1.0.14.0.20011128030619.04d91bc8>