Date: Mon, 18 Dec 2000 12:03:19 -0800 (PST) From: Joseph Scott <joseph@randomnetworks.com> To: Dan Langille <dan@langille.org> Cc: freebsd-hackers@FreeBSD.ORG, freebsd-ports@FreeBSD.ORG Subject: Re: processing incoming mail messages (FreshPorts 2) Message-ID: <Pine.BSF.4.21.0012181157100.71411-100000@mothra.ecs.csus.edu> In-Reply-To: <200012181820.HAA18715@ducky.nz.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 19 Dec 2000, Dan Langille wrote: # > If you don't want to process a message the instant it comes in # > (via feeding it to a perl script or what ever) you'll need to setup some # > sort of queue, then have a cron job come through and process the queue. # # Thanks. Since posting my original message, the proposed design has # drifted to this strategy (which I posted to the FreshPorts develop list last # night): # # On Mon, 18 Dec 2000, Dan Langille wrote: # # > ok folks, brain pick time: I'm thinking about redoing the FreshPorts # > mail-database interface. At present, each incoming email from cvs-all # > initiates a separate perl script. Not nice if you get say 100 emails all # > in 1 minute for some reason. Now I'm thinking of using procmail to # > dump each email to a separate file on disk. Then processing each # > file thus freeing up the MTA ASAP. Sound good so far? # > # > Suggestions? Ideas? # # As for processing the files once they are on disk, I like the idea of # processing the messages ASAP. Therefore, I'd like to have a daemon / # script sitting there running all the time. This process is notified when a # new message arrives (in the above case, that would be part of the # procmail delivery process). This process would then "wake up" and # process all available message files one at a time. Perhaps moving each # processed file to an archive. When there are no more message files, it # would stop and wait to be notified again. # # > If the problem is load then another approach would be to heavily # > nice(1) the perl script the is launched when a commit mail comes in. # # That's already done. It's just the volume which can occur. If a large # number of messages arrive at once. Starting up 50 perl jobs on the box # can stress it a bit. It also makes better sense to process the # messages in the order in which they arrive rather than concurrently. A # side effect will be less change of database lockouts or conflicts during # updates. On item that may help on this then is checking two things every time you add something to the queue to help balance things out, the amount of time since the last time the queue was processed and the number of items in the queue. Let's say, at a minimum you want the queue to run every 30 minutes. However, if there are a large number of commits in the queue, you may want to be able ramp up to queue processing as quickly as every 5 minutes. If there's only two items in the queue though, there's really no reason to run it every 5 minutes. Just some thoughts. *********************************************************** * Joseph Scott The Office Of Water Programs * * joseph@randomnetworks.com joseph.scott@owp.csus.edu * *********************************************************** To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-ports" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.21.0012181157100.71411-100000>