From owner-freebsd-hackers Mon Dec 18 10:21: 2 2000 From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 18 10:20:55 2000 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from ducky.nz.freebsd.org (ns1.unixathome.org [203.79.82.27]) by hub.freebsd.org (Postfix) with ESMTP id 1500437B400; Mon, 18 Dec 2000 10:20:54 -0800 (PST) Received: from wocker (wocker.int.nz.freebsd.org [192.168.0.99]) by ducky.nz.freebsd.org (8.9.3/8.9.3) with ESMTP id HAA18715; Tue, 19 Dec 2000 07:20:36 +1300 (NZDT) Message-Id: <200012181820.HAA18715@ducky.nz.freebsd.org> From: "Dan Langille" Organization: langille.org To: Joseph Scott Date: Tue, 19 Dec 2000 07:21:00 +1300 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: 7BIT Subject: Re: processing incoming mail messages (FreshPorts 2) Reply-To: dan@langille.org Cc: freebsd-hackers@FreeBSD.ORG, freebsd-ports@FreeBSD.ORG Priority: normal References: <200012161822.HAA03654@ducky.nz.freebsd.org> In-reply-to: X-mailer: Pegasus Mail for Win32 (v3.12c) Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On 18 Dec 2000, at 11:22, Joseph Scott wrote: > > On Sun, 17 Dec 2000, Dan Langille wrote: > > # Which list would be more appropriate for asking advice on designing a > # mail processing strategy for FreshPorts 2 (i.e. processing all of > # cvs-all, not just the ports)? > # > # I'm looking for recommendations and guidance on how to capture the > # incoming messages and process them one at a time. As opposed to > # starting a separate perl script for each message (which is the the > # existing strategy and is usually fine, except when large numbers of > # messages turn up in a short period of time). > > If you don't want to process a message the instant it comes in > (via feeding it to a perl script or what ever) you'll need to setup some > sort of queue, then have a cron job come through and process the queue. Thanks. Since posting my original message, the proposed design has drifted to this strategy (which I posted to the FreshPorts develop list last night): On Mon, 18 Dec 2000, Dan Langille wrote: > ok folks, brain pick time: I'm thinking about redoing the FreshPorts > mail-database interface. At present, each incoming email from cvs-all > initiates a separate perl script. Not nice if you get say 100 emails all > in 1 minute for some reason. Now I'm thinking of using procmail to > dump each email to a separate file on disk. Then processing each > file thus freeing up the MTA ASAP. Sound good so far? > > Suggestions? Ideas? As for processing the files once they are on disk, I like the idea of processing the messages ASAP. Therefore, I'd like to have a daemon / script sitting there running all the time. This process is notified when a new message arrives (in the above case, that would be part of the procmail delivery process). This process would then "wake up" and process all available message files one at a time. Perhaps moving each processed file to an archive. When there are no more message files, it would stop and wait to be notified again. > If the problem is load then another approach would be to heavily > nice(1) the perl script the is launched when a commit mail comes in. That's already done. It's just the volume which can occur. If a large number of messages arrive at once. Starting up 50 perl jobs on the box can stress it a bit. It also makes better sense to process the messages in the order in which they arrive rather than concurrently. A side effect will be less change of database lockouts or conflicts during updates. cheers -- Dan Langille The FreeBSD Diary - http://www.freebsddiary.org/ NZ ADSL - http://www.unixathome.org/adsl/ NZ Broadband - http://www.unixathome.org/broadband/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message