From owner-freebsd-questions@FreeBSD.ORG Thu Sep 11 16:35:25 2003 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3DF8A16A4BF for ; Thu, 11 Sep 2003 16:35:25 -0700 (PDT) Received: from mygirlfriday.info (adsl-65-64-145-209.dsl.stlsmo.swbell.net [65.64.145.209]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1365843FBF for ; Thu, 11 Sep 2003 16:35:24 -0700 (PDT) (envelope-from gv-list-freebsdquestions@mygirlfriday.info) Received: (qmail 13540 invoked from network); 11 Sep 2003 23:35:22 -0000 Received: from user204.net795.mo.sprint-hsd.net (HELO mork) (65.41.216.204) by mongo.mygirlfriday.info with DES-CBC3-SHA encrypted SMTP; 11 Sep 2003 23:35:22 -0000 Date: Thu, 11 Sep 2003 18:35:21 -0500 From: gv-list-freebsdquestions@mygirlfriday.info X-Mailer: The Bat! (v2.00.6) Personal Organization: Hardly X-Priority: 3 (Normal) Message-ID: <1232387734.20030911183521@mygirlfriday.info> To: freebsd-questions@freebsd.org In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Re: `top` process memory usage: SIZE vs RES X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Sep 2003 23:35:25 -0000 Hello Jesse, Thursday, September 11, 2003, 5:15:31 PM, you wrote: J> I am stress testing a FreeBSD 4.7-RELEASE server (it's a pre- J> production test server) by sending huge email messages to it J> via SMTP. I too am putting together a 4.8 box as we speak for a client. J> I'm running qmail-1.03 built from source, with the QMAILQUEUE J> patch, qmail-qfilter, a custom Python script that runs under J> qmail-qfilter, and qmail-scanner with ClamAV. J> I test the server by sending a 59M or a 99M email from a J> remote machine (connected via fxp0). J> As I watch the email travel down the qmail-smtpd->qmail-qfilter-> custom-python->>qmail-scanner->clamd->qmail-queue "pipeline", J> I watch the memory usage with `top`. J> Memory is critical in this type of application, since I run my J> qmail-smtpd "pipeline" under DJB's softlimit program. I MUST J> know how much memory to allocate for the upper limit of each J> pipeline, otherwise qmail-smtpd will terminate the transfer J> with a 451 SMTP error. right. J> Anyway, as I watch `top`, I never see more than 15M being used J> by the various pipeline programs at any given point in time, J> but my Free Memory constantly declines until it reaches about J> 526k. J> The Questions: J> ========================== J> 1.) Where is my Free memory going? given what you say custom-python->>>qmail-scanner->clamd->qmail-queue This whole scenario is very memory intensive. First you have each email "pythonized" and then qmail-scanner is *very* memory intensive, as it has initially a very heavy duty perl script for each email before being passed off to clamd. Multiply this with the default SMTP concurrency limit of 20, and you have a lot of memory usage. Clamd is known to be a memory piggie too. I switched from that to using fprot which uses less memory and is faster processing. J> I can't account for it J> in the SIZE and RES columns of the various processes. J> These are relatively constant. Maybe running vmstat -w 1 would give you a different perspective also. -- Best regards, Gary