From owner-freebsd-amd64@FreeBSD.ORG Thu May 3 22:28:53 2007 Return-Path: X-Original-To: freebsd-amd64@freebsd.org Delivered-To: freebsd-amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1D48E16A400 for ; Thu, 3 May 2007 22:28:53 +0000 (UTC) (envelope-from o.greve@axis.nl) Received: from smtp.interstroom.nl (smtp1.interstroom.nl [80.85.129.3]) by mx1.freebsd.org (Postfix) with ESMTP id 6688713C455 for ; Thu, 3 May 2007 22:28:52 +0000 (UTC) (envelope-from o.greve@axis.nl) Received: from 5571fe44.ftth.concepts.nl ([85.113.254.68]:50600 helo=[192.168.1.102]) by smtp.interstroom.nl with esmtpsa (TLS-1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.63) (envelope-from ) id 1HjjaE-0003Rz-2W; Fri, 04 May 2007 00:15:33 +0200 Mime-Version: 1.0 (Apple Message framework v752.3) Message-Id: <2BEB30C2-C9C5-43AB-9DCA-5C9A1B0AC2C0@axis.nl> To: freebsd-questions@freebsd.org, freebsd-amd64@freebsd.org From: Olaf Greve Date: Fri, 4 May 2007 00:15:30 +0200 X-Mailer: Apple Mail (2.752.3) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: How to make Apache (2.2.4) less greedy, or Sendmail less polite? X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 May 2007 22:28:53 -0000 Hi, Recently I upgraded my Apache 1.3.33 webserver to Apache 2.2.4, and ever since, I noticed that it is acting in such a way that it often is VERY greedy with my server's resources. Quite often, when running "top", a list that is as the one that appears at the bottom of this e-mail is shown: indeed pretty much solely httpd instances, that for extended periods of time almost continously pull the CPU to close to 100%, and that also consume a lot of the memory resources... Strangely enough, at other times the CPU load is just slightly above 0%, say 0.4% or so... Apart from the fact that it "doesn't feel right" to see the CPU for substantial amounts of time, almost constantly close to 100%, there is a further issue, being that sendmail rejects connections when the server load is (too) high. This is very annoying, as e-mail is also a crucial part of the server's functionality, and I don't want sendmail to reject connections, each and every time that Apache goes berserk. Now, the machine in question, is an AMD-64 machine, and it runs the AMD-64 version of FreeBSD (5.4-release) with a custom kernel. Surely, Apache can be reconfigured such that it doesn't behave so selfishly, and leaves a decent amount of resources for other stuff (such as sendmail) on the machine too. What I'm basically trying to find out is: 1-Is this normal, or can this perhaps be some (brute force) hack attempt, where something is pounding Apache heavily, trying to find/ exploit some security risk? 2-How can I inspect exactly what each httpd instance is doing (i.e. which request it is serving)? 3-How to best configure Apache 2.2.4 such that it will never use more than a specific amount of the system's resources (e.g. a CPU usage limit of 75%, and a memory limit of say 1GB)? It would be my guess that the amount of "MaxClients" should be lowered, but is that sufficient (note: current httpd-mpm.conf settings apper at the end of this e-mail, and indicate an amount of 150), and will that not somehow (all too) negatively affect the way Apache handles requests? 4-How to perhaps tell sendmail to be a bit more selfish, and stop it from rejecting connections for extended periods of time? (note: we all know just how much "fun" it can be to configure Sendmail :P so for now I've only included (a shortened version of the) RX daemon config file, and hope someone can give me a good pointer for this - or tell me where else to look). 5-When sendmail rejects (incoming) connections, does mail actually get lost, or will it (always) be handled later, when the server is less occupied? Cheers, and tnx in advance! Olafo PS: I hope anyone can give me some good ideas, and for completeness sake, I've copied some additional information that may give an insight into the issues: 1) The Sendmail "rejecting connections" issue: ps auxww | grep sendmail root 2259 0.0 0.0 9480 668 ?? Ss 20Apr07 0:38.17 sendmail: rejecting connections on daemon MSA: load average: 59 (sendmail) smmsp 2261 0.0 0.0 13628 760 ?? S 20Apr07 1:40.56 sendmail: running queue: /var/spool/mqueue-rx (sendmail) root 2262 0.0 0.0 9480 704 ?? Ss 20Apr07 0:37.85 sendmail: accepting connections (sendmail) smmsp 2265 0.0 0.0 9344 608 ?? Is 20Apr07 0:01.33 sendmail: Queue runner@00:10:00 for /var/spool/clientmqueue (sendmail) root 91503 0.0 0.0 428 320 p0 D+ 7:23PM 0:00.00 grep sendmail 2) "top" output (partial), during (apparent) heavy load: last pid: 91504; load averages: 58.76, 59.21, 60.20 up 13+07:02:40 19:24:50 163 processes: 61 running, 102 sleeping CPU states: 98.8% user, 0.0% nice, 0.4% system, 0.8% interrupt, 0.0% idle Mem: 1299M Active, 204M Inact, 289M Wired, 63M Cache, 214M Buf, 39M Free Swap: 2021M Total, 922M Used, 1099M Free, 45% Inuse, 128K In PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 91459 www 124 0 141M 15136K RUN 0:02 5.52% 5.52% httpd 91352 www 119 0 139M 12596K select 0:14 3.61% 3.61% httpd 91455 www 124 0 167M 41960K RUN 0:03 3.61% 3.61% httpd 91461 www 124 0 141M 15128K RUN 0:03 1.37% 1.37% httpd 91126 www 124 0 158M 19520K RUN 1:46 0.83% 0.83% httpd 91139 www 124 0 158M 19532K RUN 1:43 0.83% 0.83% httpd 91152 www 124 0 195M 19396K RUN 1:40 0.83% 0.83% httpd 91175 www 124 0 170M 44524K RUN 1:02 0.83% 0.83% httpd 90387 www 124 0 170M 27548K RUN 5:19 0.78% 0.78% httpd 90529 www 124 0 195M 24584K RUN 4:49 0.78% 0.78% httpd 90665 www 124 0 167M 41804K RUN 3:29 0.78% 0.78% httpd 90897 www 124 0 181M 23964K RUN 2:10 0.78% 0.78% httpd 91069 www 124 0 158M 20128K RUN 1:53 0.78% 0.78% httpd 91100 www 124 0 162M 21284K RUN 1:49 0.78% 0.78% httpd 91125 www 124 0 162M 22976K RUN 1:46 0.78% 0.78% httpd 91092 www 124 0 162M 23972K RUN 1:45 0.78% 0.78% httpd 91133 www 124 0 162M 17928K RUN 1:44 0.78% 0.78% httpd 91135 www 124 0 158M 19668K RUN 1:43 0.78% 0.78% httpd 89973 www 124 0 158M 18212K RUN 8:15 0.73% 0.73% httpd 90269 www 124 0 158M 19940K RUN 6:40 0.73% 0.73% httpd 90254 www 124 0 162M 25584K RUN 5:58 0.73% 0.73% httpd 90464 www 124 0 195M 27464K RUN 4:51 0.73% 0.73% httpd 91027 www 124 0 181M 24104K RUN 2:09 0.73% 0.73% httpd 91072 www 124 0 158M 17456K RUN 1:53 0.73% 0.73% httpd 91098 www 124 0 158M 19780K RUN 1:48 0.73% 0.73% httpd 91119 www 124 0 162M 23436K RUN 1:45 0.73% 0.73% httpd 91121 www 124 0 162M 23088K RUN 1:45 0.73% 0.73% httpd 91130 www 124 0 158M 19880K RUN 1:44 0.73% 0.73% httpd 91145 www 124 0 158M 19788K RUN 1:42 0.73% 0.73% httpd 91141 www 124 0 181M 23852K RUN 1:40 0.73% 0.73% httpd 91131 www 124 0 167M 41904K RUN 0:37 0.73% 0.73% httpd 91312 www 124 0 157M 31908K RUN 0:37 0.73% 0.73% httpd 89962 www 124 0 179M 35048K RUN 10:58 0.68% 0.68% httpd 89964 www 124 0 179M 39184K RUN 9:51 0.68% 0.68% httpd 3) "apachectl -l" output: Compiled in modules: core.c prefork.c http_core.c mod_so.c 4) Apache's usr/local/etc/apache22/extra/httpd-mpm.conf settings regarding the amount of clients etc. (note: I assume the "prefork MPM" to be the one that is used, so I removed the BeOS, NetWare and OS2 MPM configuration stuff, to save space): # # Server-Pool Management (MPM specific) # # # PidFile: The file in which the server should record its process # identification number when it starts. # # Note that this is the default PidFile for most MPMs. # PidFile /var/run/httpd.pid # # The accept serialization lock file MUST BE STORED ON A LOCAL DISK. # LockFile /var/log/accept.lock # # Only one of the below sections will be relevant on your # installed httpd. Use "apachectl -l" to find out the # active mpm. # # prefork MPM # StartServers: number of server processes to start # MinSpareServers: minimum number of server processes which are kept spare # MaxSpareServers: maximum number of server processes which are kept spare # MaxClients: maximum number of server processes allowed to start # MaxRequestsPerChild: maximum number of requests a server process serves StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 150 MaxRequestsPerChild 0 # worker MPM # StartServers: initial number of server processes to start # MaxClients: maximum number of simultaneous client connections # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # ThreadsPerChild: constant number of worker threads in each server process # MaxRequestsPerChild: maximum number of requests a server process serves StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 # [cut BeOS MPM, NetWare MPM, and OS/2 MPM configuration, as these should not be in use] 5) /etc/mail/-rx.mc: millennics# more ./millennics.nl-rx.mc dnl To be used for MTA-RX, the first MTA instance (receiving mail) dnl Insert here the usual .mc preamble, including OSTYPE and DOMAIN calls. divert(-1) # # [cut the general text] # divert(0) VERSIONID(`$FreeBSD: src/etc/sendmail/freebsd.mc,v 1.28 2003/04/18 01:25:41 gshapiro Exp $') OSTYPE(freebsd5) DOMAIN(generic) dnl Specify here also access controls, relayable domains, anti-spam measures dnl including milter settings if needed, mail submission settings, client dnl authentication, resource controls, maximum mail size and header size, dnl confMIN_FREE_BLOCKS, and other settings needed for receiving mail. dnl dnl NOTE: dnl confMIN_FREE_BLOCKS at MTA-RX should be kept higher than the same dnl setting at MTA-TX, to quench down clients when disk space is low, dnl and not to stop processing the already received mail. dnl dnl In particular, here are some settings to be considered: dnl ( see also http://www.sendmail.org/m4/anti_spam.html ) dnl dnl FEATURE(`access_db',`hash -T /etc/mail/access.db') dnl VIRTUSER_DOMAIN(`sub1.example.com')dnl list valid users here dnl VIRTUSER_DOMAIN(`sub2.example.com')dnl list valid users here dnl FEATURE(`virtusertable', `hash /etc/mail/virtusertable') dnl define(`confUSERDB_SPEC', `/etc/mail/userdb.db') dnl FEATURE(`blacklist_recipients') dnl INPUT_MAIL_FILTER(...) dnl define(`confPRIVACY_FLAGS', `noexpn,novrfy,authwarnings') nobodyreturn ? dnl define(`confDONT_PROBE_INTERFACES') dnl undefine(`USE_CW_FILE')dnl cancel use_cw_file feature, no class {w} extras dnl MASQUERADE_AS(...) FEATURE(`allmasquerade') FEATURE (`masquerade_envelope') dnl define(`confTO_IDENT', `0')dnl Disable IDENT dnl define(`confMAX_MESSAGE_SIZE',`10485760') dnl define(`confMAX_MIME_HEADER_LENGTH', `256/128') dnl define(`confNO_RCPT_ACTION', `add-to-undisclosed') dnl FEATURE(`nocanonify', ...) dnl define(`confBIND_OPTS', ...) dnl define(`confTO_RESOLVER_*... ) dnl define(`confDELAY_LA, 8) dnl define(`confREFUSE_LA', 12) dnl define(`confMIN_FREE_BLOCKS', `10000') dnl define(`confDEF_USER_ID', ...) FEATURE(access_db, `hash -o -T /etc/mail/access') FEATURE(blacklist_recipients) FEATURE(local_lmtp) FEATURE(mailertable, `hash -o /etc/mail/mailertable') FEATURE(virtusertable, `hash -o /etc/mail/virtusertable') dnl [cut relaying, and black(hole) lists, etc., as these are all commented out] dnl Uncomment the first line to change the location of the default dnl /etc/mail/local-host-names and comment out the second line. dnl define(`confCW_FILE', `-o /etc/mail/sendmail.cw') define(`confCW_FILE', `-o /etc/mail/local-host-names') dnl Uncomment both of the following lines to listen on IPv6 as well as IPv4 dnl DAEMON_OPTIONS(`Name=IPv4, Family=inet') dnl DAEMON_OPTIONS(`Name=IPv6, Family=inet6') define(`confBIND_OPTS', `WorkAroundBrokenAAAA') define(`confNO_RCPT_ACTION', `add-to-undisclosed') define(`confPRIVACY_FLAGS', `authwarnings,noexpn,novrfy') define(`confRUN_AS_USER',`smmsp:smmsp')dnl Drop privileges (see SECURITY NOTE) define(`confPID_FILE', `/var/run/sendmail-rx.pid')dnl Non-default pid file define(`STATUS_FILE', `/etc/mail/stat-rx')dnl Non-default stat file define(`QUEUE_DIR', `/var/spool/mqueue-rx')dnl Non-default queue area define(`confQUEUE_SORT_ORDER',`Modification')dnl Modif or Random are reasonable dnl Match the number of queue runners (R=) to the number of amavisd- new child dnl processes ($max_servers). 2 to 7 OK, 10 is plenty, 20 is too many QUEUE_GROUP(`mqueue', `P=/var/spool/mqueue-rx, R=2, F=f')dnl dnl Direct all mail to be forwarded to amavisd-new at 127.0.0.1:10024 FEATURE(stickyhost)dnl Keep envelope addr "u@local.host" when fwd to MAIL_HUB define(`MAIL_HUB', `esmtp:[127.0.0.1]')dnl Forward all local mail to amavisd define(`SMART_HOST',`esmtp:[127.0.0.1]')dnl Forward all other mail to amavisd define(`confDELIVERY_MODE',`q')dnl Delivery mode: queue only (a must, dnl ... otherwise the advantage of this setup of being able to specify dnl ... the number of queue runners is lost) define(`ESMTP_MAILER_ARGS',`TCP $h 10024')dnl To tcp port 10024 instead of 25 MODIFY_MAILER_FLAGS(`ESMTP', `+z')dnl Speak LMTP (this is optional) define(`SMTP_MAILER_MAXMSGS',`10')dnl Max no. of msgs in a single connection define(`confTO_DATAFINAL',`20m')dnl 20 minute timeout for content checking DAEMON_OPTIONS(`Name=MTA-RX')dnl Daemon name used in logged messages dnl Disable local delivery, as all local mail will go to MAIL_HUB undefine(`ALIAS_FILE')dnl No aliases file, all local mail goes to MAIL_HUB define(`confFORWARD_PATH')dnl Empty search path for .forward files undefine(`UUCP_RELAY')dnl undefine(`BITNET_RELAY')dnl undefine(`DECNET_RELAY')dnl MAILER(smtp)