From owner-freebsd-performance@FreeBSD.ORG Wed Aug 11 18:00:00 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22DFB1065673 for ; Wed, 11 Aug 2010 18:00:00 +0000 (UTC) (envelope-from cswiger@mac.com) Received: from asmtpout028.mac.com (asmtpout028.mac.com [17.148.16.103]) by mx1.freebsd.org (Postfix) with ESMTP id 0A92D8FC0C for ; Wed, 11 Aug 2010 17:59:59 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; charset=us-ascii Received: from cswiger1.apple.com ([17.209.4.71]) by asmtp028.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec 16 2008; 32bit)) with ESMTPSA id <0L70006FM1Z2CT70@asmtp028.mac.com> for freebsd-performance@freebsd.org; Wed, 11 Aug 2010 10:59:27 -0700 (PDT) X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=6.0.2-1004200000 definitions=main-1008110142 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.0.10011,1.0.148,0.0.0000 definitions=2010-08-11_07:2010-08-11, 2010-08-11, 1970-01-01 signatures=0 From: Chuck Swiger In-reply-to: <4C62D827.2030409@ssimicro.com> Date: Wed, 11 Aug 2010 10:59:26 -0700 Message-id: <949C0FF2-04AA-4440-82B0-F44A13B8F0C2@mac.com> References: <4C62D827.2030409@ssimicro.com> To: markham breitbach X-Mailer: Apple Mail (2.1081) Cc: freebsd-performance@freebsd.org Subject: Re: massive load average spikes X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Aug 2010 18:00:00 -0000 Hi-- On Aug 11, 2010, at 10:04 AM, markham breitbach wrote: > I am running into an issue where I am seeing load average on a server suddenly jump from > nominal values around 0.5 to anywhere from 10 up over 70 in under 1 second. This does not > seem to be related to CPU overload, and LA immediately begins to fall back again to > nominal. This does not seem to happen with any regular frequency, and can happen several > times an hour or not for hours. [ ... ] > Can anyone suggest what may be causing this or how to track that down? >From the (limited) available data, I'd imagine someone is doing wardialling of your mail service to try common username/password combinations and break in. Especially if they are connecting via POP3S / IMAPS ports and doing SSL negotiation, there's a very high burst of CPU load, as imap or pop daemons get forked to handle the requests, then quit immediately afterwards when the login attempt fails. You won't see much change in memory loading unless they do get a valid login since the Dovecot daemons are already resident & there's no real I/O made to disk until it looks up a real user's mail. Looking at tcpdump for new connection requests or checking the Dovecot mail logs for a slew of attempted logins for invalid users, and correlating with your load spikes would be a way of checking on this theory.... Regards, -- -Chuck