From owner-freebsd-stable@FreeBSD.ORG Fri Feb 17 01:43:23 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 69A7C16A420 for ; Fri, 17 Feb 2006 01:43:23 +0000 (GMT) (envelope-from atanas@asd.aplus.net) Received: from pro20.abac.com (pro20.abac.com [66.226.64.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 11A3E43D46 for ; Fri, 17 Feb 2006 01:43:22 +0000 (GMT) (envelope-from atanas@asd.aplus.net) Received: from [216.55.129.41] (asd0.aplus.net [216.55.129.41]) (authenticated bits=0) by pro20.abac.com (8.13.4/8.13.4) with ESMTP id k1H1hI2Y015883; Thu, 16 Feb 2006 17:43:19 -0800 (PST) (envelope-from atanas@asd.aplus.net) Message-ID: <43F52C00.9050904@asd.aplus.net> Date: Thu, 16 Feb 2006 17:50:56 -0800 From: Atanas User-Agent: Thunderbird 1.5 (X11/20051201) MIME-Version: 1.0 To: David Malone References: <200602162124.aa23962@salmon.maths.tcd.ie> In-Reply-To: <200602162124.aa23962@salmon.maths.tcd.ie> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 1.47 (SPF_SOFTFAIL) Cc: Rostislav Krasny , =?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?= , "Michael A. Koerber" , freebsd-stable@freebsd.org, Marian Hettwer Subject: Re: SSH login takes very long time...sometimes X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Feb 2006 01:43:23 -0000 David Malone said the following on 02/16/06 13:24: >> Just a thought, wouldn't this open a new possibility for denial of >> service attacks? > > I doubt it. I'm guessing you're thinking of an attack where someone > makes many connections to sshd in a short time and runs you out of > processes? I think you can protect against this with the MaxStartups > directive in sshd_config. The amount of time that an attacker has > to open many connections is probably not that important, as you can > open a lot of TCP connections in 1 second even with a small link. > These were different types of attacks, primarily originating from single IP addresses: 1. Dictionary attacks taking as much concurrent unauthenticated connections as possible and with speeds as fast as the server can respond. These were happening like a few up to several times a day, sometimes lasting hours. 2. Time based attacks taking again all of the available MaxStartups, but then doing nothing until the LoginGraceTime expires, then again, etc. These were not so frequent, but had the worst impact on the ssh availability. 3. Network scans on machines hosting some hundreds (or in some cases thousands) of IP addresses, causing outages lasting just a few minutes or so. >> Last year I already had to decrease the LoginGraceTime from 120 to 30 >> seconds on my production boxes, but it didn't help much, so on top of >> that I got to implement (reinvent the wheel again) a script tailing the >> auth.log and firewalling bad gyus in order to secure sshd and let my >> legitimate users in. > > Are you trying to prevent the ssh scanners that just try well-known > combinations of usernames and passwords? It is not clear that you > gain much by firewalling these off, other than having fewer log > messages. > All of the above three. It wasn't just a matter of too much log messages. The type 1. for instance, besides the ssh unavailability, and depending on the MaxStartups setting, can bring a server to its knees by dedicating all of its available resources for bouncing unauthenticated ssh requests. I tried setting a 'limit' ipfw firewall rule, something like: ipfw add allow tcp from any to any 22 in setup limit src-addr 5 I already had success with such a rule for a first level SMTP DoS protection before sendmail got its per-client and rate connection limits built in (since 8.13.0), and still keep that on, just in case. But unlike sendmail, ssh instances when hammered with bogus requests are way more CPU intensive. I couldn't afford limiting the ssh connectivity to just one single session per client IP (someone might need multiple ssh sessions while working on something, right?), and in case of multiple sessions enabled, machines would be still vulnerable to CPU overload (i.e. bouncing tons of useless ssh authentication attempts). So the best option for me was to implement a log analyzer script placing temporary blocks on the firewall when necessary. Like after 5 "Illegal user" or "Failed password for" or "Did not receive identification string" events, the script simply denies that IP right away on the firewall for one hour. So far this works well (for about 6 months already) and I no longer see unusual load spikes or ssh connectivity outages like before. >> I really miss the inetd features. A setting like "nowait/100/20/5" >> (/max-child[/max-connections-per-ip-per-minute[/max-child-per-ip]]) >> would effectively bounce the bad guys, but AFAIK (correct me if I'm >> wrong), ssh is no longer supposed to work via inetd and still has no >> such capabilities. > > You can still run sshd through inetd (or, at least, the -i option > is still documented in the sshd man page). If does suggest that you > may need to reduce the key size to make this practical (increasing > LoginGraceTime here may help too ;-) > I knew that, but actually never tried it thinking it would be too slow. Now I just ran a ssh-keygen and found that it takes only a few seconds for a 1024-bit and several for 2048-bit key. So it's not that much bad and running it with 512 or 1024-bit key through inetd seems feasible enough. The default ssh key length in FreeBSD-6 however just got doubled from 1024 to 2048-bit. I believe there's a reason for that and don't like the idea of going down. Regards, Atanas