From owner-freebsd-isp@FreeBSD.ORG Thu Jul 13 16:20:39 2006 Return-Path: X-Original-To: freebsd-isp@freebsd.org Delivered-To: freebsd-isp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 41C5216A4EB for ; Thu, 13 Jul 2006 16:20:39 +0000 (UTC) (envelope-from gary@tbe.net) Received: from kerplunk.tbe.net (kerplunk.tbe.net [209.123.115.134]) by mx1.FreeBSD.org (Postfix) with ESMTP id E9D1643D46 for ; Thu, 13 Jul 2006 16:20:38 +0000 (GMT) (envelope-from gary@tbe.net) Received: by kerplunk.tbe.net (Postfix, from userid 1001) id D65995CAD; Thu, 13 Jul 2006 12:16:16 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by kerplunk.tbe.net (Postfix) with ESMTP id C08F55CAC; Thu, 13 Jul 2006 12:16:16 -0400 (EDT) Date: Thu, 13 Jul 2006 12:16:16 -0400 (EDT) From: "Gary D. Margiotta" To: Arie Kachler In-Reply-To: <44B66D42.6030302@telcom.net> Message-ID: <20060713120423.I63493@kerplunk.tbe.net> References: <44B66D42.6030302@telcom.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-isp@freebsd.org Subject: Re: compromised machines and entire network health X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jul 2006 16:20:39 -0000 On Thu, 13 Jul 2006, Arie Kachler wrote: > Hello, > > In the past several years, we have had a few incidents of servers of > customers that are compromised and then flood our entire network and bring > down almost everything. The sql slammer worm for example. > > Is there a solution to this? I know all computers should be kept up to date > so this does not happen, but most times customers are not as attentive to > patches as we sysadmins are. > Assuming that there will always be machines with security issues, is there a > way to prevent a compromised computer to bring down an entire network? > > Any suggestions will be greatly appreciated. > > Arie Kachler Firewall each machine, or see if you can do rate limiting on the machines to minimize the amount of traffic each machine can pump out at any given time. You can try to do it at the machine level, or you can look for smart hardware such as smart switches or IDS systems that will do it for you. We limit each port on our switch to 10Mbit (which shouldn't be able to flood the entire network and take it down... maybe slow it up a slight bit, but nothing catastrophic), and we have alarms to trigger when bandwith exceeds a certain threshold for a certain length of time. The port gets shut off if the alarm fires, and the customer is advised of their problem, and is required to fix it before we allow it back on the network. No exceptions. We enforce a policy for customers to patch their machines themselves, or they have us do it for them as a managed service. The customer is responsible for any damages related to any hacks/worms/mistakes, and the machines are removed from the network until they are fixed, either by them or by us standing on a console. If they don't upkeep their systems on their own, we do it for them and charge them for it. If they refuse to pay, we shut off their machine, confiscate their hardware, and go after them for any other time and materials related to the problem. Mostly it doesn't get that far, but you have to be prepared for it with a published policy outlining these types of things. Most customers get the point after they see the initial bill for damages their machine caused, and they just have us manage their systems for them, it's easier (and cheaper) for them, and safer for us, plus they are not responsible for any more damages if a machine we manage has a problem. -Gary