From owner-freebsd-net@FreeBSD.ORG Fri May 19 00:37:15 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3C79716A444 for ; Fri, 19 May 2006 00:37:15 +0000 (UTC) (envelope-from cswiger@mac.com) Received: from pi.codefab.com (pi.codefab.com [199.103.21.227]) by mx1.FreeBSD.org (Postfix) with ESMTP id E02F043D5D for ; Fri, 19 May 2006 00:37:05 +0000 (GMT) (envelope-from cswiger@mac.com) Received: from localhost (localhost [127.0.0.1]) by pi.codefab.com (Postfix) with ESMTP id 643935DB5; Thu, 18 May 2006 20:37:04 -0400 (EDT) X-Virus-Scanned: amavisd-new at codefab.com Received: from pi.codefab.com ([127.0.0.1]) by localhost (pi.codefab.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jZsGygv+jnIk; Thu, 18 May 2006 20:37:03 -0400 (EDT) Received: from [192.168.1.251] (pool-68-160-242-211.ny325.east.verizon.net [68.160.242.211]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by pi.codefab.com (Postfix) with ESMTP id A34565D0B; Thu, 18 May 2006 20:37:02 -0400 (EDT) Message-ID: <446D132C.7010905@mac.com> Date: Thu, 18 May 2006 20:37:00 -0400 From: Chuck Swiger User-Agent: Thunderbird 1.5.0.2 (Windows/20060308) MIME-Version: 1.0 To: Michael Jeung References: <3D7C7275-432A-448D-82D6-AB551A1CE256@cisdata.net> In-Reply-To: <3D7C7275-432A-448D-82D6-AB551A1CE256@cisdata.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: Load Balancing X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 May 2006 00:37:15 -0000 Michael Jeung wrote: > We currently use DNS round-robin to balance traffic to servers. We've > recently run into situations where multiple search engine spiders are > crawling our webservers. They appear to be targeting specific > webservers by IP address. This defeats DNS round-robin and as a > result 1 of our webservers starts responding very slowly since it's > full of search engine spiders. Set up a /robots.txt file in your webserver root which contains appropriate Disallow or Crawl-delay entries to reduce the rate at which spiders make requests, or eliminate such traffic entirely. For example: http://www.pkix.net/robots.txt -- -Chuck