Date: Thu, 18 May 2006 20:37:00 -0400 From: Chuck Swiger <cswiger@mac.com> To: Michael Jeung <mjeung@cisdata.net> Cc: freebsd-net@freebsd.org Subject: Re: Load Balancing Message-ID: <446D132C.7010905@mac.com> In-Reply-To: <3D7C7275-432A-448D-82D6-AB551A1CE256@cisdata.net> References: <3D7C7275-432A-448D-82D6-AB551A1CE256@cisdata.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Michael Jeung wrote: > We currently use DNS round-robin to balance traffic to servers. We've > recently run into situations where multiple search engine spiders are > crawling our webservers. They appear to be targeting specific > webservers by IP address. This defeats DNS round-robin and as a > result 1 of our webservers starts responding very slowly since it's > full of search engine spiders. Set up a /robots.txt file in your webserver root which contains appropriate Disallow or Crawl-delay entries to reduce the rate at which spiders make requests, or eliminate such traffic entirely. For example: http://www.pkix.net/robots.txt -- -Chuck
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?446D132C.7010905>