From owner-freebsd-isp  Wed Aug 19 12:57:10 1998
Return-Path: <owner-freebsd-isp@FreeBSD.ORG>
Received: (from majordom@localhost)
          by hub.freebsd.org (8.8.8/8.8.8) id MAA17480
          for freebsd-isp-outgoing; Wed, 19 Aug 1998 12:57:10 -0700 (PDT)
          (envelope-from owner-freebsd-isp@FreeBSD.ORG)
Received: from etinc.com (et-gw.etinc.com [207.252.1.2])
          by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id MAA17472
          for <freebsd-isp@FreeBSD.ORG>; Wed, 19 Aug 1998 12:57:03 -0700 (PDT)
          (envelope-from dennis@etinc.com)
Received: from dbsys (dbsys.etinc.com [207.252.1.18]) by etinc.com (8.8.7/8.6.9) with SMTP id QAA06877; Wed, 19 Aug 1998 16:08:52 -0400 (EDT)
Message-Id: <199808192008.QAA06877@etinc.com>
X-Sender: dennis@etinc.com
X-Mailer: QUALCOMM Windows Eudora Pro Version 4.0
Date: Wed, 19 Aug 1998 16:07:06 -0400
To: Graeme Tait <graeme@echidna.com>, freebsd-isp@FreeBSD.ORG
From: Dennis <dennis@etinc.com>
Subject: Re: Rate limiting HTTP requests
Cc: info@boatbooks.com
In-Reply-To: <35DB1584.1733@echidna.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Sender: owner-freebsd-isp@FreeBSD.ORG
Precedence: bulk
X-Loop: FreeBSD.org

At 11:12 AM 8/19/98 -0700, Graeme Tait wrote:
>Is there any mechanism possible in FreeBSD or Apache, for limiting the 
>rate at which HTTP GET requests are served?
>
>Specifically, we want to accommodate search engines that try to index a 
>large site we will host. Unfortunately, experience indicates that they 
>may fetch data at an uncomfortably high rate.
>
>The server will be fine with this. The problem is that our bandwidth 
>charges will be based on the 95th percentile, 5-minute traffic average. 
>For those who aren't familiar with this, the colocation service has their 
>router log the total upstream and downstream bits transferred through our 
>network connection in each 5-minute interval. At the end of each calendar 
>month, the top 5% of these 5-minute intervals (traffic-wise) are 
>discarded, and the remaining highest value determines the billable rate 
>in Mbps.
>
>The problem here is that historical data (on the host where the site 
>concerned is presently) indicates the search engines create undesirable 
>(and potentially expensive) peaks.
>
>We don't want to lock the search engines out. Ideally, we would simply 
>degrade service to them (while maintaining full service for regular 
>users) so as to keep the peak data rate under some set limit. The rate 
>limiting criterion could be based on degrading service for any accessor 
>(remote IP) generating more than a certain level of traffic on a 
>sustained basis.
>
>I hope this is not asking too much, but from our point of view, the 
>dollars involved could be considerable!

We sell a software add-in to freebsd that can limit traffic from or to
specific
IP addresses, hosts or networks. Info at http://www.etinc.com/bwmgr.htm

dennis

To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-isp" in the body of the message