Date: Wed, 19 Aug 1998 16:07:06 -0400 From: Dennis <dennis@etinc.com> To: Graeme Tait <graeme@echidna.com>, freebsd-isp@FreeBSD.ORG Cc: info@boatbooks.com Subject: Re: Rate limiting HTTP requests Message-ID: <199808192008.QAA06877@etinc.com> In-Reply-To: <35DB1584.1733@echidna.com>
next in thread | previous in thread | raw e-mail | index | archive | help
At 11:12 AM 8/19/98 -0700, Graeme Tait wrote: >Is there any mechanism possible in FreeBSD or Apache, for limiting the >rate at which HTTP GET requests are served? > >Specifically, we want to accommodate search engines that try to index a >large site we will host. Unfortunately, experience indicates that they >may fetch data at an uncomfortably high rate. > >The server will be fine with this. The problem is that our bandwidth >charges will be based on the 95th percentile, 5-minute traffic average. >For those who aren't familiar with this, the colocation service has their >router log the total upstream and downstream bits transferred through our >network connection in each 5-minute interval. At the end of each calendar >month, the top 5% of these 5-minute intervals (traffic-wise) are >discarded, and the remaining highest value determines the billable rate >in Mbps. > >The problem here is that historical data (on the host where the site >concerned is presently) indicates the search engines create undesirable >(and potentially expensive) peaks. > >We don't want to lock the search engines out. Ideally, we would simply >degrade service to them (while maintaining full service for regular >users) so as to keep the peak data rate under some set limit. The rate >limiting criterion could be based on degrading service for any accessor >(remote IP) generating more than a certain level of traffic on a >sustained basis. > >I hope this is not asking too much, but from our point of view, the >dollars involved could be considerable! We sell a software add-in to freebsd that can limit traffic from or to specific IP addresses, hosts or networks. Info at http://www.etinc.com/bwmgr.htm dennis To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199808192008.QAA06877>