Date: Wed, 19 Aug 1998 11:12:20 -0700 From: Graeme Tait <graeme@echidna.com> To: freebsd-isp@FreeBSD.ORG Cc: info@boatbooks.com Subject: Rate limiting HTTP requests Message-ID: <35DB1584.1733@echidna.com>
next in thread | raw e-mail | index | archive | help
Is there any mechanism possible in FreeBSD or Apache, for limiting the rate at which HTTP GET requests are served? Specifically, we want to accommodate search engines that try to index a large site we will host. Unfortunately, experience indicates that they may fetch data at an uncomfortably high rate. The server will be fine with this. The problem is that our bandwidth charges will be based on the 95th percentile, 5-minute traffic average. For those who aren't familiar with this, the colocation service has their router log the total upstream and downstream bits transferred through our network connection in each 5-minute interval. At the end of each calendar month, the top 5% of these 5-minute intervals (traffic-wise) are discarded, and the remaining highest value determines the billable rate in Mbps. The problem here is that historical data (on the host where the site concerned is presently) indicates the search engines create undesirable (and potentially expensive) peaks. We don't want to lock the search engines out. Ideally, we would simply degrade service to them (while maintaining full service for regular users) so as to keep the peak data rate under some set limit. The rate limiting criterion could be based on degrading service for any accessor (remote IP) generating more than a certain level of traffic on a sustained basis. I hope this is not asking too much, but from our point of view, the dollars involved could be considerable! -- Graeme Tait - Echidna To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?35DB1584.1733>