From owner-freebsd-www  Fri Jul 25 01:55:32 1997
Return-Path: <owner-freebsd-www>
Received: (from root@localhost)
          by hub.freebsd.org (8.8.5/8.8.5) id BAA22955
          for www-outgoing; Fri, 25 Jul 1997 01:55:32 -0700 (PDT)
Received: from schubert.promo.de (schubert.Promo.DE [194.45.188.65])
          by hub.freebsd.org (8.8.5/8.8.5) with ESMTP id BAA22950
          for <www@FreeBSD.ORG>; Fri, 25 Jul 1997 01:55:29 -0700 (PDT)
Received: from [194.45.188.81] (stefan.Promo.DE [194.45.188.81])
	by schubert.promo.de (8.8.5/8.8.5) with ESMTP id KAA11572;
	Fri, 25 Jul 1997 10:49:34 +0200 (MET DST)
X-Sender: stefan@mail.promo.de
Message-Id: <l03102801affe19c9821a@[194.45.188.81]>
In-Reply-To: <3.0.32.19970724213535.008c8310@asis.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Date: Fri, 25 Jul 1997 10:54:08 +0200
To: Megan McCormack <meganm@asis.com>, www@FreeBSD.ORG
From: Stefan Bethke <stefan@promo.de>
Subject: Re: robots.txt
Sender: owner-freebsd-www@FreeBSD.ORG
X-Loop: FreeBSD.org
Precedence: bulk

At 6:35 Uhr +0200 25.07.1997, Megan McCormack wrote:
>Hello,
>
>Does it make sense to disallow stats.html in robots.txt
>so that mirror sites error logs don't overflow with
>error messages of robots trying to scoop up the links to
>local pages like ~jkh on the mirrors?

No, rather make analog produce absolute URLs.

Btw, /robots.txt currently reads
User-agent: *
Disallow: /cgi-bin/
Disallow: /cgi/
Disallow: ~/jfieber/data/cgi/

it should be
...
Disallow: /~jfieber/data/cgi/


Stefan

--
Stefan Bethke
Promo Datentechnik      |  Tel. +49-40-851744-0
+ Systemberatung GmbH   |  Fax. +49-40-851744-44
Eduardstrasse 46-48     |  e-mail: stefan@Promo.DE
D-20257 Hamburg         |  http://www.Promo.DE/