Date: Thu, 24 Jul 1997 21:35:37 -0700 From: Megan McCormack <meganm@asis.com> To: www@freebsd.org Subject: robots.txt Message-ID: <3.0.32.19970724213535.008c8310@asis.com>
next in thread | raw e-mail | index | archive | help
Hello,
Does it make sense to disallow stats.html in robots.txt
so that mirror sites error logs don't overflow with
error messages of robots trying to scoop up the links to
local pages like ~jkh on the mirrors?
Megan
************** Chop wood, carry water, check e-mail ************
Megan McCormack
http://www.asis.com/~meganm
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3.0.32.19970724213535.008c8310>