Date: Tue, 15 Dec 1998 22:40:31 +0100 (CET) From: Marc van Woerkom <marc@netcologne.de> To: jim_turk@ix.netcom.com Cc: doc@FreeBSD.ORG Subject: Re: CGI scripts for FreeBSD documentation project Message-ID: <199812152140.WAA95774@oranje.my.domain> In-Reply-To: <Pine.BSF.4.05.9812102125120.273-100000@localhost> (message from Jim Turk on Thu, 10 Dec 1998 21:41:20 -0700 (MST))
next in thread | previous in thread | raw e-mail | index | archive | help
Hello Jim! > "checkbot" will do what you requested with some modification Your solution, however you realize it, should honor the conventions for robots i.e. the robots.txt file that tells you where you are welcome and where not. On www.freebsd.org it looks like this: User-agent: * Disallow: /cgi-bin/ Disallow: /cgi/ Disallow: /~jfieber/data/cgi/ Disallow: /~wosch/test/ Disallow: /statistic Disallow: /stat Disallow: /gifs/ Regards, Marc To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-doc" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199812152140.WAA95774>