From owner-freebsd-doc Tue Dec 15 13:39:58 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id NAA29609 for freebsd-doc-outgoing; Tue, 15 Dec 1998 13:39:58 -0800 (PST) (envelope-from owner-freebsd-doc@FreeBSD.ORG) Received: from mail2.netcologne.de (mail2.netcologne.de [194.8.194.103]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id NAA29586 for ; Tue, 15 Dec 1998 13:39:55 -0800 (PST) (envelope-from marc@netcologne.de) Received: from oranje.my.domain (dial-ra-nc1-28.netcologne.de [195.14.244.28]) by mail2.netcologne.de (8.9.1/8.9.1) with ESMTP id WAA27431; Tue, 15 Dec 1998 22:39:39 +0100 (MET) X-Ncc-Regid: de.netcologne Received: (from marc@localhost) by oranje.my.domain (8.9.1/8.9.1) id WAA95774; Tue, 15 Dec 1998 22:40:31 +0100 (CET) (envelope-from marc) Date: Tue, 15 Dec 1998 22:40:31 +0100 (CET) Message-Id: <199812152140.WAA95774@oranje.my.domain> From: Marc van Woerkom To: jim_turk@ix.netcom.com CC: doc@FreeBSD.ORG In-reply-to: (message from Jim Turk on Thu, 10 Dec 1998 21:41:20 -0700 (MST)) Subject: Re: CGI scripts for FreeBSD documentation project Reply-to: van.woerkom@netcologne.de Sender: owner-freebsd-doc@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hello Jim! > "checkbot" will do what you requested with some modification Your solution, however you realize it, should honor the conventions for robots i.e. the robots.txt file that tells you where you are welcome and where not. On www.freebsd.org it looks like this: User-agent: * Disallow: /cgi-bin/ Disallow: /cgi/ Disallow: /~jfieber/data/cgi/ Disallow: /~wosch/test/ Disallow: /statistic Disallow: /stat Disallow: /gifs/ Regards, Marc To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-doc" in the body of the message