From owner-freebsd-hackers Thu Nov 9 18:27:54 1995 Return-Path: owner-hackers Received: (from root@localhost) by freefall.freebsd.org (8.6.12/8.6.6) id SAA24127 for hackers-outgoing; Thu, 9 Nov 1995 18:27:54 -0800 Received: from strider.ibenet.it (root@strider.ibe.net [194.179.130.1]) by freefall.freebsd.org (8.6.12/8.6.6) with ESMTP id SAA24113 for ; Thu, 9 Nov 1995 18:27:39 -0800 Received: (from piero@localhost) by strider.ibenet.it (8.6.12/8.6.12) id DAA29839; Fri, 10 Nov 1995 03:23:13 +0100 From: Piero Serini Message-Id: <199511100223.DAA29839@strider.ibenet.it> Subject: Re: robot.txt file To: gcrutcher@datatrek.com (Gary Crutcher) Date: Fri, 10 Nov 1995 03:23:12 +0100 (MET) Cc: hackers@freebsd.org In-Reply-To: <199511100150.RAA11236@dtihost.datatrek.com> from "Gary Crutcher" at Nov 9, 95 05:50:44 pm Reply-To: piero@strider.ibenet.it Operating-System: FreeBSD 1.1.5.1 X-Phone-Number: +39 (2) 58113562 X-NCC-RegID: it.ibenet X-Mailer: ELM [version 2.4 PL23] Content-Type: text Content-Length: 856 Sender: owner-hackers@freebsd.org Precedence: bulk Hello. Quoting from Gary Crutcher (Fri Nov 10 02:50:44 1995): > I have been checking my apache server's logfile and noticed that a number > sites have been trying to access a file called "robot.txt". What is this > file and do I need it on my system. The file /robot.txt, if present, tells the robots if they are welcome to your site or not, and if they are tells them which directory they cannot access, if any. For more informations, check: http://info.webcrawler.com/mak/projects/robots/robots.html I find the whole very intersting, but if you only want to exclude robots, follow the link "A standard for Robot exclusion". Bye, -- # $Id: .signature,v 1.12 1995/08/14 12:10:54 piero Exp $ Piero Serini Via Giambologna, 1 I 20136 Milano - ITALY