From owner-svn-doc-all@FreeBSD.ORG Fri May 31 23:03:58 2013 Return-Path: Delivered-To: svn-doc-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 18992FCF for ; Fri, 31 May 2013 23:03:58 +0000 (UTC) (envelope-from lists@eitanadler.com) Received: from mail-pb0-x22f.google.com (mail-pb0-x22f.google.com [IPv6:2607:f8b0:400e:c01::22f]) by mx1.freebsd.org (Postfix) with ESMTP id E5E19CB for ; Fri, 31 May 2013 23:03:57 +0000 (UTC) Received: by mail-pb0-f47.google.com with SMTP id rr4so2933528pbb.6 for ; Fri, 31 May 2013 16:03:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eitanadler.com; s=0xdeadbeef; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type; bh=yijB4Sw9N7TjJLd0KwveN2aCqnqP1bp6mv/kDnyZgAo=; b=nHowz8Wi1rIVed4xE9qUj1Sv226lc4SXf7PatpPn+7L+aJHyp9OADgNtGROt06jBmB D+7xFlRtV4oFe8Ie4GUPDeQvd6bYGpl0gVWsml/JGArXGFBgKIbSsWRl9WRGzG3KVG/d gwc/Ql3bhIztLBFnAgHpbqUcxwNWMFNX0Q2/4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :x-gm-message-state; bh=yijB4Sw9N7TjJLd0KwveN2aCqnqP1bp6mv/kDnyZgAo=; b=e84Iac5yHNM1VjVGDYKWv4Aq/YZBLJygm1GKsQQibv/FVoBQ2uyc+98WD0oQ/hJRI2 a8imG58aar091pLKKnguSWSW0UWyGY7rTt6GXnHg8jHaXW4tSxxfO44nYjO+16p/scFS db8Ab35gKgxVPvyjYmIXCqNa2hue5rK9WuImmj1TwEBNQHXpFS0saldNJzNJytCabmrb xjUanN1EpxHsk/p+6Cg1AxTYtG8LzsV9OZpkmFwmeDstL+2JjjctY/3Ry6t8mthaArEn IFY9+3BWZoPVdYkQKh1x6M37UQwENy7qchQLo0U0YbIjGvJi14Hfw5vKdy1M9/pfry1K JbDQ== X-Received: by 10.66.192.7 with SMTP id hc7mr15508803pac.206.1370041437730; Fri, 31 May 2013 16:03:57 -0700 (PDT) MIME-Version: 1.0 Sender: lists@eitanadler.com Received: by 10.70.91.139 with HTTP; Fri, 31 May 2013 16:03:27 -0700 (PDT) In-Reply-To: <20130531225923.GA1619@glenbarber.us> References: <201305312254.r4VMsjo0003070@svn.freebsd.org> <20130531225923.GA1619@glenbarber.us> From: Eitan Adler Date: Sat, 1 Jun 2013 01:03:27 +0200 X-Google-Sender-Auth: fCuA1H38_wo37SwiXlH-1m1q5JA Message-ID: Subject: Re: svn commit: r41802 - head/en_US.ISO8859-1/htdocs To: Glen Barber Content-Type: text/plain; charset=UTF-8 X-Gm-Message-State: ALoCoQm5wGLG1shMI7eB9JRdYJ56tzSaTunQbinR350yOs6D/q4aw7HNMb9LIJcidDIpyMxT1nu9 Cc: svn-doc-head@freebsd.org, svn-doc-all@freebsd.org, doc-committers@freebsd.org X-BeenThere: svn-doc-all@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "SVN commit messages for the entire doc trees \(except for " user" , " projects" , and " translations" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 23:03:58 -0000 On 1 June 2013 00:59, Glen Barber wrote: > On Fri, May 31, 2013 at 10:54:45PM +0000, Eitan Adler wrote: >> Author: eadler >> Date: Fri May 31 22:54:45 2013 >> New Revision: 41802 >> URL: http://svnweb.freebsd.org/changeset/doc/41802 >> >> Log: >> There appear to be configuration errors causing an absurd amount of links to be generated. >> For now, block access to search engines until the problem is properly resolved. >> >> While here, add $FreeBSD$ >> >> Modified: >> head/en_US.ISO8859-1/htdocs/robots.txt >> >> Modified: head/en_US.ISO8859-1/htdocs/robots.txt >> ============================================================================== >> --- head/en_US.ISO8859-1/htdocs/robots.txt Fri May 31 19:10:06 2013 (r41801) >> +++ head/en_US.ISO8859-1/htdocs/robots.txt Fri May 31 22:54:45 2013 (r41802) >> @@ -1,4 +1,8 @@ >> +# $FreeBSD$ >> + >> User-agent: * >> >> Disallow: /cgi/ >> Disallow: /statistic >> +Disallow: /publish.html/hu-hu/ > > This is wrong. > > http://www.freebsd.org/index.html/omgwtfbbqhax I was fixing a specific instance reported by automated tools run by a search engine. A more general fix needs to implemented by clusteradm@. >> +Disallow: /cgi/ > > This already exists in the file... ... Interesting, I am still getting reports of these URLs being indexed. I'll look into that. -- Eitan Adler Source, Ports, Doc committer Bugmeister, Ports Security teams