From owner-freebsd-www@FreeBSD.ORG  Sun Aug 21 09:10:12 2011
Return-Path: <owner-freebsd-www@FreeBSD.ORG>
Delivered-To: freebsd-www@hub.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 130F31065670
	for <freebsd-www@hub.freebsd.org>; Sun, 21 Aug 2011 09:10:12 +0000 (UTC)
	(envelope-from gnats@FreeBSD.org)
Received: from freefall.freebsd.org (freefall.freebsd.org
	[IPv6:2001:4f8:fff6::28])
	by mx1.freebsd.org (Postfix) with ESMTP id 038678FC16
	for <freebsd-www@hub.freebsd.org>; Sun, 21 Aug 2011 09:10:12 +0000 (UTC)
Received: from freefall.freebsd.org (localhost [127.0.0.1])
	by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p7L9ABnr012316
	for <freebsd-www@freefall.freebsd.org>; Sun, 21 Aug 2011 09:10:11 GMT
	(envelope-from gnats@freefall.freebsd.org)
Received: (from gnats@localhost)
	by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p7L9ABaO012315;
	Sun, 21 Aug 2011 09:10:11 GMT (envelope-from gnats)
Date: Sun, 21 Aug 2011 09:10:11 GMT
Message-Id: <201108210910.p7L9ABaO012315@freefall.freebsd.org>
To: freebsd-www@FreeBSD.org
From: "Simon L. B. Nielsen" <simon@FreeBSD.org>
Cc: 
Subject: Re: www/159652: PR is not visible to search engines
X-BeenThere: freebsd-www@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
Reply-To: "Simon L. B. Nielsen" <simon@FreeBSD.org>
List-Id: FreeBSD Project Webmasters <freebsd-www.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-www>,
	<mailto:freebsd-www-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-www>
List-Post: <mailto:freebsd-www@freebsd.org>
List-Help: <mailto:freebsd-www-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-www>,
	<mailto:freebsd-www-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 21 Aug 2011 09:10:12 -0000

The following reply was made to PR www/159652; it has been noted by GNATS.

From: "Simon L. B. Nielsen" <simon@FreeBSD.org>
To: Marcin Wisnicki <mwisnicki+freebsd@gmail.com>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: www/159652: PR is not visible to search engines
Date: Sun, 21 Aug 2011 11:05:37 +0200

 On 10 Aug 2011, at 19:19, Marcin Wisnicki wrote:
 
 > Currently www.freebsd.org/robots.txt disallows access to /cgi/ and =
 that includes query-pr.cgi which makes bugs invisible to search engines =
 :(
 
 The problem is that query-cgi allows searches which are quiet heavy =
 resource wise (taking minutes to finish). If we just remove the =
 robots.txt from that area and the query cgi any reasonable amount of =
 hardware won't be enough. We even have extra protection which only =
 allows two query-pr's to run at once to avoid killing www.FreeBSD.org.
 
 I think it would be very nice if we could allow this, but it requires =
 someone to figure out a way to make query-pr.cgi not do searches when =
 indexes by robots. The simple, and also rather clean solution IMO, would =
 probably be to separate the viewing of a single PR, and searching for =
 PR's into different scripts - then we can allow allow robots to index =
 the real PR's without a problem. The problem for that is that somebody =
 has to do it and somebody else has to review before it's put into =
 production.
 
 --=20
 Simon L. B. Nielsen
 Hat: FreeBSD.org admins team