Date: Sun, 21 Aug 2011 09:10:11 GMT From: "Simon L. B. Nielsen" <simon@FreeBSD.org> To: freebsd-www@FreeBSD.org Subject: Re: www/159652: PR is not visible to search engines Message-ID: <201108210910.p7L9ABaO012315@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR www/159652; it has been noted by GNATS. From: "Simon L. B. Nielsen" <simon@FreeBSD.org> To: Marcin Wisnicki <mwisnicki+freebsd@gmail.com> Cc: freebsd-gnats-submit@FreeBSD.org Subject: Re: www/159652: PR is not visible to search engines Date: Sun, 21 Aug 2011 11:05:37 +0200 On 10 Aug 2011, at 19:19, Marcin Wisnicki wrote: > Currently www.freebsd.org/robots.txt disallows access to /cgi/ and = that includes query-pr.cgi which makes bugs invisible to search engines = :( The problem is that query-cgi allows searches which are quiet heavy = resource wise (taking minutes to finish). If we just remove the = robots.txt from that area and the query cgi any reasonable amount of = hardware won't be enough. We even have extra protection which only = allows two query-pr's to run at once to avoid killing www.FreeBSD.org. I think it would be very nice if we could allow this, but it requires = someone to figure out a way to make query-pr.cgi not do searches when = indexes by robots. The simple, and also rather clean solution IMO, would = probably be to separate the viewing of a single PR, and searching for = PR's into different scripts - then we can allow allow robots to index = the real PR's without a problem. The problem for that is that somebody = has to do it and somebody else has to review before it's put into = production. --=20 Simon L. B. Nielsen Hat: FreeBSD.org admins team
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201108210910.p7L9ABaO012315>