From owner-dev-commits-doc-all@freebsd.org Fri Aug 13 22:57:40 2021 Return-Path: Delivered-To: dev-commits-doc-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 67E8366179F for ; Fri, 13 Aug 2021 22:57:40 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gmf8h2SwDz4sQQ; Fri, 13 Aug 2021 22:57:40 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3D2A616A8A; Fri, 13 Aug 2021 22:57:40 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 17DMvetr007560; Fri, 13 Aug 2021 22:57:40 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 17DMveYC007559; Fri, 13 Aug 2021 22:57:40 GMT (envelope-from git) Date: Fri, 13 Aug 2021 22:57:40 GMT Message-Id: <202108132257.17DMveYC007559@gitrepo.freebsd.org> To: doc-committers@FreeBSD.org, dev-commits-doc-all@FreeBSD.org From: "Danilo G. Baio" Subject: git: 6af7769111 - main - documentation/robots.txt: Disallow Past Releases Documentation MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: dbaio X-Git-Repository: doc X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 6af77691110d19019b389fad3a0700286fdf0ef1 Auto-Submitted: auto-generated X-BeenThere: dev-commits-doc-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the doc repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Aug 2021 22:57:40 -0000 The branch main has been updated by dbaio: URL: https://cgit.FreeBSD.org/doc/commit/?id=6af77691110d19019b389fad3a0700286fdf0ef1 commit 6af77691110d19019b389fad3a0700286fdf0ef1 Author: Danilo G. Baio AuthorDate: 2021-08-08 14:12:03 +0000 Commit: Danilo G. Baio CommitDate: 2021-08-13 22:51:16 +0000 documentation/robots.txt: Disallow Past Releases Documentation It was noticed that Google/DDG sometimes are sending users to the past releases documentation. To avoid misinformation, disallow this archive from the search engines. This robots.txt only covers docs.FreeBSD.org and not www.FreeBSD.org, so also remove unnecessary items. Differential Revision: https://reviews.freebsd.org/D31459 --- documentation/themes/beastie/layouts/robots.txt | 19 +------------------ 1 file changed, 1 insertion(+), 18 deletions(-) diff --git a/documentation/themes/beastie/layouts/robots.txt b/documentation/themes/beastie/layouts/robots.txt index b94bcd43cf..f59c629169 100644 --- a/documentation/themes/beastie/layouts/robots.txt +++ b/documentation/themes/beastie/layouts/robots.txt @@ -1,19 +1,2 @@ -# $FreeBSD$ - User-agent: * -#Crawl-delay: 10 -Disallow: /cgi/confirm-code.cgi -Disallow: /cgi/cvsweb.cgi -Disallow: /cgi/dosendpr.cgi -Disallow: /cgi/getmsg.cgi -Disallow: /cgi/mailindex.cgi -Disallow: /cgi/man.cgi?apropos=2& -Disallow: /cgi/mid.cgi -Disallow: /cgi/mirror.cgi -Disallow: /cgi/missing_handler.cgi -Disallow: /cgi/ports.cgi -Disallow: /cgi/query-pr-summary.cgi -Disallow: /cgi/query-pr.cgi -Disallow: /cgi/search.cgi -Disallow: /cgi/url.cgi -Disallow: /statistic +Disallow: /doc/