From owner-dev-commits-doc-all@freebsd.org Tue Jan 26 19:17:23 2021 Return-Path: Delivered-To: dev-commits-doc-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 98DD04FD418 for ; Tue, 26 Jan 2021 19:17:23 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DQGhM411gz3qCl; Tue, 26 Jan 2021 19:17:23 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 7C6576FD3; Tue, 26 Jan 2021 19:17:23 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 10QJHNHH031759; Tue, 26 Jan 2021 19:17:23 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 10QJHNWt031758; Tue, 26 Jan 2021 19:17:23 GMT (envelope-from git) Date: Tue, 26 Jan 2021 19:17:23 GMT Message-Id: <202101261917.10QJHNWt031758@gitrepo.freebsd.org> To: doc-committers@FreeBSD.org, dev-commits-doc-all@FreeBSD.org From: Sergio Carlavilla Delgado Subject: git: 7808bd27ab - main - Enable robots.txt in docs.freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: carlavilla X-Git-Repository: doc X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 7808bd27abab51b4602005cdcd593ac18e59256c Auto-Submitted: auto-generated X-BeenThere: dev-commits-doc-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the doc repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jan 2021 19:17:23 -0000 The branch main has been updated by carlavilla: URL: https://cgit.FreeBSD.org/doc/commit/?id=7808bd27abab51b4602005cdcd593ac18e59256c commit 7808bd27abab51b4602005cdcd593ac18e59256c Author: Sergio Carlavilla Delgado AuthorDate: 2021-01-26 19:12:37 +0000 Commit: Sergio Carlavilla Delgado CommitDate: 2021-01-26 19:17:05 +0000 Enable robots.txt in docs.freebsd.org Enable again robots.txt for docs.freebsd.org PR: 253031 Submitted by: wosch@ --- documentation/config/_default/config.toml | 1 + documentation/themes/beastie/layouts/robots.txt | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/documentation/config/_default/config.toml b/documentation/config/_default/config.toml index ed455cd1b0..9d0acfa24e 100644 --- a/documentation/config/_default/config.toml +++ b/documentation/config/_default/config.toml @@ -12,6 +12,7 @@ disableKinds = [ "taxonomy", "taxonomyTerm" ] authors = [ "carlavilla@FreeBSD.org" ] preserveTOC = true ignoreFiles = [ "chapters-order.adoc$", "toc.adoc$", "toc-tables.adoc$", "toc-figures.adoc$", "toc-examples.adoc$", "toc-1.adoc$", "toc-2.adoc$", "toc-3.adoc$", "toc-4.adoc$", "toc-5.adoc$", "books.adoc$", "chapter.adoc$" ] +enableRobotsTXT = true [params] websiteURL = "https://www-dev.freebsd.org/" diff --git a/documentation/themes/beastie/layouts/robots.txt b/documentation/themes/beastie/layouts/robots.txt new file mode 100644 index 0000000000..b94bcd43cf --- /dev/null +++ b/documentation/themes/beastie/layouts/robots.txt @@ -0,0 +1,19 @@ +# $FreeBSD$ + +User-agent: * +#Crawl-delay: 10 +Disallow: /cgi/confirm-code.cgi +Disallow: /cgi/cvsweb.cgi +Disallow: /cgi/dosendpr.cgi +Disallow: /cgi/getmsg.cgi +Disallow: /cgi/mailindex.cgi +Disallow: /cgi/man.cgi?apropos=2& +Disallow: /cgi/mid.cgi +Disallow: /cgi/mirror.cgi +Disallow: /cgi/missing_handler.cgi +Disallow: /cgi/ports.cgi +Disallow: /cgi/query-pr-summary.cgi +Disallow: /cgi/query-pr.cgi +Disallow: /cgi/search.cgi +Disallow: /cgi/url.cgi +Disallow: /statistic