Date: Sat, 2 Sep 2023 07:38:42 GMT From: Wolfram Schneider <wosch@FreeBSD.org> To: doc-committers@FreeBSD.org, dev-commits-doc-all@FreeBSD.org Subject: git: 82284de091 - main - remove non-standard Crawl-delay directive Message-ID: <202309020738.3827cgvH031852@gitrepo.freebsd.org>
index | next in thread | raw e-mail
The branch main has been updated by wosch: URL: https://cgit.FreeBSD.org/doc/commit/?id=82284de091cf1a05aa33538a49d6990ebbc043e3 commit 82284de091cf1a05aa33538a49d6990ebbc043e3 Author: Wolfram Schneider <wosch@FreeBSD.org> AuthorDate: 2023-09-02 07:31:36 +0000 Commit: Wolfram Schneider <wosch@FreeBSD.org> CommitDate: 2023-09-02 07:31:36 +0000 remove non-standard Crawl-delay directive --- documentation/themes/beastie/layouts/robots.txt | 1 - website/themes/beastie/layouts/robots.txt | 1 - 2 files changed, 2 deletions(-) diff --git a/documentation/themes/beastie/layouts/robots.txt b/documentation/themes/beastie/layouts/robots.txt index fd409b7c52..2499651555 100644 --- a/documentation/themes/beastie/layouts/robots.txt +++ b/documentation/themes/beastie/layouts/robots.txt @@ -1,5 +1,4 @@ User-agent: * -#Crawl-delay: 10 Disallow: /statistic Disallow: /doc/ diff --git a/website/themes/beastie/layouts/robots.txt b/website/themes/beastie/layouts/robots.txt index 27f4152f84..8b6d6fb9b0 100644 --- a/website/themes/beastie/layouts/robots.txt +++ b/website/themes/beastie/layouts/robots.txt @@ -1,5 +1,4 @@ User-agent: * -#Crawl-delay: 10 Disallow: /cgi/getmsg.cgi Disallow: /cgi/mailindex.cgi Disallow: /cgi/man.cgi?apropos=2&help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?202309020738.3827cgvH031852>