Date: Wed, 23 Aug 2023 06:17:52 GMT From: Wolfram Schneider <wosch@FreeBSD.org> To: doc-committers@FreeBSD.org, dev-commits-doc-all@FreeBSD.org Subject: git: e09c06b9c2 - main - block CCBot (Common Crawl) from accessing cgi scripts Message-ID: <202308230617.37N6Hq2N096858@gitrepo.freebsd.org>
next in thread | raw e-mail | index | archive | help
The branch main has been updated by wosch: URL: https://cgit.FreeBSD.org/doc/commit/?id=e09c06b9c2fa70e4884edae327d294c5efc3eb02 commit e09c06b9c2fa70e4884edae327d294c5efc3eb02 Author: Wolfram Schneider <wosch@FreeBSD.org> AuthorDate: 2023-08-23 06:17:36 +0000 Commit: Wolfram Schneider <wosch@FreeBSD.org> CommitDate: 2023-08-23 06:17:36 +0000 block CCBot (Common Crawl) from accessing cgi scripts --- website/themes/beastie/layouts/robots.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/website/themes/beastie/layouts/robots.txt b/website/themes/beastie/layouts/robots.txt index 9bec7f76cf..1248457c75 100644 --- a/website/themes/beastie/layouts/robots.txt +++ b/website/themes/beastie/layouts/robots.txt @@ -14,3 +14,6 @@ Disallow: /statistic User-agent: GPTBot Disallow: /cgi +User-agent: CCBot +Disallow: /cgi +
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?202308230617.37N6Hq2N096858>