Date: Tue, 3 Oct 2023 12:26:20 GMT From: Wolfram Schneider <wosch@FreeBSD.org> To: doc-committers@FreeBSD.org, dev-commits-doc-all@FreeBSD.org Subject: git: 67b05e8601 - main - block SEO bots from accessing cgi scripts Message-ID: <202310031226.393CQKOn086636@gitrepo.freebsd.org>
next in thread | raw e-mail | index | archive | help
The branch main has been updated by wosch: URL: https://cgit.FreeBSD.org/doc/commit/?id=67b05e86012db7e1be22ab28203f8bc9f6a8148a commit 67b05e86012db7e1be22ab28203f8bc9f6a8148a Author: Wolfram Schneider <wosch@FreeBSD.org> AuthorDate: 2023-10-03 12:26:09 +0000 Commit: Wolfram Schneider <wosch@FreeBSD.org> CommitDate: 2023-10-03 12:26:09 +0000 block SEO bots from accessing cgi scripts --- website/themes/beastie/layouts/robots.txt | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/website/themes/beastie/layouts/robots.txt b/website/themes/beastie/layouts/robots.txt index b3bb9e880b..70cc4362be 100644 --- a/website/themes/beastie/layouts/robots.txt +++ b/website/themes/beastie/layouts/robots.txt @@ -11,3 +11,21 @@ Disallow: /cgi User-agent: CCBot Disallow: /cgi +# SEO +User-agent: SemrushBot +Disallow: /cgi + +User-agent: MJ12bot +Disallow: /cgi + +User-agent: BLEXBot +Disallow: /cgi + +User-agent: dotbot +Disallow: /cgi + +User-agent: SeekportBot +Disallow: /cgi + +User-agent: AhrefsBot +Disallow: /cgi
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?202310031226.393CQKOn086636>