Nginx Googlebot忽略X-Robots-Tag
我们有一些PDF文件在我们的网站上，我们不希望机器人索引 我已将以下内容添加到我们的NGINX配置中Nginx Googlebot忽略X-Robots-Tag,nginx,Nginx,我们有一些PDF文件在我们的网站上，我们不希望机器人索引 我已将以下内容添加到我们的NGINX配置中 location ~* \.pdf$ { add_header X-Robots-Tag "noindex, noarchive, nosnippet, nofollow"; } 但是日志文件不断显示如下示例 66.249.79.230 - - [08/Oct/2020:13:13:13 +0100] "GET /341282.pdf HTTP/1.1&
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, noarchive, nosnippet, nofollow";
}
66.249.79.230 - - [08/Oct/2020:13:13:13 +0100] "GET /341282.pdf HTTP/1.1" 301 194 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.56 - - [08/Oct/2020:13:14:55 +0100] "GET /341282.pdf HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"