Web scraping 我在抓取此页时遇到scrapy超时错误

Web scraping 我在抓取此页时遇到scrapy超时错误,web-scraping,scrapy,web-crawler,scrapy-spider,Web Scraping,Scrapy,Web Crawler,Scrapy Spider,我无法抓取此页面https://www.adidas.pe/,刮擦式爬行我的蜘蛛返回: 2018-12-17 15:36:39 [scrapy.core.engine] INFO: Spider opened 2018-12-17 15:36:39 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2018-12-17 15:36:39 [sc

我无法抓取此页面
https://www.adidas.pe/
刮擦式爬行我的蜘蛛
返回:

2018-12-17 15:36:39 [scrapy.core.engine] INFO: Spider opened
2018-12-17 15:36:39 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-12-17 15:36:39 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6024
2018-12-17 15:36:39 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.adidas.pe/> from <GET http://adidas.pe/>
2018-12-17 15:37:39 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-12-17 15:38:39 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

而且不起作用

您可以尝试在
settings.py
中更改
USER\u AGENT
,它对我有效。我的
设置.py

COOKIES_ENABLED = True
ROBOTSTXT_OBEY = False
 # -*- coding: utf-8 -*-

# Scrapy settings for adidas project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://doc.scrapy.org/en/latest/topics/settings.html
#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://doc.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'adidas'

SPIDER_MODULES = ['adidas.spiders']
NEWSPIDER_MODULE = 'adidas.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'