Python Scrapy无法访问开始URL:DEBUG:Crawled（200）和错误_Python_Web Scraping_Scrapy_Web Crawler

Python Scrapy无法访问开始URL:DEBUG:Crawled（200）和错误

python web-scraping scrapy web-crawler

Python Scrapy无法访问开始URL:DEBUG:Crawled（200）和错误,python,web-scraping,scrapy,web-crawler,Python,Web Scraping,Scrapy,Web Crawler,我正试图用Scrapy从一个运动鞋网站上为一个大学项目搜集信息。其想法是让Scrapy跟踪每只鞋的每个链接，并获取四个信息点（名称、发布日期、零售价格、转售价格）。然后返回到以前的站点，点击下一个链接，再次执行相同的抓取操作。在页面末尾，单击下一页并重复，直到没有更多链接然而，当Scrapy试图到达给定的开始url时，我总是遇到一条调试和错误消息 2020-04-06 11:59:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET http

我正试图用Scrapy从一个运动鞋网站上为一个大学项目搜集信息。其想法是让Scrapy跟踪每只鞋的每个链接，并获取四个信息点（名称、发布日期、零售价格、转售价格）。然后返回到以前的站点，点击下一个链接，再次执行相同的抓取操作。在页面末尾，单击下一页并重复，直到没有更多链接

然而，当Scrapy试图到达给定的开始url时，我总是遇到一条调试和错误消息

2020-04-06 11:59:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://stockx.com/sneakers/release-date?page=1> (referer: None)
2020-04-06 11:59:56 [scrapy.core.scraper] ERROR: Spider error processing <GET https://stockx.com/sneakers/release-date?page=1> (referer: None)

到目前为止，我所尝试的：
- 已将用户代理更改为非默认
- 在settings.py中将ROBOTSTXT_obe更改为False
- 在settings.py中将下载延迟更改为7
- 使用Scrapy Shell获取更多信息，在告知查看（响应）时生成一个空白链接
- 检查要刮取的信息是否经过JavaScript加密（不应）
- 已将开始url更改为开始url，Scrapy不接受
- 通过VPN连接进行

我还尝试了一个简单得多的网站的相同代码结构。但是，我收到了相同的错误消息，这使我得出结论，代码一定有问题

整个跟踪：

2020-04-06 14:33:02 [scrapy.core.engine] INFO: Spider opened
2020-04-06 14:33:02 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-04-06 14:33:02 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-04-06 14:33:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://stockx.com/sneakers/release-date?page=1> (referer: None)
2020-04-06 14:33:03 [scrapy.core.scraper] ERROR: Spider error processing <GET https://stockx.com/sneakers/release-date?page=1> (referer: None)
Traceback (most recent call last):
  File "/Applications/anaconda3/lib/python3.7/site-packages/parsel/selector.py", line 238, in xpath
    **kwargs)
  File "src/lxml/etree.pyx", line 1581, in lxml.etree._Element.xpath
  File "src/lxml/xpath.pxi", line 305, in lxml.etree.XPathElementEvaluator.__call__
  File "src/lxml/xpath.pxi", line 225, in lxml.etree._XPathEvaluatorBase._handle_result
lxml.etree.XPathEvalError: Invalid predicate

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/utils/defer.py", line 117, in iter_errback
    yield next(it)
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/utils/python.py", line 345, in __next__
    return next(self.data)
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/utils/python.py", line 345, in __next__
    return next(self.data)
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
    for x in result:
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/referer.py", line 338, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "/Users/ritterm/Desktop/Data2Dollar_Coding/Group_project/stockx_200406/stockx_200406/spiders/spider_200406.py", line 20, in parse
    links = response.xpath('//a[@class="TileBody-sc-1d2ws1l-0 bKAXcS"/@href').extract()
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/http/response/text.py", line 117, in xpath
    return self.selector.xpath(query, **kwargs)
  File "/Applications/anaconda3/lib/python3.7/site-packages/parsel/selector.py", line 242, in xpath
    six.reraise(ValueError, ValueError(msg), sys.exc_info()[2])
  File "/Applications/anaconda3/lib/python3.7/site-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/Applications/anaconda3/lib/python3.7/site-packages/parsel/selector.py", line 238, in xpath
    **kwargs)
  File "src/lxml/etree.pyx", line 1581, in lxml.etree._Element.xpath
  File "src/lxml/xpath.pxi", line 305, in lxml.etree.XPathElementEvaluator.__call__
  File "src/lxml/xpath.pxi", line 225, in lxml.etree._XPathEvaluatorBase._handle_result
ValueError: XPath error: Invalid predicate in //a[@class="TileBody-sc-1d2ws1l-0 bKAXcS"/@href
2020-04-06 14:33:03 [scrapy.core.engine] INFO: Closing spider (finished)

2020-04-06 14:33:02[刮屑堆芯发动机]信息：卡盘已打开
2020-04-06 14:33:02[scrapy.extensions.logstats]信息：爬网0页（0页/分钟），爬网0项（0项/分钟）
2020-04-06 14:33:02[scrapy.extensions.telnet]信息：telnet控制台监听127.0.0.1:6023
2020-04-06 14:33:03[刮屑核心引擎]调试：爬网（200）（参考：无）
2020-04-06 14:33:03[刮片机]错误：十字轴错误处理（参考：无）
回溯（最近一次呼叫最后一次）：
xpath中的文件“/Applications/anaconda3/lib/python3.7/site packages/parsel/selector.py”，第238行
**kwargs）
lxml.etree.\u Element.xpath中的文件“src/lxml/etree.pyx”，第1581行
文件“src/lxml/xpath.pxi”，第305行，位于lxml.etree.XPathElementEvaluator中__
文件“src/lxml/xpath.pxi”，第225行，在lxml.etree.\u xpatheevaluorbase.\u handle\u result中
lxml.etree.xpathevaleror:无效谓词
在处理上述异常期间，发生了另一个异常：
回溯（最近一次呼叫最后一次）：
文件“/Applications/anaconda3/lib/python3.7/site packages/scrapy/utils/defer.py”，第117行，在iter\u errback中
下一个（it）
文件“/Applications/anaconda3/lib/python3.7/site packages/scrapy/utils/python.py”，第345行，下一页__
返回下一个（self.data）
文件“/Applications/anaconda3/lib/python3.7/site packages/scrapy/utils/python.py”，第345行，下一页__
返回下一个（self.data）
文件“/Applications/anaconda3/lib/python3.7/site packages/scrapy/core/spidermw.py”，第64行，可评估
对于iterable中的r：
文件“/Applications/anaconda3/lib/python3.7/site packages/scrapy/spidermiddleware/offsite.py”，第29行，进程中输出
对于结果中的x：
文件“/Applications/anaconda3/lib/python3.7/site packages/scrapy/core/spidermw.py”，第64行，可评估
对于iterable中的r：
文件“/Applications/anaconda3/lib/python3.7/site packages/scrapy/spidermiddleware/referer.py”，第338行，在
返回（_set_referer（r）表示结果中的r或（））
文件“/Applications/anaconda3/lib/python3.7/site packages/scrapy/core/spidermw.py”，第64行，可评估
对于iterable中的r：
文件“/Applications/anaconda3/lib/python3.7/site packages/scrapy/spidermiddleware/urlength.py”，第37行，在
返回（结果中的r表示r或（）如果_过滤器（r））
文件“/Applications/anaconda3/lib/python3.7/site packages/scrapy/core/spidermw.py”，第64行，可评估
对于iterable中的r：
文件“/Applications/anaconda3/lib/python3.7/site packages/scrapy/spidermiddleware/depth.py”，第58行，in
返回（结果中的r表示r或（）如果_过滤器（r））
文件“/Applications/anaconda3/lib/python3.7/site packages/scrapy/core/spidermw.py”，第64行，可评估
对于iterable中的r：
文件“/Users/ritterm/Desktop/Data2Dollar\u Coding/Group\u project/stockx\u 200406/stockx\u 200406/spider/spider\u 200406.py”，第20行，解析
links=response.xpath（'//a[@class=“TileBody-sc-1d2ws1l-0 bKAXcS”/@href'）.extract（）
xpath中的文件“/Applications/anaconda3/lib/python3.7/site packages/scrapy/http/response/text.py”，第117行
返回self.selector.xpath（查询，**kwargs）
xpath中的文件“/Applications/anaconda3/lib/python3.7/site packages/parsel/selector.py”，第242行
重新发送（ValueError，ValueError（msg），sys.exc_info（）[2]）
文件“/Applications/anaconda3/lib/python3.7/site-packages/six.py”，第692行，重新登录
通过_回溯（tb）提升值
xpath中的文件“/Applications/anaconda3/lib/python3.7/site packages/parsel/selector.py”，第238行
**kwargs）
lxml.etree.\u Element.xpath中的文件“src/lxml/etree.pyx”，第1581行
文件“src/lxml/xpath.pxi”，第305行，位于lxml.etree.XPathElementEvaluator中__
文件“src/lxml/xpath.pxi”，第225行，在lxml.etree.\u xpatheevaluorbase.\u handle\u result中
ValueError:XPath错误：在//a[@class=“TileBody-sc-1d2ws1l-0 bKAXcS”/@href中的谓词无效
2020-04-06 14:33:03[刮屑堆芯发动机]信息：关闭十字轴（已完成）

非常感谢您的任何建议和想法。

您的代码中存在多个bug，这些bug会阻止scrapy成功

首先，如前所述，将允许的_域更正为

allowed_domains=['stockx.com']

或完全删除该行

此外，您的基本URL错误。请将其更改为：

BASE\u URL=https://stockx.com“

此外，正如堆栈跟踪所显示的，xpath中存在错误。我通过使用一个非常简单的css选择器来获取指向每个鞋页的链接，从而解决了这个问题：

response.css（'.browse grid a:：attr（href）'）。extract（）

总之，以下代码应该完全满足您的要求：

import scrapy
蜘蛛纲200
2020-04-06 14:33:02 [scrapy.core.engine] INFO: Spider opened
2020-04-06 14:33:02 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-04-06 14:33:02 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-04-06 14:33:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://stockx.com/sneakers/release-date?page=1> (referer: None)
2020-04-06 14:33:03 [scrapy.core.scraper] ERROR: Spider error processing <GET https://stockx.com/sneakers/release-date?page=1> (referer: None)
Traceback (most recent call last):
  File "/Applications/anaconda3/lib/python3.7/site-packages/parsel/selector.py", line 238, in xpath
    **kwargs)
  File "src/lxml/etree.pyx", line 1581, in lxml.etree._Element.xpath
  File "src/lxml/xpath.pxi", line 305, in lxml.etree.XPathElementEvaluator.__call__
  File "src/lxml/xpath.pxi", line 225, in lxml.etree._XPathEvaluatorBase._handle_result
lxml.etree.XPathEvalError: Invalid predicate

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/utils/defer.py", line 117, in iter_errback
    yield next(it)
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/utils/python.py", line 345, in __next__
    return next(self.data)
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/utils/python.py", line 345, in __next__
    return next(self.data)
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
    for x in result:
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/referer.py", line 338, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "/Users/ritterm/Desktop/Data2Dollar_Coding/Group_project/stockx_200406/stockx_200406/spiders/spider_200406.py", line 20, in parse
    links = response.xpath('//a[@class="TileBody-sc-1d2ws1l-0 bKAXcS"/@href').extract()
  File "/Applications/anaconda3/lib/python3.7/site-packages/scrapy/http/response/text.py", line 117, in xpath
    return self.selector.xpath(query, **kwargs)
  File "/Applications/anaconda3/lib/python3.7/site-packages/parsel/selector.py", line 242, in xpath
    six.reraise(ValueError, ValueError(msg), sys.exc_info()[2])
  File "/Applications/anaconda3/lib/python3.7/site-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/Applications/anaconda3/lib/python3.7/site-packages/parsel/selector.py", line 238, in xpath
    **kwargs)
  File "src/lxml/etree.pyx", line 1581, in lxml.etree._Element.xpath
  File "src/lxml/xpath.pxi", line 305, in lxml.etree.XPathElementEvaluator.__call__
  File "src/lxml/xpath.pxi", line 225, in lxml.etree._XPathEvaluatorBase._handle_result
ValueError: XPath error: Invalid predicate in //a[@class="TileBody-sc-1d2ws1l-0 bKAXcS"/@href
2020-04-06 14:33:03 [scrapy.core.engine] INFO: Closing spider (finished)

allowed_domains = ['stockx.com']