Scrapy 爬行器爬行器：爬行器错误处理引发NotImpmentedError

我一直在试着让我的头脑清醒过来，但我没有太多的运气超越基本的东西。当我运行spider时，我会得到一个spider错误处理页面和一个尚未实现的spider豁免，但是如果我使用

scrapy fetch

则会输出html响应，因此不是说站点不可用。下面包括输出以及我的项目、spider和设置值

Items.py

class MycrawlerItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
title = scrapy.Field()
files = scrapy.Field()
file_urls = scrapy.Field()

import scrapy
from scrapy.spiders import Rule
from bs4 import BeautifulSoup
from scrapy.linkextractors import LinkExtractor
from librarycrawler.items import LibrarycrawlerItem
class CrawlSpider(scrapy.Spider):
    name = "mycrawler"
    allowed_domains = ["example.com"]
    start_urls = [
        "http://www.example.com"
    ]
    #LinkExtractor(),
    rules = (
        Rule(LinkExtractor(),callback='scrape_page', follow=True)
    )

    def scrape_page(self,response):
        page_soup = BeautifulSoup(response.body,"html.parser")
        ScrapedPageTitle = page_soup.title.get_text()
        item = LibrarycrawlerItem()
        item['title'] =ScrapedPageTitle
        item['file_urls'] = response.url

        yield item

ITEM_PIPELINES = { 
'scrapy.pipelines.files.FilesPipeline':300,
}

FILES_STORE = 'C:\MySpider\mycrawler\ExtractedText'

mycrawler.py

class MycrawlerItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
title = scrapy.Field()
files = scrapy.Field()
file_urls = scrapy.Field()

import scrapy
from scrapy.spiders import Rule
from bs4 import BeautifulSoup
from scrapy.linkextractors import LinkExtractor
from librarycrawler.items import LibrarycrawlerItem
class CrawlSpider(scrapy.Spider):
    name = "mycrawler"
    allowed_domains = ["example.com"]
    start_urls = [
        "http://www.example.com"
    ]
    #LinkExtractor(),
    rules = (
        Rule(LinkExtractor(),callback='scrape_page', follow=True)
    )

    def scrape_page(self,response):
        page_soup = BeautifulSoup(response.body,"html.parser")
        ScrapedPageTitle = page_soup.title.get_text()
        item = LibrarycrawlerItem()
        item['title'] =ScrapedPageTitle
        item['file_urls'] = response.url

        yield item

ITEM_PIPELINES = { 
'scrapy.pipelines.files.FilesPipeline':300,
}

FILES_STORE = 'C:\MySpider\mycrawler\ExtractedText'

设置.py

class MycrawlerItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
title = scrapy.Field()
files = scrapy.Field()
file_urls = scrapy.Field()

import scrapy
from scrapy.spiders import Rule
from bs4 import BeautifulSoup
from scrapy.linkextractors import LinkExtractor
from librarycrawler.items import LibrarycrawlerItem
class CrawlSpider(scrapy.Spider):
    name = "mycrawler"
    allowed_domains = ["example.com"]
    start_urls = [
        "http://www.example.com"
    ]
    #LinkExtractor(),
    rules = (
        Rule(LinkExtractor(),callback='scrape_page', follow=True)
    )

    def scrape_page(self,response):
        page_soup = BeautifulSoup(response.body,"html.parser")
        ScrapedPageTitle = page_soup.title.get_text()
        item = LibrarycrawlerItem()
        item['title'] =ScrapedPageTitle
        item['file_urls'] = response.url

        yield item

ITEM_PIPELINES = { 
'scrapy.pipelines.files.FilesPipeline':300,
}

FILES_STORE = 'C:\MySpider\mycrawler\ExtractedText'

终端输出

[scrapy] C:\MySpider\mycrawler>scrapy crawl mycrawler -o mycrawler.csv
2016-06-03 16:11:47 [scrapy] INFO: Scrapy 1.0.3 started (bot: mycrawler)
2016-06-03 16:11:47 [scrapy] INFO: Optional features available: ssl, http11, boto
2016-06-03 16:11:47 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'mycrawler.spiders', 'FEED_URI': 'mycrawler.csv', 'DEPTH_LIMIT': 3, 'SPIDER_MODULES': ['mycrawler.spiders'], 'BOT_NAME': 'mycrawler', 'USER_AGENT': 'mycrawler(+http://www.example.com)', 'FEED_FORMAT': 'csv'}
2016-06-03 16:11:48 [scrapy] INFO: Enabled extensions: CloseSpider, FeedExporter, TelnetConsole, LogStats, CoreStats, SpiderState
2016-06-03 16:11:48 [boto] DEBUG: Retrieving credentials from metadata server.
2016-06-03 16:11:49 [boto] ERROR: Caught exception reading instance dataTraceback (most recent call last):
  File "C:\Anaconda3\envs\scrapy\lib\site-packages\boto\utils.py", line 210, inretry_url
    r = opener.open(req, timeout=timeout)
  File "C:\Anaconda3\envs\scrapy\lib\urllib2.py", line 431, in open
    response = self._open(req, data)
  File "C:\Anaconda3\envs\scrapy\lib\urllib2.py", line 449, in _open
    '_open', req)
  File "C:\Anaconda3\envs\scrapy\lib\urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "C:\Anaconda3\envs\scrapy\lib\urllib2.py", line 1227, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "C:\Anaconda3\envs\scrapy\lib\urllib2.py", line 1197, in do_open
    raise URLError(err)
URLError: <urlopen error timed out>
2016-06-03 16:11:49 [boto] ERROR: Unable to read instance data, giving up
2016-06-03 16:11:49 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddl
eware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultH
eadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMidd
leware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-06-03 16:11:49 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddlewa
re, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-06-03 16:11:49 [scrapy] INFO: Enabled item pipelines: FilesPipeline
2016-06-03 16:11:49 [scrapy] INFO: Spider opened
2016-06-03 16:11:49 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 i
tems (at 0 items/min)
2016-06-03 16:11:49 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-06-03 16:11:49 [scrapy] DEBUG: Redirecting (meta refresh) to <GET http://myexample.com> from <GEThttp://myexample.com>
2016-06-03 16:11:50 [scrapy] DEBUG: Crawled (200) <GET http://myexample.com> (referer: None)
2016-06-03 16:11:50 [scrapy] ERROR: Spider error processing <GET http://www.example.com> (referer: None)
Traceback (most recent call last):  File "C:\Anaconda3\envs\scrapy\lib\site-packages\twisted\internet\defer.py", line 588, in _runCallbacks    current.result = callback(current.result, *args, **kw)
  File "C:\Anaconda3\envs\scrapy\lib\site-packages\scrapy\spiders\__init__.py",line 76, in parse    raise NotImplementedErrorNotImplementedError
2016-06-03 16:11:50 [scrapy] INFO: Closing spider (finished)
2016-06-03 16:11:50 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 449,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 23526,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 2,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 6, 3, 15, 11, 50, 227000),
 'log_count/DEBUG': 4,
 'log_count/ERROR': 3,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'spider_exceptions/NotImplementedError': 1,
 'start_time': datetime.datetime(2016, 6, 3, 15, 11, 49, 722000)}
2016-06-03 16:11:50 [scrapy] INFO: Spider closed (finished)

[scrapy]C:\MySpider\mycrawler>scrapy crawler mycrawler-o mycrawler.csv
2016-06-03 16:11:47[scrapy]信息：scrapy 1.0.3已启动（机器人程序：mycrawler）
2016-06-03 16:11:47[scrapy]信息：可选功能：ssl、http11、boto
2016-06-03 16:11:47[scrapy]信息：覆盖的设置：{'NEWSPIDER_MODULE'：'mycrawler.spiders'，'FEED_URI'：'mycrawler.csv'，'DEPTH_LIMIT'：3，'SPIDER_MODULES'：['mycrawler.spiders'，'BOT NAME'：'mycrawler'，'USER_AGENT'：'mycrawler(+http://www.example.com)'，'FEED_FORMAT'：'csv'}
2016-06-03 16:11:48[scrapy]信息：启用的扩展：CloseSpider、FeedExporter、TelnetConsole、LogStats、CoreStats、SpiderState
2016-06-03 16:11:48[boto]调试：从元数据服务器检索凭据。
2016-06-03 16:11:49[boto]错误：捕获异常读取实例数据回溯（最近一次调用）：
文件“C:\Anaconda3\envs\scrapy\lib\site packages\boto\utils.py”，第210行，inrry\u url
r=打开器。打开（请求，超时=超时）
打开文件“C:\Anaconda3\envs\scrapy\lib\urllib2.py”，第431行
响应=自身打开（请求，数据）
文件“C:\Anaconda3\envs\scrapy\lib\urllib2.py”，第449行，处于打开状态
"开放",
文件“C:\Anaconda3\envs\scrapy\lib\urllib2.py”，第409行，在调用链中
结果=func（*args）
文件“C:\Anaconda3\envs\scrapy\lib\urllib2.py”，第1227行，在http\u open中
返回self.do_open（httplib.HTTPConnection，req）
文件“C:\Anaconda3\envs\scrapy\lib\urllib2.py”，第1197行，打开
引发URL错误（err）
URL错误：
2016-06-03 16:11:49[boto]错误：无法读取实例数据，放弃
2016-06-03 16:11:49[scrapy]信息：已启用的下载程序中间件：httpauthmidl
软件，DownloadTimeoutMiddleware，UserAgentMiddleware，RetryMiddleware，DefaultH
eadersMiddleware，MetaRefreshMiddleware，HttpCompressionMiddleware，RedirectMiddle
leware、CookiesMiddleware、ChunkedTransferMiddleware、DownloaderStats
2016-06-03 16:11:49[scrapy]信息：启用的蜘蛛中间件：HttpErrorMiddlewa
re、OffsiteMiddleware、RefererMiddle、UrlLengthMiddleware、DepthMiddleware
2016-06-03 16:11:49[碎片]信息：已启用的项目管道：文件管道
2016-06-03 16:11:49[刮屑]信息：蜘蛛打开
2016-06-03 16:11:49[抓取]信息：抓取0页（以0页/分钟的速度），抓取0页/分钟
tems（0项/分钟）
2016-06-03 16:11:49[scrapy]调试：Telnet控制台监听127.0.0.1:6023
2016-06-03 16:11:49[scrapy]调试：重定向（元刷新）到
2016-06-03 16:11:50[scrapy]调试：爬网（200）（参考：无）
2016-06-03 16:11:50[scrapy]错误：蜘蛛错误处理（参考：无）
回溯（最近一次调用）：文件“C:\Anaconda3\envs\scrapy\lib\site packages\twisted\internet\defer.py”，第588行，在_runcallbackscurrent.result=callback（current.result，*args，**kw）中
文件“C:\Anaconda3\envs\scrapy\lib\site packages\scrapy\spiders\\uuuu init\uuuu.py”，第76行，位于parse raise NotImplementedError或NotImplementedError中
2016-06-03 16:11:50[scrapy]信息：关闭卡盘（已完成）
2016-06-03 16:11:50[scrapy]信息：倾销scrapy统计数据：
{'downloader/request_bytes'：449，
“下载程序/请求计数”：2，
“下载器/请求\方法\计数/获取”：2，
“downloader/response_字节”：23526，
“下载程序/响应计数”：2，
“下载程序/响应状态\计数/200”：2，
“完成原因”：“完成”，
“完成时间”：datetime.datetime（2016,6,3,15,11,50,227000），
“日志计数/调试”：4，
“日志计数/错误”：3，
“日志计数/信息”：7，
“响应\u已接收\u计数”：1，
“调度程序/出列”：2，
“调度程序/出列/内存”：2，
“调度程序/排队”：2，
“调度程序/排队/内存”：2，
“spider_异常/未实现错误”：1，
“开始时间”：datetime.datetime（2016,6,3,15,11,49722000）}
2016-06-03 16:11:50[刮擦]信息：蜘蛛网关闭（完成）

如果您需要该功能，您需要从scrapy的爬行蜘蛛中创建子类，例如：

来自scrapy.item导入字段，item
从scrapy.LinkExtractor导入LinkExtractor
从scrapy.spider导入规则
从scrapy.spider.crawl导入爬行蜘蛛
类别图书馆（项目）：
标题=字段（）
文件\u URL=字段（）
类别MyCrawlSpider（爬行蜘蛛）：
名称='sample'
允许的_域=['example.com'，'iana.org']
起始URL=['http://www.example.com']
规则=(
规则（LinkExtractor（），callback='scrape_page'），
)
def刮水页面（自我，响应）：
item=LibraryCrawleItem（）
item['title']=response.xpath（'//title/text（）'）。extract_first（）
项['file_url']=response.url
收益项目

为了更好地理解规则是如何工作的，请参阅，顺便说一句，您也可以在

parse

方法中使用

LinkExtractor

，而无需对

爬行蜘蛛进行子分类