Python Scrapy Spider不保存数据_Python_Web Scraping_Terminal_Scrapy_Scrapy Spider

Python Scrapy Spider不保存数据

python web-scraping terminal scrapy

Python Scrapy Spider不保存数据,python,web-scraping,terminal,scrapy,scrapy-spider,Python,Web Scraping,Terminal,Scrapy,Scrapy Spider,我正在尝试使用Scrapy将篮球队的时间表保存到CSV文件中。我在这些文件中编写了以下代码：设置.py BOT_NAME = 'test_project' SPIDER_MODULES = ['test_project.spiders'] NEWSPIDER_MODULE = 'test_project.spiders' FEED_FORMAT = "csv" FEED_URI = "cportboys.csv" # Crawl responsibly by identifying yo

我正在尝试使用Scrapy将篮球队的时间表保存到CSV文件中。我在这些文件中编写了以下代码：

设置.py

BOT_NAME = 'test_project'

SPIDER_MODULES = ['test_project.spiders']
NEWSPIDER_MODULE = 'test_project.spiders'

FEED_FORMAT = "csv"
FEED_URI = "cportboys.csv"

# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'test_project (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

import scrapy


class KhsaabotSpider(scrapy.Spider):
    name = 'khsaabot'
    allowed_domains = ['https://scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978']
    start_urls = ['http://https://scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978/']

def parse(self, response):
    date = response.css('.mdate::text').extract()
    opponent = response.css('.opponent::text').extract()
    place = response.css('.schedule-loc::text').extract()


    for item in zip(date,opponent,place):
        scraped_info = {
            'date' : item[0],
            'opponent' : item[1],
            'place' : item[2],
        }

        yield scraped_info

khsaabot.py

BOT_NAME = 'test_project'

SPIDER_MODULES = ['test_project.spiders']
NEWSPIDER_MODULE = 'test_project.spiders'

FEED_FORMAT = "csv"
FEED_URI = "cportboys.csv"

# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'test_project (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

import scrapy


class KhsaabotSpider(scrapy.Spider):
    name = 'khsaabot'
    allowed_domains = ['https://scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978']
    start_urls = ['http://https://scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978/']

def parse(self, response):
    date = response.css('.mdate::text').extract()
    opponent = response.css('.opponent::text').extract()
    place = response.css('.schedule-loc::text').extract()


    for item in zip(date,opponent,place):
        scraped_info = {
            'date' : item[0],
            'opponent' : item[1],
            'place' : item[2],
        }

        yield scraped_info

现在，我不确定这里出了什么问题，当我在终端中使用“scrapy crawl khsaabot”运行它时，它没有给出任何错误，并且似乎工作得很好。然而，为了防止终端中发生的问题，我也包括了我在那里得到的输出：

2017-12-27 17:21:49 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: test_project)
2017-12-27 17:21:49 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'test_project', 'FEED_FORMAT': 'csv', 'FEED_URI': 'cportboys.csv', 'NEWSPIDER_MODULE': 'test_project.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['test_project.spiders']}
2017-12-27 17:21:49 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2017-12-27 17:21:49 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-12-27 17:21:49 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-12-27 17:21:49 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-12-27 17:21:49 [scrapy.core.engine] INFO: Spider opened
2017-12-27 17:21:49 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-12-27 17:21:49 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-12-27 17:21:49 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://https/robots.txt> (failed 1 times): DNS lookup failed: no results for hostname lookup: https.
2017-12-27 17:21:49 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://https/robots.txt> (failed 2 times): DNS lookup failed: no results for hostname lookup: https.
2017-12-27 17:21:49 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://https/robots.txt> (failed 3 times): DNS lookup failed: no results for hostname lookup: https.
2017-12-27 17:21:49 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET http://https/robots.txt>: DNS lookup failed: no results for hostname lookup: https.
Traceback (most recent call last):
  File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/internet/endpoints.py", line 954, in startConnectionAttempts
    "no results for hostname lookup: {}".format(self._hostStr)
twisted.internet.error.DNSLookupError: DNS lookup failed: no results for hostname lookup: https.
2017-12-27 17:21:49 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://https//scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978/> (failed 1 times): DNS lookup failed: no results for hostname lookup: https.
2017-12-27 17:21:49 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://https//scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978/> (failed 2 times): DNS lookup failed: no results for hostname lookup: https.
2017-12-27 17:21:49 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://https//scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978/> (failed 3 times): DNS lookup failed: no results for hostname lookup: https.
2017-12-27 17:21:49 [scrapy.core.scraper] ERROR: Error downloading <GET http://https//scoreboard.12dt.com/scoreboard/khsaa/kybbk17/?id=51978/>
Traceback (most recent call last):
  File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/Users/devsandbox/anaconda3/lib/python3.6/site-packages/twisted/internet/endpoints.py", line 954, in startConnectionAttempts
    "no results for hostname lookup: {}".format(self._hostStr)
twisted.internet.error.DNSLookupError: DNS lookup failed: no results for hostname lookup: https.
2017-12-27 17:21:49 [scrapy.core.engine] INFO: Closing spider (finished)
2017-12-27 17:21:49 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 6,
 'downloader/exception_type_count/twisted.internet.error.DNSLookupError': 6,
 'downloader/request_bytes': 1416,
 'downloader/request_count': 6,
 'downloader/request_method_count/GET': 6,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 12, 27, 23, 21, 49, 579649),
 'log_count/DEBUG': 7,
 'log_count/ERROR': 2,
 'log_count/INFO': 7,
 'memusage/max': 50790400,
 'memusage/startup': 50790400,
 'retry/count': 4,
 'retry/max_reached': 2,
 'retry/reason_count/twisted.internet.error.DNSLookupError': 4,
 'scheduler/dequeued': 3,
 'scheduler/dequeued/memory': 3,
 'scheduler/enqueued': 3,
 'scheduler/enqueued/memory': 3,
 'start_time': datetime.datetime(2017, 12, 27, 23, 21, 49, 323652)}
2017-12-27 17:21:49 [scrapy.core.engine] INFO: Spider closed (finished)

2017-12-27 17:21:49[scrapy.utils.log]信息：scrapy 1.4.0已启动（bot:test_项目）
2017-12-27 17:21:49[scrapy.utils.log]信息：覆盖的设置：{'BOT_NAME'：'test_project'，'FEED_FORMAT'：'csv'，'FEED_URI'：'cportboys.csv'，'NEWSPIDER_MODULE'：'test_project.SPIDER'，'ROBOTSTXT_-obe'：True'，'SPIDER_MODULES'：['test_project.SPIDER']
2017-12-27 17:21:49[scrapy.middleware]信息：启用的扩展：
['scrapy.extensions.corestats.corestats'，
'scrapy.extensions.telnet.TelnetConsole'，
'scrapy.extensions.memusage.MemoryUsage'，
'scrapy.extensions.feedexport.FeedExporter'，
'scrapy.extensions.logstats.logstats']
2017-12-27 17:21:49[剪贴簿中间件]信息：启用的下载程序中间件：
['scrapy.downloaderMiddleware.robotstxt.RobotsTxtMiddleware'，
'scrapy.downloaderMiddleware.httpauth.HttpAuthMiddleware'，
'scrapy.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware'，
'scrapy.downloaderMiddleware.defaultheaders.DefaultHeadersMiddleware'，
'scrapy.DownloaderMiddleware.useragent.UserAgentMiddleware'，
'scrapy.DownloaderMiddleware.retry.RetryMiddleware'，
'scrapy.DownloaderMiddleware.redirect.MetaRefreshMiddleware'，
'scrapy.DownloaderMiddleware.httpcompression.HttpCompressionMiddleware'，
'scrapy.DownloaderMiddleware.redirect.RedirectMiddleware'，
“scrapy.DownloaderMiddleware.cookies.CookiesMiddleware”，
'scrapy.downloadermiddleware.httpproxy.HttpProxyMiddleware'，
'scrapy.downloadermiddleware.stats.DownloaderStats']
2017-12-27 17:21:49[剪贴簿中间件]信息：启用的蜘蛛中间件：
['scrapy.spidermiddleware.httperror.httperror中间件'，
'刮皮.SpiderMiddleware.场外.场外Iddleware'，
“scrapy.Spidermiddleware.referer.RefererMiddleware”，
'scrapy.spiderMiddleware.urllength.UrlLengthMiddleware'，
'scrapy.spidermiddleware.depth.DepthMiddleware']
2017-12-27 17:21:49[scrapy.middleware]信息：启用的项目管道：
[]
2017-12-27 17:21:49[刮屑.堆芯.发动机]信息：卡盘已打开
2017-12-27 17:21:49[scrapy.extensions.logstats]信息：爬网0页（0页/分钟），爬网0项（0项/分钟）
2017-12-27 17:21:49[scrapy.extensions.telnet]调试：telnet控制台监听127.0.0.1:6023
2017-12-27 17:21:49[scrapy.downloadermiddleware.retry]调试：重试（失败1次）：DNS查找失败：主机名查找没有结果：https。
2017-12-27 17:21:49[scrapy.downloadermiddleware.retry]调试：重试（失败2次）：DNS查找失败：主机名查找没有结果：https。
2017-12-27 17:21:49[scrapy.downloadermiddleware.retry]调试：放弃重试（失败3次）：DNS查找失败：主机名查找没有结果：https。
2017-12-27 17:21:49[scrapy.downloadermiddleware.robotstxt]错误：下载错误：DNS查找失败：主机名查找没有结果：https。
回溯（最近一次呼叫最后一次）：
文件“/Users/devsandbox/anaconda3/lib/python3.6/site packages/twisted/internet/defer.py”，第1384行，在内联回调中
结果=结果。通过ExceptionToGenerator（g）
文件“/Users/devsandbox/anaconda3/lib/python3.6/site packages/twisted/python/failure.py”，第408行，位于ThroweExceptionToGenerator中
返回g.throw（self.type、self.value、self.tb）
文件“/Users/devsandbox/anaconda3/lib/python3.6/site packages/scrapy/core/downloader/middleware.py”，第43行，进程中请求
defer.returnValue（（yield download_func（request=request，spider=spider）））
文件“/Users/devsandbox/anaconda3/lib/python3.6/site packages/twisted/internet/defer.py”，第653行，在runCallbacks中
current.result=回调（current.result，*args，**kw）
文件“/Users/devsandbox/anaconda3/lib/python3.6/site packages/twisted/internet/endpoints.py”，第954行，在StartConnection中
“主机名查找没有结果：{}”。格式（self.\u hostStr）
twisted.internet.error.DNSLookupError:DNS查找失败：主机名查找没有结果：https。
2017-12-27 17:21:49[scrapy.downloadermiddleware.retry]调试：重试（失败1次）：DNS查找失败：主机名查找没有结果：https。
2017-12-27 17:21:49[scrapy.downloadermiddleware.retry]调试：重试（失败2次）：DNS查找失败：主机名查找没有结果：https。
2017-12-27 17:21:49[scrapy.downloadermiddleware.retry]调试：放弃重试（失败3次）：DNS查找失败：主机名查找没有结果：https。
2017-12-27 17:21:49[scrapy.core.scraper]错误：下载错误
回溯（最近一次呼叫最后一次）：
文件“/Users/devsandbox/anaconda3/lib/python3.6/site packages/twisted/internet/defer.py”，第1384行，在内联回调中
结果=结果。通过ExceptionToGenerator（g）
文件“/Users/devsandbox/anaconda3/lib/python3.6/site packages/twisted/python/failure.py”，第408行，位于ThroweExceptionToGenerator中
返回g.throw（self.type、self.value、self.tb）
文件“/Users/devsandbox/anaconda3/lib/python3.6/site packages/scrapy/core/downloader/middleware.py”，第43行，进程中请求
defer.returnValue（（yield download_func（request=request，spider=spider）））
文件“/Users/devsandbox/anaconda3/lib/python3.6/site packages/twisted/internet/defer.py”，第653行，在runCallbacks中
current.result=回调（current.result，*args，**kw）
文件“/Users/devsandbox/anaconda3/lib/python3.6/site packages/twisted/internet/endpoints.py”，第954行，在StartConnection中
“主机名查找没有结果：{}”。格式（self.\u hostStr）
twisted.internet.error.DNSLookupError:DNS查找失败：主机名查找没有结果：https。
2017-12-27 17:21