Python 局部使用刮油轮毂刮油毂的飞溅物_Python_Scrapy_Scrapy Splash_Scrapinghub_Splash Js Render

Python 局部使用刮油轮毂刮油毂的飞溅物

python scrapy

Python 局部使用刮油轮毂刮油毂的飞溅物,python,scrapy,scrapy-splash,scrapinghub,splash-js-render,Python,Scrapy,Scrapy Splash,Scrapinghub,Splash Js Render,我在scrapinghub上得到了一个splash的支持，我想从本地机器上运行的脚本中使用它。到目前为止，我提供的说明如下： 1）编辑设置文件： #I got this one from my scraping hub account SPLASH_URL = 'http://xx.x0-splash.scrapinghub.com' DOWNLOADER_MIDDLEWARES = { 'scrapy_splash.SplashCookiesMiddleware': 723,

我在scrapinghub上得到了一个splash的支持，我想从本地机器上运行的脚本中使用它。到目前为止，我提供的说明如下：

1）编辑设置文件：

#I got this one from my scraping hub account
SPLASH_URL = 'http://xx.x0-splash.scrapinghub.com'


DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}

SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}

DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'

HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

从这一点来看，我有一个dobt，当我试图在浏览器上打开spash服务器时，它会要求我输入用户名，我不知道在哪里设置scrapy

2） spider文件：

import scrapy
import json
from scrapy import  Request
from scrapy_splash import SplashRequest
import scrapy_splash


class ListSpider(scrapy.Spider):

    name = 'list'
    allowed_domains = ['https://medium.com/']
    start_urls = ['https://medium.com/']

    def parse(self, response):
        print (response.body)
        with open('data/cookies_file.json') as f:
            cookies_data = json.loads(f.read())[0]
        #print (cookies_data)
        url = 'https://medium.com/' 
        #cookies=cookies_data,
        yield Request(url,  callback=self.afterlogin,meta={'splash': {'args': {'html': 1, 'png': 1,}}})






    def afterlogin(self,response):
        with open(data_dir + 'after_login_page.html','w') as f:
            f.write(str(response.body))

我没有收到错误，但我不确定splash是否也在工作，除了服务器ip之外，scraping还提供了一个密码，我不知道该在哪里使用这个脚本

在使用splashrequest并添加API密钥后，这是我得到的回溯，网站的内容仍然没有加载

2019-07-17 10:10:08 [scrapy.core.engine] INFO: Spider opened
2019-07-17 10:10:08 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-07-17 10:10:08 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2019-07-17 10:10:09 [scrapy.core.downloader.tls] WARNING: Remote certificate is not valid for hostname "www.meetmindful.com"; '*.meetmindful.com'!='www.meetmindful.com'
2019-07-17 10:10:09 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.meetmindful.com/> (referer: None)
2019-07-17 10:10:13 [scrapy.core.downloader.tls] WARNING: Remote certificate is not valid for hostname "uyu74ur0-splash.scrapinghub.com"; '*.scrapinghub.com'!='uyu74ur0-splash.scrapinghub.com'
2019-07-17 10:10:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://app.meetmindful.com/login via https://uyu74ur0-splash.scrapinghub.com/render.html> (referer: None)
2019-07-17 10:10:20 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://app.meetmindful.com/grid via https://uyu74ur0-splash.scrapinghub.com/render.html> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
2019-07-17 10:10:21 [scrapy.core.downloader.tls] WARNING: Remote certificate is not valid for hostname "uyu74ur0-splash.scrapinghub.com"; '*.scrapinghub.com'!='uyu74ur0-splash.scrapinghub.com'
2019-07-17 10:10:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://app.meetmindful.com/grid via https://uyu74ur0-splash.scrapinghub.com/render.html> (referer: None)
2019-07-17 10:10:26 [scrapy.core.engine] INFO: Closing spider (finished)
2019-07-17 10:10:26 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 1,
 'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived': 1,
 'downloader/request_bytes': 2952,
 'downloader/request_count': 4,
 'downloader/request_method_count/GET': 1,
 'downloader/request_method_count/POST': 3,
 'downloader/response_bytes': 28104,
 'downloader/response_count': 3,
 'downloader/response_status_count/200': 3,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 7, 17, 14, 10, 26, 292646),
 'log_count/DEBUG': 5,
 'log_count/INFO': 8,
 'log_count/WARNING': 3,
 'memusage/max': 54104064,
 'memusage/startup': 54104064,
 'request_depth_max': 2,
 'response_received_count': 3,
 'retry/count': 1,
 'retry/reason_count/twisted.web._newclient.ResponseNeverReceived': 1,
 'scheduler/dequeued': 6,
 'scheduler/dequeued/memory': 6,
 'scheduler/enqueued': 6,
 'scheduler/enqueued/memory': 6,
 'splash/render.html/request_count': 2,
 'splash/render.html/response_count/200': 2,
 'start_time': datetime.datetime(2019, 7, 17, 14, 10, 8, 200073)}
2019-07-17 10:10:26 [scrapy.core.engine] INFO: Spider closed (finished)

2019-07-17 10:10:08[scrapy.core.engine]信息：卡盘已打开
2019-07-17 10:10:08[scrapy.extensions.logstats]信息：爬网0页（0页/分钟），爬网0项（0项/分钟）
2019-07-17 10:10:08[scrapy.extensions.telnet]调试：telnet控制台监听127.0.0.1:6023
2019-07-17 10:10:09[scrapy.core.downloader.tls]警告：远程证书对主机名“www.meetmindful.com”；“*”无效。meetmindful.com'！='www.meetmindful.com'
2019-07-17 10:10:09[scrapy.core.engine]调试：爬网（200）（参考：无）
2019-07-17 10:10:13[scrapy.core.downloader.tls]警告：远程证书对于主机名“uyu74ur0 splash.scrapinghub.com”；“*”无效。scrapinghub.com'！='uyu74ur0 splash.scrapinghub.com'
2019-07-17 10:10:14[刮屑核心引擎]调试：爬网（200）（参考：无）
2019-07-17 10:10:20[scrapy.downloadermiddleware.retry]调试：重试（失败1次）：[]
2019-07-17 10:10:21[scrapy.core.downloader.tls]警告：远程证书对于主机名“uyu74ur0 splash.scrapinghub.com”；“*”无效。scrapinghub.com'！='uyu74ur0 splash.scrapinghub.com'
2019-07-17 10:10:23[刮屑核心引擎]调试：爬网（200）（参考：无）
2019-07-17 10:10:26[刮屑芯发动机]信息：关闭卡盘（已完成）
2019-07-17 10:10:26[scrapy.statscollectors]信息：倾销scrapy统计数据：
{'downloader/exception_count'：1，
'downloader/exception\u type\u count/twisted.web.\u newclient.ResponseNeverReceived'：1，
“下载程序/请求字节”：2952，
“下载程序/请求计数”：4，
“downloader/request\u method\u count/GET”：1，
“下载程序/请求方法/计数/发布”：3，
“downloader/response_字节”：28104，
“下载程序/响应计数”：3，
“下载/响应状态\计数/200”：3，
“完成原因”：“完成”，
“完成时间”：datetime.datetime（2019,7,17,14,10,26,292646），
“日志计数/调试”：5，
“日志计数/信息”：8，
“日志计数/警告”：3，
“memusage/max”：54104064，
“memusage/startup”：54104064，
“请求深度最大值”：2，
“收到的响应数”：3，
“重试/计数”：1，
“重试/reason\u count/twisted.web.\u newclient.ResponseNeverReceived”：1，
“调度程序/出列”：6，
“调度程序/出列/内存”：6，
“调度程序/排队”：6，
“调度程序/排队/内存”：6，
'splash/render.html/request_count'：2，
'splash/render.html/response\u count/200'：2，
“开始时间”：datetime.datetime（2019,7,17,14,10,820073）}
2019-07-17 10:10:26[刮屑芯发动机]信息：十字轴关闭（完成）

编辑：

这是我得到的完整日志

INFO: Scrapy 1.5.2 started (bot: meetmindfull)
INFO: Versions: lxml 4.3.2.0, libxml2 2.9.9, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 19.2.0, Python 3.7.3 (default, Mar 27 2019, 22:11:17) - [GCC 7.3.0], pyOpenSSL 19.0.0 (OpenSSL 1.1.1  11 Sep 2018), cryptography 2.6.1, Platform Linux-4.15.0-20-generic-x86_64-with-debian-buster-sid
INFO: Overridden settings: {'BOT_NAME': 'meetmindfull', 'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter', 'HTTPCACHE_STORAGE': 'scrapy_splash.SplashAwareFSCacheStorage', 'LOG_FILE': 'log.txt', 'LOG_FORMAT': '%(levelname)s: %(message)s', 'NEWSPIDER_MODULE': 'meetmindfull.spiders', 'SPIDER_MODULES': ['meetmindfull.spiders']}
INFO: Telnet Password: 4a122ec20dcf75e1
INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy_splash.SplashCookiesMiddleware',
 'scrapy_splash.SplashMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy_splash.SplashDeduplicateArgsMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
INFO: Enabled item pipelines:
[]
INFO: Spider opened
INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
DEBUG: Telnet console listening on 127.0.0.1:6023
WARNING: Remote certificate is not valid for hostname "www.meetmindful.com"; '*.meetmindful.com'!='www.meetmindful.com'
DEBUG: Crawled (200) <GET https://www.meetmindful.com/> (referer: None)
WARNING: Remote certificate is not valid for hostname "uyu74ur0-splash.scrapinghub.com"; '*.scrapinghub.com'!='uyu74ur0-splash.scrapinghub.com'
DEBUG: Crawled (200) <GET https://app.meetmindful.com/login via https://uyu74ur0-splash.scrapinghub.com/render.html> (referer: None)
DEBUG: Retrying <GET https://app.meetmindful.com/grid via https://uyu74ur0-splash.scrapinghub.com/render.html> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
WARNING: Remote certificate is not valid for hostname "uyu74ur0-splash.scrapinghub.com"; '*.scrapinghub.com'!='uyu74ur0-splash.scrapinghub.com'
DEBUG: Crawled (200) <GET https://app.meetmindful.com/grid via https://uyu74ur0-splash.scrapinghub.com/render.html> (referer: None)
INFO: Closing spider (finished)
INFO: Dumping Scrapy stats:
{'downloader/exception_count': 1,



'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived': 1,
 'downloader/request_bytes': 2952,
 'downloader/request_count': 4,
 'downloader/request_method_count/GET': 1,
 'downloader/request_method_count/POST': 3,
 'downloader/response_bytes': 28096,
 'downloader/response_count': 3,
 'downloader/response_status_count/200': 3,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 7, 17, 14, 47, 46, 604347),
 'log_count/DEBUG': 5,
 'log_count/INFO': 8,
 'log_count/WARNING': 3,
 'memusage/max': 54267904,
 'memusage/startup': 54267904,
 'request_depth_max': 2,
 'response_received_count': 3,
 'retry/count': 1,
 'retry/reason_count/twisted.web._newclient.ResponseNeverReceived': 1,
 'scheduler/dequeued': 6,
 'scheduler/dequeued/memory': 6,
 'scheduler/enqueued': 6,
 'scheduler/enqueued/memory': 6,
 'splash/render.html/request_count': 2,
 'splash/render.html/response_count/200': 2,
 'start_time': datetime.datetime(2019, 7, 17, 14, 47, 28, 791792)}
INFO: Spider closed (finished)

信息：Scrapy 1.5.2已启动（bot:meetmindfull）
信息：版本：lxml 4.3.2.0、libxml2.9.9、cssselect 1.0.3、parsel 1.5.1、w3lib 1.20.0、Twisted 19.2.0、Python 3.7.3（默认值，2019年3月27日，22:11:17）-（GCC 7.3.0）、pyOpenSSL 19.0.0（OpenSSL 1.1.1.1，2018年9月11日）、加密2.6.1、平台Linux-4.15.0-20-generic-x86_64-with-debian-buster-sid
信息：覆盖的设置：{'BOT_NAME'：'meetmindfull'，'DUPEFILTER_CLASS'：'scrapy_splash.splashawredupefilter'，'HTTPCACHE_存储'：'scrapy_splash.splashawrefscachestorage'，'LOG_文件'：'LOG.txt'，'LOG_格式'：'%（levelname）s:%（message）s'，'NEWSPIDER_模块'：'meetmindfull.SPIDER'，'SPIDER模块：['meetmindfull.SPIDER']
信息：Telnet密码：4a122ec20dcf75e1
信息：已启用的扩展：
['scrapy.extensions.corestats.corestats'，
'scrapy.extensions.telnet.TelnetConsole'，
'scrapy.extensions.memusage.MemoryUsage'，
'scrapy.extensions.logstats.logstats']
信息：已启用下载程序中间件：
['scrapy.downloaderMiddleware.httpauth.HttpAuthMiddleware'，
'scrapy.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware'，
'scrapy.downloaderMiddleware.defaultheaders.DefaultHeadersMiddleware'，
'scrapy.DownloaderMiddleware.useragent.UserAgentMiddleware'，
'scrapy.DownloaderMiddleware.retry.RetryMiddleware'，
'scrapy.DownloaderMiddleware.redirect.MetaRefreshMiddleware'，
'scrapy.DownloaderMiddleware.redirect.RedirectMiddleware'，
“scrapy.DownloaderMiddleware.cookies.CookiesMiddleware”，
“刮擦飞溅，飞溅”，
"刮花,溅花",，
'scrapy.downloadermiddleware.httpproxy.HttpProxyMiddleware'，
'scrapy.DownloaderMiddleware.httpcompression.HttpCompressionMiddleware'，
'scrapy.downloadermiddleware.stats.DownloaderStats']
信息：已启用的蜘蛛中间件：
['scrapy.spidermiddleware.httperror.httperror中间件'，
“刮花，刮花”，
'刮皮.SpiderMiddleware.场外.场外Iddleware'，
“scrapy.Spidermiddleware.referer.RefererMiddleware”，
'scrapy.spiderMiddleware.urllength.UrlLengthMiddleware'，
'scrapy.spidermiddleware.depth.DepthMiddleware']
信息：已启用的项目管道：
[]
信息：蜘蛛打开了
信息：爬网0页（0页/分钟），刮取0项（0项/分钟）
调试：Telnet控制台在127.0.0.1:6023上侦听
警告：远程证书对于主机名“www.meetmindful.com”；“*”无效。meetmindful.com'！='www.meetmindful.com'
调试：已爬网（200）（引用程序：无）
警告：远程证书对于主机名“uyu74ur0 splash.scrapinghub.com”；“*”无效。scrapinghub.com'！='uyu74ur0 splash.scrapinghub.com'
调试：已爬网（200）（引用程序：无）
调试：重试（失败1次）：[]
警告：远程证书对于主机名“uyu74ur0 splash.scrapinghub.com”；“*”无效。s