Fatal编程技术网
  • python/
  • Python 无法分析scrapy splash lua脚本的html响应

Python 无法分析scrapy splash lua脚本的html响应

python web-scraping

Python 无法分析scrapy splash lua脚本的html响应,python,web-scraping,scrapy-splash,splash-js-render,Python,Web Scraping,Scrapy Splash,Splash Js Render,我试图解析从SplashRequest执行端点返回的html，该端点应该返回html，但是当我将其传递给回调函数时，它不会打印任何内容（不解析）。我的日志没有显示错误，代码如下 import scrapy from scrapy_splash import SplashRequest class Kapow_crawler(scrapy.Spider): name = "Kapow_crawler" def start_requests(self): u

我试图解析从SplashRequest执行端点返回的html，该端点应该返回html，但是当我将其传递给回调函数时，它不会打印任何内容（不解析）。我的日志没有显示错误，代码如下

import scrapy

from scrapy_splash import SplashRequest


class Kapow_crawler(scrapy.Spider):
    name = "Kapow_crawler"

    def start_requests(self):

        urls = [
            "https://vmatrix1.brevardclerk.us/beca/name_search.cfm"
        ]
        for url in urls:
            yield SplashRequest(url=url, callback=self.parse_first_page,endpoint='render.html')

    def parse_first_page(self, response):
        id_list=[]
        inputs = response.css('input').extract()
        input_count = 0
        for inp in inputs:
            input_count = input_count+1 
            if 'Yes'.upper() in inp.upper():
                disc_yes = input_count -1 
                try:
                    id_list.append(['id',response.css('input::attr(id)')[disc_yes].extract()])
                except Exception:
                    id_list.append(['name',response.css('input::attr(name)')[disc_yes].extract()])
                if 'radio'.upper() in inp.upper():
                    inputs = response.css('input').extract()
                    input_count = 0
                    for inp in inputs:
                        input_count = input_count+1 
                        if 'Submit'.upper() in inp.upper():
                            disc_submit = input_count -1
                            try:                               
id_list.append(['id',response.css('input::attr(id)')[disc_submit].extract()])
                            except Exception:             
id_list.append(['name',response.css('input::attr(name)')[disc_submit].extract()])
        if len(id_list) == 2:
           disclaimer_script="""
           function main(splash)
             assert(splash:go("{0}"))
             assert(splash:wait(0.5))
             local disc_check = splash:select('[{1}="{2}"]')
             local disc_sub = splash:select('[{3}="{4}"]')
             assert(disc_check:mouse_click())
             assert(disc_sub:mouse_click())
             assert(splash:wait(0.5))
             return splash:html()
           end
           """.format(response.url,id_list[0][0],id_list[0][1],id_list[1][0],id_list[1][1])
        if len(id_list) >= 1:
            yield SplashRequest(url=response.url, callback=self.parse_second_page, endpoint='execute',args={'har':1,'html':1,'lua_source':disclaimer_script,'wait':0.5})
        else:
            yield SplashRequest(url=response.url, callback=self.parse_second_page, endpoint='render.html')


    def parse_second_page(self, response):
        inputs = response.css('input').extract()
        for inp in inputs:
            yield print(inp)
我在下面添加了日志，您可以看到没有错误，但是SplashRequestHTML也没有被解析。splash:html（）应该呈现html，我知道回调可以将响应传递给不同的解析方法。任何协助都将不胜感激

2018-10-03 14:57:00 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: 
kapow_crawler)
2018-10-03 14:57:00 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 
2.9.8, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 
3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit 
(AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.0.2o  27 Mar 2018), cryptography 
2.2.2, Platform Windows-10-10.0.15063-SP0
2018-10-03 14:57:00 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 
'kapow_crawler', 'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter', 
'HTTPCACHE_STORAGE': 'scrapy_splash.SplashAwareFSCacheStorage', 'LOG_FILE': 
'C:\\Users\\jswordy\\.spyder-py3\\kapow_crawler\\log.txt', 
'NEWSPIDER_MODULE': 'kapow_crawler.spiders', 'SPIDER_MODULES': 
['kapow_crawler.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; 
Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 
Safari/537.36'}
2018-10-03 14:57:00 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2018-10-03 14:57:00 [scrapy.middleware] INFO: Enabled downloader 
middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'kapow_crawler.middlewares.KapowCrawlerDownloaderMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy_splash.SplashCookiesMiddleware',
 'scrapy_splash.SplashMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-10-03 14:57:00 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy_splash.SplashDeduplicateArgsMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'kapow_crawler.middlewares.KapowCrawlerSpiderMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-10-03 14:57:00 [scrapy.middleware] INFO: Enabled item pipelines:
['kapow_crawler.pipelines.KapowCrawlerPipeline',
 'kapow_crawler.pipelines.ScreenshotPipeline']
2018-10-03 14:57:00 [scrapy.core.engine] INFO: Spider opened
2018-10-03 14:57:00 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 
pages/min), scraped 0 items (at 0 items/min)
2018-10-03 14:57:00 [Kapow_crawler] INFO: Spider opened: Kapow_crawler
2018-10-03 14:57:00 [Kapow_crawler] INFO: Spider opened: Kapow_crawler
2018-10-03 14:57:00 [scrapy.extensions.telnet] DEBUG: Telnet console 
listening on 127.0.0.1:6023
2018-10-03 14:57:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET 
https://vmatrix1.brevardclerk.us/beca/name_search.cfm via 
http://localhost:8050/render.html> (referer: None)
2018-10-03 14:57:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET 
https://vmatrix1.brevardclerk.us/beca/name_search.cfm via 
http://localhost:8050/execute> (referer: None)  
2018-10-03 14:57:02 [scrapy.core.engine] INFO: Closing spider (finished)
2018-10-03 14:57:02 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 2024,
 'downloader/request_count': 2,
 'downloader/request_method_count/POST': 2,
 'downloader/response_bytes': 7633,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 2,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2018, 10, 3, 19, 57, 2, 715628),
 'log_count/DEBUG': 3,
 'log_count/INFO': 9,
 'request_depth_max': 1,
 'response_received_count': 2,
 'scheduler/dequeued': 4,
 'scheduler/dequeued/memory': 4,
 'scheduler/enqueued': 4,
 'scheduler/enqueued/memory': 4,
 'splash/execute/request_count': 1,
 'splash/execute/response_count/200': 1,
 'splash/render.html/request_count': 1,
 'splash/render.html/response_count/200': 1,
 'start_time': datetime.datetime(2018, 10, 3, 19, 57, 0, 758627)}
2018-10-03 14:57:02 [scrapy.core.engine] INFO: Spider closed (finished)

2018-10-03 14:57:00[scrapy.utils.log]信息：scrapy 1.5.1已启动（机器人：
卡波乌（履带式）
2018-10-03 14:57:00[scrapy.utils.log]信息：版本：lxml 4.2.1.0，libxml2
2.9.8、cssselect 1.0.3、parsel 1.5.0、w3lib 1.19.0、Twisted 18.7.0、Python
3.6.5 | Anaconda，Inc.|（默认，2018年3月29日，13:32:41）[MSC v.1900 64位
（AMD64）]，pyOpenSSL 18.0.0（OpenSSL 1.0.2，2018年3月27日），密码学
2.2.2，平台Windows-10-10.0.15063-SP0
2018-10-03 14:57:00[scrapy.crawler]信息：覆盖的设置：{'BOT_NAME'：
“kapow_爬虫”，“DUPEFILTER_类”：“刮擦溅起，溅起水花的DUPEFILTER”，
“HTTPCACHE\u存储”：“scrapy\u splash.SplashAwareFSCacheStorage”，“日志文件”：
'C:\\Users\\jswordy\\.spyder-py3\\kapow\u crawler\\log.txt'，
“NEWSPIDER_模块”：“kapow_crawler.SPIDER”，“SPIDER_模块”：
['kapow_crawler.spider']，'USER_AGENT'：'Mozilla/5.0（WindowsNT10.0；
Win64；x64）AppleWebKit/537.36（KHTML，类似Gecko）Chrome/69.0.3497.100
Safari/537.36'}
2018-10-03 14:57:00[scrapy.middleware]信息：启用的扩展：
['scrapy.extensions.corestats.corestats'，
'scrapy.extensions.telnet.TelnetConsole'，
'scrapy.extensions.logstats.logstats']
2018-10-03 14:57:00[scrapy.middleware]信息：已启用下载程序
中间商：
['scrapy.downloaderMiddleware.httpauth.HttpAuthMiddleware'，
'scrapy.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware'，
'scrapy.downloaderMiddleware.defaultheaders.DefaultHeadersMiddleware'，
'scrapy.DownloaderMiddleware.useragent.UserAgentMiddleware'，
“kapow_crawler.Middleware.KapowCrawlerDownloaderMiddleware”，
'scrapy.DownloaderMiddleware.retry.RetryMiddleware'，
'scrapy.DownloaderMiddleware.redirect.MetaRefreshMiddleware'，
'scrapy.DownloaderMiddleware.redirect.RedirectMiddleware'，
“scrapy.DownloaderMiddleware.cookies.CookiesMiddleware”，
“刮擦飞溅，飞溅”，
"刮花,溅花",，
'scrapy.downloadermiddleware.httpproxy.HttpProxyMiddleware'，
'scrapy.DownloaderMiddleware.httpcompression.HttpCompressionMiddleware'，
'scrapy.downloadermiddleware.stats.DownloaderStats']
2018-10-03 14:57:00[scrapy.middleware]信息：启用的蜘蛛中间件：
['scrapy.spidermiddleware.httperror.httperror中间件'，
“刮花，刮花”，
'刮皮.SpiderMiddleware.场外.场外Iddleware'，
“kapow_crawler.Middleware.KapowCrawlerSpiderMiddleware”，
“scrapy.Spidermiddleware.referer.RefererMiddleware”，
'scrapy.spiderMiddleware.urllength.UrlLengthMiddleware'，
'scrapy.spidermiddleware.depth.DepthMiddleware']
2018-10-03 14:57:00[scrapy.middleware]信息：启用的项目管道：
['kapow_crawler.pipelines.KapowCrawlerPipeline'，
“kapow_crawler.pipeline.ScreenshotPipeline”]
2018-10-03 14:57:00[刮屑.堆芯.发动机]信息：星形轮已打开
2018-10-03 14:57:00[scrapy.extensions.logstats]信息：爬网0页（0
页面/分钟），刮取0个项目（0个项目/分钟）
2018-10-03 14:57:00[Kapow_crawler]信息：蜘蛛打开：Kapow_crawler
2018-10-03 14:57:00[Kapow_crawler]信息：蜘蛛打开：Kapow_crawler
2018-10-03 14:57:00[scrapy.extensions.telnet]调试：telnet控制台
收听127.0.0.1:6023
2018-10-03 14:57:01[刮屑核心引擎]调试：爬网（200）（参考：无）
2018-10-03 14:57:02[刮屑核心引擎]调试：爬网（200）（参考：无）
2018-10-03 14:57:02[刮屑芯发动机]信息：关闭卡盘（已完成）
2018-10-03 14:57:02[scrapy.statscollectors]信息：倾销scrapy统计数据：
{'downloader/request_bytes'：2024，
“下载程序/请求计数”：2，
“下载程序/请求方法/计数/发布”：2，
“downloader/response_字节”：7633，
“下载程序/响应计数”：2，
“下载程序/响应状态\计数/200”：2，
“完成原因”：“完成”，
“完成时间”：datetime.datetime（2018,10,3,19,57,2715628），
“日志计数/调试”：3，
“日志计数/信息”：9，
“请求深度最大值”：1，
“响应\u已收到\u计数”：2，
“调度程序/出列”：4，
“调度程序/出列/内存”：4，
“调度程序/排队”：4，
“调度程序/排队/内存”：4，
“启动/执行/请求计数”：1，
“启动/执行/响应计数/200”：1，
'splash/render.html/request_count'：1，
'splash/render.html/response\u count/200'：1，
“开始时间”：datetime.datetime（2018,10,3,19,57,0758627）}
2018-10-03 14:57:02[刮屑堆芯发动机]信息：十字轴关闭（完成）