Python 在表单中输入数据后,在scrapy中输出空值

Python 在表单中输入数据后,在scrapy中输出空值,python,python-3.x,scrapy,scrapy-shell,Python,Python 3.x,Scrapy,Scrapy Shell,我正在尝试刮取一个有表单的网页,一旦输入,它将在同一页面中获取数据。 单击按钮后,指向页面的链接不会更改。 我正在尝试在填写表格后刮取数据,以下是我的代码: 该网站的链接为: after_enter函数运行,这意味着我认为表单已成功提交,但我想要的输出没有出现。 有人能帮我理解我代码中的错误吗? 任何帮助都将不胜感激,我不想使用硒。所以,如果有人能帮我擦掉这一页,那将非常有帮助。 此外,我还使ROBOTSTXT_obe=False 因此,robots.txt不应该成为问题。 这

我正在尝试刮取一个有表单的网页,一旦输入,它将在同一页面中获取数据。
单击按钮后,指向页面的链接不会更改。

我正在尝试在填写表格后刮取数据,以下是我的代码:

该网站的链接为:


after_enter函数运行,这意味着我认为表单已成功提交,但我想要的输出没有出现。

有人能帮我理解我代码中的错误吗?
任何帮助都将不胜感激,我不想使用硒。所以,如果有人能帮我擦掉这一页,那将非常有帮助。

此外,我还使ROBOTSTXT_obe=False
因此,robots.txt不应该成为问题。
这是我收到的输出,没有错误,也没有输出,请看我哪里出错了。

(基本)C:\Users\diveh\email\u validator>scrapy crawl email
2019-12-29 12:06:44[scrapy.utils.log]信息:scrapy 1.8.0已启动(bot:email\u验证程序)
2019-12-29 12:06:44[scrapy.utils.log]信息:版本:lxml 4.4.1.0,libxml2.9.9,cssselect 1.1.0,parsel 1.5.2,w3lib 1.21.0,Twisted 19.10.0,Python 3.7.4(默认,2019年8月9日18:34:13)[MSC v.1915 64位(AMD64)],pyOpenSSL 19.0.0(OpenSSL 1.1.1.1d 2019年9月10日),密码学2.7,平台Windows-10.0.18362-SP0
2019-12-29 12:06:44[抓取程序]信息:覆盖的设置:{'BOT_NAME':'email_validator','NEWSPIDER_MODULE':'email_validator.SPIDER','SPIDER_MODULES':['email_validator.SPIDER']
2019-12-29 12:06:44[scrapy.extensions.telnet]信息:telnet密码:2b5415ce25525ddc
2019-12-29 12:06:44[scrapy.middleware]信息:启用的扩展:
['scrapy.extensions.corestats.corestats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.logstats']
2019-12-29 12:06:44[剪贴簿中间件]信息:启用的下载程序中间件:
['scrapy.downloaderMiddleware.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloaderMiddleware.defaultheaders.DefaultHeadersMiddleware',
'scrapy.DownloaderMiddleware.useragent.UserAgentMiddleware',
'scrapy.DownloaderMiddleware.retry.RetryMiddleware',
'scrapy.DownloaderMiddleware.redirect.MetaRefreshMiddleware',
'scrapy.DownloaderMiddleware.httpcompression.HttpCompressionMiddleware',
'scrapy.DownloaderMiddleware.redirect.RedirectMiddleware',
“scrapy.DownloaderMiddleware.cookies.CookiesMiddleware”,
'scrapy.downloadermiddleware.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddleware.stats.DownloaderStats']
2019-12-29 12:06:44[scrapy.middleware]信息:启用的蜘蛛中间件:
['scrapy.spidermiddleware.httperror.httperror中间件',
'刮皮.SpiderMiddleware.场外.场外Iddleware',
“scrapy.Spidermiddleware.referer.RefererMiddleware”,
'scrapy.spiderMiddleware.urllength.UrlLengthMiddleware',
'scrapy.spidermiddleware.depth.DepthMiddleware']
2019-12-29 12:06:44[碎片中间件]信息:启用的项目管道:
[]
2019-12-29 12:06:44[刮屑芯发动机]信息:十字轴已打开
2019-12-29 12:06:44[scrapy.extensions.logstats]信息:爬网0页(0页/分钟),爬网0项(0项/分钟)
2019-12-29 12:06:44[scrapy.extensions.telnet]信息:telnet控制台监听127.0.0.1:6023
2019-12-29 12:06:45[scrapy.downloadermiddleware.redirect]调试:重定向(301)到
2019-12-29 12:06:46[刮屑核心引擎]调试:爬网(200)(参考:无)
2019-12-29 12:06:46[刮屑核心引擎]调试:爬网(200)(参考:https://phpans.com/free-email-validator/)
2019-12-29 12:06:46[scrapy.core.scraper]调试:从
{'Name':无}
2019-12-29 12:06:46[刮屑芯发动机]信息:关闭卡盘(已完成)
2019-12-29 12:06:46[斯拉比统计局]信息:倾销斯拉比统计局:
{'downloader/request_bytes':891,
“下载程序/请求计数”:3,
“下载器/请求\方法\计数/获取”:2,
“下载程序/请求方法\计数/发布”:1,
“downloader/response_字节”:5076,
“下载程序/响应计数”:3,
“下载程序/响应状态\计数/200”:2,
“下载程序/响应状态\计数/301”:1,
“已用时间秒”:2.08303,
“完成原因”:“完成”,
“完成时间”:datetime.datetime(2019,12,29,6,36,46,7725),
“物料刮取计数”:1,
“日志计数/调试”:4,
“日志计数/信息”:10,
“请求深度最大值”:1,
“响应\u已收到\u计数”:2,
“调度程序/出列”:3,
“调度程序/出列/内存”:3,
“调度程序/排队”:3,
“调度程序/排队/内存”:3,
“开始时间”:datetime.datetime(2019,12,29,6,36,44694695)}
2019-12-29 12:06:46[刮屑堆芯发动机]信息:十字轴关闭(完成)

正如在输出中一样,在表单中输入数据并进行验证后,我得到的名称输出为none,应该是一些值。因此,我认为必须做出一些改变,如果有人能帮助我,那就太好了
提前感谢:)

当我在这个页面上使用表单时,它什么也不显示。你在这页上看到了什么?。此外,此页面使用JavaScript,Scrapy无法运行JavaScript。您可能需要Selenium在此页面上运行JavaScript。请阅读相关内容。
# -*- coding: utf-8 -*-
# -*- coding: utf-8 -*-
import scrapy
from scrapy.utils.response import open_in_browser



class EmailSpider(scrapy.Spider):
    name = 'email'
    allowed_domains = ['phpans.com/free-email-validator']
    start_urls = ['http://phpans.com/free-email-validator/']

    def parse(self, response):

        return scrapy.FormRequest.from_response(response,
                                                formdata={'email':'ndiveshjain@gmail.com','validate':'true'},
                                                callback=self.after_enter,
                                                dont_filter = True,
                                                )
    def after_enter(self,response):

        chec = response.xpath('//*[@id="message"]').extract_first()
        yield{'Name':chec}
    (base) C:\Users\Divesh\email_validator>scrapy crawl email
2019-12-29 12:06:44 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: email_validator)
2019-12-29 12:06:44 [scrapy.utils.log] INFO: Versions: lxml 4.4.1.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)], pyOpenSSL 19.0.0 (OpenSSL 1.1.1d  10 Sep 2019), cryptography 2.7, Platform Windows-10-10.0.18362-SP0
2019-12-29 12:06:44 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'email_validator', 'NEWSPIDER_MODULE': 'email_validator.spiders', 'SPIDER_MODULES': ['email_validator.spiders']}
2019-12-29 12:06:44 [scrapy.extensions.telnet] INFO: Telnet Password: 2b5415ce25525ddc
2019-12-29 12:06:44 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2019-12-29 12:06:44 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-12-29 12:06:44 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-12-29 12:06:44 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-12-29 12:06:44 [scrapy.core.engine] INFO: Spider opened
2019-12-29 12:06:44 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-12-29 12:06:44 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-12-29 12:06:45 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://phpans.com/free-email-validator/> from <GET http://phpans.com/free-email-validator/>
2019-12-29 12:06:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://phpans.com/free-email-validator/> (referer: None)
2019-12-29 12:06:46 [scrapy.core.engine] DEBUG: Crawled (200) <POST https://phpans.com/free-email-validator/> (referer: https://phpans.com/free-email-validator/)
2019-12-29 12:06:46 [scrapy.core.scraper] DEBUG: Scraped from <200 https://phpans.com/free-email-validator/>
{'Name': None}
2019-12-29 12:06:46 [scrapy.core.engine] INFO: Closing spider (finished)
2019-12-29 12:06:46 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 891,
 'downloader/request_count': 3,
 'downloader/request_method_count/GET': 2,
 'downloader/request_method_count/POST': 1,
 'downloader/response_bytes': 5076,
 'downloader/response_count': 3,
 'downloader/response_status_count/200': 2,
 'downloader/response_status_count/301': 1,
 'elapsed_time_seconds': 2.08303,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 12, 29, 6, 36, 46, 777725),
 'item_scraped_count': 1,
 'log_count/DEBUG': 4,
 'log_count/INFO': 10,
 'request_depth_max': 1,
 'response_received_count': 2,
 'scheduler/dequeued': 3,
 'scheduler/dequeued/memory': 3,
 'scheduler/enqueued': 3,
 'scheduler/enqueued/memory': 3,
 'start_time': datetime.datetime(2019, 12, 29, 6, 36, 44, 694695)}
2019-12-29 12:06:46 [scrapy.core.engine] INFO: Spider closed (finished)