Python scrapy中未处理或不允许HTTP状态代码
我对scrapy是新手，遇到以下错误“HTTP状态码未处理或不允许”，我已经尝试了解决方案 myspider.pyPython scrapy中未处理或不允许HTTP状态代码,python,scrapy,Python,Scrapy,我对scrapy是新手，遇到以下错误“HTTP状态码未处理或不允许”，我已经尝试了解决方案 myspider.py import scrapy class SubmitformSpider(scrapy.Spider): name = "formsubmit_xpath" website_url = "https://www.knowyourgst.com/gst-number-search/" start_urls = ["https://www.knowyourgst
import scrapy
class SubmitformSpider(scrapy.Spider):
name = "formsubmit_xpath"
website_url = "https://www.knowyourgst.com/gst-number-search/"
start_urls = ["https://www.knowyourgst.com/gst-number-search/"]
def parse(self,response):
# extract csrf_token value
token = response.css('input[name="csrfmiddlewaretoken"]::attr(value)').extract_first()
# create a dictionary for form values
data = {
'csrfmiddlewaretoken':token,
'gstnum':'27ARFPS6606L1ZA'
}
# submit a post request to it.
headers= {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0'}
yield scrapy.FormRequest(url=self.website_url, formdata = data, callback = self.parse_formrequest, headers=headers)
def parse_formrequest(self,response):
yield{
'BusinessName' : quote.xpath('/html/body/div[1]/div[2]/div[2]/div/table/tbody/tr[1]/td[2]/text()').extract_first()
}
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36'
import scrapy
class SubmitformSpider(scrapy.Spider):
name = "formsubmit_xpath"
website_url = "https://www.knowyourgst.com/gst-number-search/"
start_urls = ["https://www.knowyourgst.com/gst-number-search/"]
def parse(self,response):
# extract csrf_token value
token = response.css('input[name="csrfmiddlewaretoken"]::attr(value)').extract_first()
# create a dictionary for form values
data = {
'csrfmiddlewaretoken':token,
'gstnum':'27ARFPS6606L1ZA'
}
# submit a post request to it.
headers= {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0'}
yield scrapy.FormRequest(url=self.website_url, formdata = data, callback = self.parse_formrequest, headers=headers)
def parse_formrequest(self,response):
yield{
'BusinessName' : quote.xpath('/html/body/div[1]/div[2]/div[2]/div/table/tbody/tr[1]/td[2]/text()').extract_first()
}
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36'
2020-04-20 18:23:14 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: formsubmit_getresult)
2020-04-20 18:23:14 [scrapy.utils.log] INFO: Versions: lxml 4.3.4.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.7.3 (default, Mar 27 2019, 22:11:17) - [GCC 7.3.0], pyOpenSSL 19.0.0 (OpenSSL 1.1.1f 31 Mar 2020), cryptography 2.7, Platform Linux-5.3.0-46-generic-x86_64-with-debian-buster-sid
2020-04-20 18:23:14 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'formsubmit_getresult', 'NEWSPIDER_MODULE': 'formsubmit_getresult.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['formsubmit_getresult.spiders'], 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36'}
2020-04-20 18:23:14 [scrapy.extensions.telnet] INFO: Telnet Password: 0d70cf8b14bad003
2020-04-20 18:23:14 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2020-04-20 18:23:14 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-04-20 18:23:14 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-04-20 18:23:14 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-04-20 18:23:14 [scrapy.core.engine] INFO: Spider opened
2020-04-20 18:23:14 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-04-20 18:23:14 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-04-20 18:23:14 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.knowyourgst.com/robots.txt/> from <GET https://www.knowyourgst.com/robots.txt>
2020-04-20 18:23:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.knowyourgst.com/robots.txt/> (referer: None)
2020-04-20 18:23:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.knowyourgst.com/gst-number-search/> (referer: None)
2020-04-20 18:23:15 [scrapy.core.engine] DEBUG: Crawled (403) <POST https://www.knowyourgst.com/gst-number-search/> (referer: https://www.knowyourgst.com/gst-number-search/)
2020-04-20 18:23:15 [scrapy.core.scraper] DEBUG: Scraped from <403 https://www.knowyourgst.com/gst-number-search/>
{'BusinessName': None}
2020-04-20 18:23:15 [scrapy.core.engine] INFO: Closing spider (finished)
2020-04-20 18:23:15 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1586,
'downloader/request_count': 4,
'downloader/request_method_count/GET': 3,
'downloader/request_method_count/POST': 1,
'downloader/response_bytes': 6620,
'downloader/response_count': 4,
'downloader/response_status_count/200': 2,
'downloader/response_status_count/301': 1,
'downloader/response_status_count/403': 1,
'elapsed_time_seconds': 1.018804,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2020, 4, 20, 12, 53, 15, 113726),
'item_scraped_count': 1,
'log_count/DEBUG': 5,
'log_count/INFO': 10,
'memusage/max': 54525952,
'memusage/startup': 54525952,
'request_depth_max': 1,
'response_received_count': 3,
'robotstxt/request_count': 1,
'robotstxt/response_count': 1,
'robotstxt/response_status_count/200': 1,
'scheduler/dequeued': 2,
'scheduler/dequeued/memory': 2,
'scheduler/enqueued': 2,
'scheduler/enqueued/memory': 2,
'start_time': datetime.datetime(2020, 4, 20, 12, 53, 14, 94922)}
2020-04-20 18:23:15 [scrapy.core.engine] INFO: Spider closed (finished)
2020-04-20 18:23:14[scrapy.utils.log]信息：scrapy 1.8.0已启动（bot:formsubmit\u getresult）
2020-04-20 18:23:14[scrapy.utils.log]信息：版本：lxml 4.3.4.0，libxml2.9.9，cssselect 1.1.0，parsel 1.5.2，w3lib 1.21.0，Twisted 19.10.0，Python 3.7.3（默认，2019年3月27日，22:11:17）-[GCC 7.3.0]，pyOpenSSL 19.0.0（OpenSSL 1.1.1f 2020年3月31日），密码学2.7，平台Linux-5.3.0-46-generic-664-with-debian-Xuser
2020-04-20 18:23:14[scrapy.crawler]信息：覆盖的设置：{'BOT_NAME'：'formsubmit_getresult'，'NEWSPIDER_MODULE'：'formsubmit_getresult.SPIDER'，'ROBOTSTXT_obe'：True，'SPIDER_MODULES'：['formsubmit_getresult.SPIDER']，'USER_AGENT'：'Mozilla/5.0（X11；Linux x86_64）AppleWebKit/537.36（KHTML，像Gecko）Chrome/81.0.4044.92 Safari/537.36}
2020-04-20 18:23:14[scrapy.extensions.telnet]信息：telnet密码：0d70cf8b14bad003
2020-04-20 18:23:14[scrapy.middleware]信息：启用的扩展：
['scrapy.extensions.corestats.corestats'，
'scrapy.extensions.telnet.TelnetConsole'，
'scrapy.extensions.memusage.MemoryUsage'，
'scrapy.extensions.logstats.logstats']
2020-04-20 18:23:14[scrapy.middleware]信息：启用的下载程序中间件：
['scrapy.downloaderMiddleware.robotstxt.RobotsTxtMiddleware'，
'scrapy.downloaderMiddleware.httpauth.HttpAuthMiddleware'，
'scrapy.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware'，
'scrapy.downloaderMiddleware.defaultheaders.DefaultHeadersMiddleware'，
'scrapy.DownloaderMiddleware.useragent.UserAgentMiddleware'，
'scrapy.DownloaderMiddleware.retry.RetryMiddleware'，
'scrapy.DownloaderMiddleware.redirect.MetaRefreshMiddleware'，
'scrapy.DownloaderMiddleware.httpcompression.HttpCompressionMiddleware'，
'scrapy.DownloaderMiddleware.redirect.RedirectMiddleware'，
“scrapy.DownloaderMiddleware.cookies.CookiesMiddleware”，
'scrapy.downloadermiddleware.httpproxy.HttpProxyMiddleware'，
'scrapy.downloadermiddleware.stats.DownloaderStats']
2020-04-20 18:23:14[scrapy.middleware]信息：启用的蜘蛛中间件：
['scrapy.spidermiddleware.httperror.httperror中间件'，
'刮皮.SpiderMiddleware.场外.场外Iddleware'，
“scrapy.Spidermiddleware.referer.RefererMiddleware”，
'scrapy.spiderMiddleware.urllength.UrlLengthMiddleware'，
'scrapy.spidermiddleware.depth.DepthMiddleware']
2020-04-20 18:23:14[scrapy.middleware]信息：启用的项目管道：
[]
2020-04-20 18:23:14[刮屑.堆芯.发动机]信息：十字轴已打开
2020-04-20 18:23:14[scrapy.extensions.logstats]信息：抓取0页（以0页/分钟的速度），抓取0项（以0项/分钟的速度）
2020-04-20 18:23:14[scrapy.extensions.telnet]信息：telnet控制台监听127.0.0.1:6023
2020-04-20 18:23:14[scrapy.downloadermiddleware.redirect]调试：重定向（301）到
2020-04-20 18:23:14[scrapy.core.engine]调试：爬网（200）（参考：无）
2020-04-20 18:23:14[scrapy.core.engine]调试：爬网（200）（参考：无）
2020-04-20 18:23:15[scrapy.core.engine]调试：爬网（403）（参考：https://www.knowyourgst.com/gst-number-search/)
2020-04-20 18:23:15[scrapy.core.scraper]调试：从
{'BusinessName'：无}
2020-04-20 18:23:15[刮屑堆芯发动机]信息：关闭卡盘（完成）
2020-04-20 18:23:15[斯拉比统计局]信息：倾销斯拉比统计局：
{'downloader/request_bytes'：1586，
“下载程序/请求计数”：4，
“下载程序/请求方法\计数/获取”：3，
“下载程序/请求方法\计数/发布”：1，
“下载程序/响应字节”：6620，
“下载程序/响应计数”：4，
“下载程序/响应状态\计数/200”：2，
“下载程序/响应状态\计数/301”：1，
“下载器/响应\状态\计数/403”：1，
“已用时间秒”：1.018804，
“完成原因”：“完成”，
“完成时间”：datetime.datetime（2020,4,20,12,53,15,113726），
“物料刮取计数”：1，
“日志计数/调试”：5，
“日志计数/信息”：10，
“memusage/max”：54525952，
“memusage/startup”：54525952，
“请求深度最大值”：1，
“收到的响应数”：3，
“机器人文本/请求计数”：1，
“机器人文本/响应计数”：1，
“robotstxt/response\u status\u count/200”：1，
“调度程序/出列”：2，
“调度程序/出列/内存”：2，
“调度程序/排队”：2，
“调度程序/排队/内存”：2，
“开始时间”：datetime.datetime（2020,4,20,12,53,14,94922）}
2020-04-20 18:23:15[刮屑堆芯发动机]信息：十字轴关闭（完成）
token = response.css('input[name="csrfmiddlewaretoken"]::attr(value)').extract_first()
您是否尝试将状态代码添加到
handle\u httpstatus\u listvalueinput[name=“csrfmiddlewaretoken”]：：attr（value）