Python 3.x 爬行时在scrapy中出现错误403
以下是我为“Blacar”网站编写的代码 在运行上述程序时,我发现错误如下 2018-06-11 00:07:05[scrapy.extensions.telnet]调试:telnet控制台 收听127.0.0.1:6023 2018-06-11 00:07:06[刮屑.核心.发动机] 调试:已爬网(403)http://www.blablacar.in/robots.txt> (参考:无)2018-06-11 00:07:06[刮屑核心引擎]调试: 爬网(403)http://www.blablacar.in/ride-sharing/new-delhi/chandigarh>(参考: 无)2018-06-11 00:07:06[scrapy.spidermiddleware.httperror]信息: 忽略响应:HTTP 状态代码未处理或不允许2018-06-11 00:07:06 [scrapy.core.engine]信息:关闭十字轴(已完成)Python 3.x 爬行时在scrapy中出现错误403,python-3.x,web-scraping,scrapy,web-crawler,data-extraction,Python 3.x,Web Scraping,Scrapy,Web Crawler,Data Extraction,以下是我为“Blacar”网站编写的代码 在运行上述程序时,我发现错误如下 2018-06-11 00:07:05[scrapy.extensions.telnet]调试:telnet控制台 收听127.0.0.1:6023 2018-06-11 00:07:06[刮屑.核心.发动机] 调试:已爬网(403)http://www.blablacar.in/robots.txt> (参考:无)2018-06-11 00:07:06[刮屑核心引擎]调试: 爬网(403)http://www.blab
通常在html中,403错误表示您无权访问该页面
请使用其他网站重试。如果没有显示相同的错误,则可能是由网站响应引起的。通常是html,403错误表示您无权访问该页面
请使用其他网站重试。如果没有显示相同的错误,则可能是由于网站响应引起的。您需要配置用户代理。我使用配置的用户代理在我的站点中运行您的代码,并获得状态代码200 1靠近settings.py放置名为utils.py的新文件
import random
user_agent_list = [
# Chrome
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
'Mozilla/5.0 (Windows NT 5.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36'
# Firefox
'Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)',
'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)',
'Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (Windows NT 6.2; WOW64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)',
'Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)',
'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)',
'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)',
'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0'
]
def get_random_agent():
return random.choice(user_agent_list)
2将以下内容添加到settings.py文件:
from <SCRAPY_PROJECT>.utils import get_random_agent
USER_AGENT = get_random_agent()
from.utils导入获取随机代理
USER\u AGENT=get\u random\u AGENT()
您需要配置用户代理。我使用配置的用户代理在我的站点中运行您的代码,并获得状态代码200
1靠近settings.py放置名为utils.py的新文件
import random
user_agent_list = [
# Chrome
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
'Mozilla/5.0 (Windows NT 5.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36'
# Firefox
'Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)',
'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)',
'Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (Windows NT 6.2; WOW64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)',
'Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)',
'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)',
'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)',
'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0'
]
def get_random_agent():
return random.choice(user_agent_list)
2将以下内容添加到settings.py文件:
from <SCRAPY_PROJECT>.utils import get_random_agent
USER_AGENT = get_random_agent()
from.utils导入获取随机代理
USER\u AGENT=get\u random\u AGENT()
,您可以使用handle\u httpstatus\u list spider属性
就你而言:
class BlablaSpider(scrapy.Spider):
name = 'blabla'
allowed_domains = ['blablacar.in']
start_urls = ['http://www.blablacar.in/ride-sharing/new-delhi/chandigarh']
handle_httpstatus_list = [403]
,可以使用handle\u httpstatus\u list spider属性
就你而言:
class BlablaSpider(scrapy.Spider):
name = 'blabla'
allowed_domains = ['blablacar.in']
start_urls = ['http://www.blablacar.in/ride-sharing/new-delhi/chandigarh']
handle_httpstatus_list = [403]
那么,如何从爬行过程中显示403错误的网站进行爬行,以及如何从爬行过程中显示403错误的网站进行爬行