Python Scrapy中的代理配置

我需要在我的Scrapy项目中配置代理。当我尝试运行它时，它会抛出以下错误：

错误

---

从错误中可以看出，它正在返回，正如你所看到的，它说的是“未经授权”

状态代码407表示：HTTP 407 Proxy Authentication Required client错误状态响应代码表示尚未应用请求，因为它缺少位于浏览器和可访问请求资源的服务器之间的代理服务器的有效身份验证凭据

看不到你的代码，帮不了你什么忙

有一个图书馆可以帮助您做到这一点：

2019-05-09 19:36:50 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-05-09 19:36:50 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.myip.com/> (failed 1 times): Could not open CONNECT tunnel with proxy x.x.x.x:xxxx [{'status': 407, 'reason': b'Unauthorized'}]

class FirstSpyder(CrawlSpider):
    # The name of the spider
    name = "FirstSpyder"
    def __init__(self, *args, **kwargs):
        super(FirstSpyder, self).__init__(*args, **kwargs)
        self.proxy_pool = ['http://x.x.x.x:xxxx']

    # The domains that are allowed (links to other domains are skipped)
    allowed_domains = ["myip.com"]

    # The URLs to start with
    start_urls = ['https://www.myip.com/']

    def start_requests(self):
        for url in self.start_urls:
            request = Request(url, dont_filter=True, callback=self.parse)
            # set the meta['item'] to use the item in the next call back
            request.meta['proxy'] = random.choice(self.proxy_pool)
            yield request

    def parse(self, response):
        ip = response.css('#ip::text').get()
        print ("IPV4::",ip)