如何处理scrapy shell中的错误302

我正在尝试刮取一个被重定向的页面，我尝试设置一个用户代理，但它也不起作用

我在另一个问题中看到了这一点：

meta = {'dont_redirect': True,'handle_httpstatus_list': [302]}

---

如何在scrapy shell中测试它？

当使用

scrapy shell

时，最简单的方法可能是在命令行上使用

REDIRECT\u ENABLED=0设置禁用
RedirectMiddleware

比较这一点，完全禁用重定向：

$ scrapy shell -s REDIRECT_ENABLED=0
2016-02-09 10:16:27 [scrapy] INFO: Scrapy 1.0.4 started (bot: scrapybot)
2016-02-09 10:16:27 [scrapy] INFO: Optional features available: ssl, http11
2016-02-09 10:16:27 [scrapy] INFO: Overridden settings: {'REDIRECT_ENABLED': '0', 'LOGSTATS_INTERVAL': 0, 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter'}
2016-02-09 10:16:30 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2016-02-09 10:16:32 [scrapy] INFO: Enabled downloader middlewares:
HttpAuthMiddleware, 
DownloadTimeoutMiddleware, 
UserAgentMiddleware,
RetryMiddleware,
DefaultHeadersMiddleware,
MetaRefreshMiddleware,
HttpCompressionMiddleware,
CookiesMiddleware,
ChunkedTransferMiddleware,
DownloaderStats
2016-02-09 10:16:33 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-02-09 10:16:33 [scrapy] INFO: Enabled item pipelines: 
2016-02-09 10:16:33 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-02-09 10:16:39 [root] DEBUG: Using default logger

（您可以注意到，
重定向中间件
不在“已启用的下载程序中间件”列表中）

默认情况下：

$ scrapy shell
2016-02-09 10:17:18 [scrapy] INFO: Scrapy 1.0.4 started (bot: scrapybot)
2016-02-09 10:17:18 [scrapy] INFO: Optional features available: ssl, http11
2016-02-09 10:17:18 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0, 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter'}
2016-02-09 10:17:19 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2016-02-09 10:17:19 [scrapy] INFO: Enabled downloader middlewares:
HttpAuthMiddleware,
DownloadTimeoutMiddleware,
UserAgentMiddleware,
RetryMiddleware,
DefaultHeadersMiddleware,
MetaRefreshMiddleware,
HttpCompressionMiddleware,
RedirectMiddleware,
CookiesMiddleware,
ChunkedTransferMiddleware,
DownloaderStats
2016-02-09 10:17:19 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-02-09 10:17:19 [scrapy] INFO: Enabled item pipelines: 
2016-02-09 10:17:19 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-02-09 10:17:19 [root] DEBUG: Using default logger