Python 不接受Scrapy CSRF Cookie并导致302重定向_Python_Scrapy_Session Cookies_Csrf

Python 不接受Scrapy CSRF Cookie并导致302重定向

python scrapy

Python 不接受Scrapy CSRF Cookie并导致302重定向,python,scrapy,session-cookies,csrf,Python,Scrapy,Session Cookies,Csrf,我对编程基本上是新手，所以我希望这不是一个愚蠢的问题。我在谷歌上搜索过，花了4个小时试图解决这个问题，但没有找到答案，因此我非常感谢为解决这个问题提供的建议/步骤。谢谢大家! 以下是到目前为止我对蜘蛛的看法：从scrapy.spider导入BaseSpider 从scrapy.selector导入HtmlXPathSelector 从tutorial.items导入TutorialItem 从scrapy.http导入FormRequest，请求 class LoginSpider(BaseS

我对编程基本上是新手，所以我希望这不是一个愚蠢的问题。我在谷歌上搜索过，花了4个小时试图解决这个问题，但没有找到答案，因此我非常感谢为解决这个问题提供的建议/步骤。谢谢大家!

以下是到目前为止我对蜘蛛的看法：从scrapy.spider导入BaseSpider 从scrapy.selector导入HtmlXPathSelector 从tutorial.items导入TutorialItem 从scrapy.http导入FormRequest，请求

class LoginSpider(BaseSpider):
    name = 'pinterest'
    start_urls = ['https://www.pinterest.com/login/']

    def parse(self, response):
        return FormRequest.from_response(response,
                    formdata={'username_or_email': '...', 'password': '...'},
                    callback=self.after_login, dont_filter = True)

    def after_login(self, response):
        print response.url

据我所知，Scrapy自动处理cookies，因此CSRF令牌被传递。我在我的设置中将COOKIES\u设置为ENABLED，COOKIES\u DEBUG设置为True：

SPIDER_MIDDLEWARES = {'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware':     
700,}
USER_AGENT = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like 
Gecko) Chrome/29.0.1547.66 Safari/537.36"
COOKIES_ENABLED = True
COOKIES_DEBUG = True

以下是调试的输出：

2013-09-27 11:11:42-0700 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Received cookies from: <200 https://
www.pinterest.com/login/>
        Set-Cookie: csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K; Domain=.pinteres
t.com; expires=Fri, 26-Sep-2014 18:11:46 GMT; Max-Age=31449600; Path=/
        Set-Cookie: _pinterest_sess="eJwr9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8
vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfXMyjb0c/c0AIpX+Ia4ZfpmBeX4uqSbRFYlG0SFuFb
4ZjlWRLkHGkZWuRp6AvUBAEY1IrA="; Domain=.pinterest.com; expires=Mon, 22-Sep-2014
18:11:46 GMT; Max-Age=31103999; Path=/
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Crawled (200) <GET https://www.pinte
rest.com/login/> (referer: None)
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Sending cookies to: <POST https://ww
w.pinterest.com/login/>
        Cookie: csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K; _pinterest_sess="eJw
r9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfX
Myjb0c/c0AIpX+Ia4ZfpmBeX4uqSbRFYlG0SFuFb4ZjlWRLkHGkZWuRp6AvUBAEY1IrA="
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Redirecting (302) to <GET http://www
.pinterest.com/csrf_error/> from <POST https://www.pinterest.com/login/>
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Sending cookies to: <GET http://www.
pinterest.com/csrf_error/>
        Cookie: csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K; _pinterest_sess="eJw
r9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfX
Myjb0c/c0AIpX+Ia4ZfpmBeX4uqSbRFYlG0SFuFb4ZjlWRLkHGkZWuRp6AvUBAEY1IrA="
2013-09-27 11:11:44-0700 [pinterest] DEBUG: Crawled (200) <GET http://www.pinter
est.com/csrf_error/> (referer: https://www.pinterest.com/login/)
http://www.pinterest.com/csrf_error/

2013-09-27 11:11:42-0700[scrapy]调试：在0.0.0.0:6080上侦听Web服务
2013-09-27 11:11:43-0700[pinterest]调试：从以下地址接收cookie：
设置Cookie:csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K；域=.pinteres
t、 com；expires=2014年9月26日星期五18:11:46 GMT；最大年龄=31449600；路径=/
设置Cookie:\u pinterest\u sess=“eJwr9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8
vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfXMyjb0c/c0AIpX+IA4ZFPMBEEX4UQSBRFYLG0SFUFB
4ZjlWRLkHGkZWuRp6AvUBAEY1IrA=“；域名=.pinterest.com；到期日期=2014年9月22日星期一
格林尼治标准时间18:11:46；最大年龄=31103999；路径=/
2013-09-27 11:11:43-0700[pinterest]调试：已爬网（200）（参考：无）
2013-09-27 11:11:43-0700[pinterest]调试：将cookie发送到：
Cookie:csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K_pinterest_sess=“eJw
r9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfX
Myjb0c/c0AIpX+Ia4ZfpmBeX4uqSbRFYlG0SFuFb4ZjlWRLkHGkZWuRp6AvUBAEY1IrA=”
2013-09-27 11:11:43-0700[pinterest]调试：重定向（302）到
2013-09-27 11:11:43-0700[pinterest]调试：将cookie发送到：
Cookie:csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K_pinterest_sess=“eJw
r9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfX
Myjb0c/c0AIpX+Ia4ZfpmBeX4uqSbRFYlG0SFuFb4ZjlWRLkHGkZWuRp6AvUBAEY1IrA=”
2013-09-27 11:11:44-0700[pinterest]调试：爬网（200）（参考：https://www.pinterest.com/login/)
http://www.pinterest.com/csrf_error/

问题是，在设置cookie并将其发送到登录页面后，我仍然会收到CSRF错误并被重定向。我是不是做错了什么，不能用浏览器模拟登录过程？我尝试将用户代理设置为iPhone，代码为200，没有重定向，但response.url显示为“”，因此它仍然无法正确登录

真的很感谢我能得到的所有帮助。谢谢大家!

看起来非移动登录页面使用XHR请求来执行登录。您可以尝试深入研究XHR请求，也许还有javascript代码，看看为了在scrapy中重现请求必须做些什么

但是，正如您所指出的，有一个移动登录页面可以通过更改用户代理来启用

后一种方法的问题是，电子邮件的字段是

email

，而不是

username\u或\u email

这是一个有细微变化的爬行器：

from scrapy.http import FormRequest
from scrapy.spider import BaseSpider


class LoginSpider(BaseSpider):
    name = 'pinterest'
    start_urls = ['https://www.pinterest.com/login/']
    # you can set the user agent either in the settings or the spider
    user_agent = ('Mozilla/5.0 (iPhone; CPU iPhone OS 5_0 like Mac OS X) '
                  'AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 '
                  'Mobile/9A334 Safari/7534.48.3')

    def parse(self, response):
        data = {'email': 'XXX@xxx', 'password': 'xxx'}
        # no need for dont_filter
        return FormRequest.from_response(response, formdata=data, callback=self.after_login)

    def after_login(self, response):
        print response.url

输出：

$ scrapy runspider pinterest.py
2013-09-28 19:16:58-0400 [scrapy] INFO: Scrapy 0.16.5 started (bot: scrapybot)
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled item pipelines: 
2013-09-28 19:16:58-0400 [pinterest] INFO: Spider opened
2013-09-28 19:16:58-0400 [pinterest] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-09-28 19:17:01-0400 [pinterest] DEBUG: Crawled (200) <GET https://www.pinterest.com/login/> (referer: None)
2013-09-28 19:17:09-0400 [pinterest] DEBUG: Redirecting (302) to <GET http://www.pinterest.com/> from <POST https://www.pinterest.com/login/?next=%2Flogin%2F>
2013-09-28 19:17:09-0400 [pinterest] DEBUG: Redirecting (302) to <GET http://www.pinterest.com/join/discover/> from <GET http://www.pinterest.com/>
2013-09-28 19:17:10-0400 [pinterest] DEBUG: Crawled (200) <GET http://www.pinterest.com/join/discover/> (referer: https://www.pinterest.com/login/)
http://www.pinterest.com/join/discover/

$scrapy runspider pinterest.py
2013-09-28 19:16:58-0400[scrapy]信息：scrapy 0.16.5已启动（bot:scrapybot）
2013-09-28 19:16:58-0400[scrapy]调试：启用的扩展：LogStats、TelnetConsole、CloseSpider、WebService、CoreStats、SpiderState
2013-09-28 19:16:58-0400[scrapy]调试：启用的下载中间件：HttpAuthMiddleware、DownloadTimeoutMiddleware、UserAgentMiddleware、RetryMiddleware、DefaultHeadersMiddleware、RedirectMiddleware、Cookies中间件、HttpCompressionMiddleware、ChunkedTransferMiddleware、DownloadersStats
2013-09-28 19:16:58-0400[scrapy]调试：启用的spider中间件：HttpErrorMiddleware、OffsiteMiddleware、RefererMiddleware、UrlLengthMiddleware、DepthMiddleware
2013-09-28 19:16:58-0400[scrapy]调试：启用的项目管道：
2013-09-28 19:16:58-0400[pinterest]信息：蜘蛛开启
2013-09-28 19:16:58-0400[pinterest]信息：爬网0页（以0页/分钟的速度），抓取0项（以0项/分钟的速度）
2013-09-28 19:16:58-0400[scrapy]调试：Telnet控制台在0.0.0.0上侦听：6023
2013-09-28 19:16:58-0400[scrapy]调试：Web服务侦听0.0.0.0:6080
2013-09-28 19:17:01-0400[pinterest]调试：爬网（200）（参考：无）
2013-09-28 19:17:09-0400[pinterest]调试：重定向（302）到
2013-09-28 19:17:09-0400[pinterest]调试：重定向（302）到
2013-09-28 19:17:10-0400[pinterest]调试：爬网（200）（参考：https://www.pinterest.com/login/)
http://www.pinterest.com/join/discover/

看起来非移动登录页面使用XHR请求来执行登录。您可以尝试深入研究XHR请求，也许还有javascript代码，看看为了在scrapy中重现请求必须做些什么

但是，正如您所指出的，有一个移动登录页面可以通过更改用户代理来启用

后一种方法的问题是，电子邮件的字段是

email

，而不是

username\u或\u email

这是一个有细微变化的爬行器：

from scrapy.http import FormRequest
from scrapy.spider import BaseSpider


class LoginSpider(BaseSpider):
    name = 'pinterest'
    start_urls = ['https://www.pinterest.com/login/']
    # you can set the user agent either in the settings or the spider
    user_agent = ('Mozilla/5.0 (iPhone; CPU iPhone OS 5_0 like Mac OS X) '
                  'AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 '
                  'Mobile/9A334 Safari/7534.48.3')

    def parse(self, response):
        data = {'email': 'XXX@xxx', 'password': 'xxx'}
        # no need for dont_filter
        return FormRequest.from_response(response, formdata=data, callback=self.after_login)

    def after_login(self, response):
        print response.url

输出：

$ scrapy runspider pinterest.py
2013-09-28 19:16:58-0400 [scrapy] INFO: Scrapy 0.16.5 started (bot: scrapybot)
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled item pipelines: 
2013-09-28 19:16:58-0400 [pinterest] INFO: Spider opened
2013-09-28 19:16:58-0400 [pinterest] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-09-28 19:17:01-0400 [pinterest] DEBUG: Crawled (200) <GET https://www.pinterest.com/login/> (referer: None)
2013-09-28 19:17:09-0400 [pinterest] DEBUG: Redirecting (302) to <GET http://www.pinterest.com/> from <POST https://www.pinterest.com/login/?next=%2Flogin%2F>
2013-09-28 19:17:09-0400 [pinterest] DEBUG: Redirecting (302) to <GET http://www.pinterest.com/join/discover/> from <GET http://www.pinterest.com/>
2013-09-28 19:17:10-0400 [pinterest] DEBUG: Crawled (200) <GET http://www.pinterest.com/join/discover/> (referer: https://www.pinterest.com/login/)
http://www.pinterest.com/join/discover/

$scrapy runspider pinterest.py
2013-09-28 19:16:58-0400[scrapy]信息：scrapy 0.16.5已启动（bot:scrapybot）
2013-09-28 19:16:58-0400[scrapy]调试：启用的扩展：LogStats、TelnetConsole、CloseSpider、WebService、CoreStats、SpiderState
2013-09-28 19:16:58-0400[scrapy]调试：启用的下载中间件：HttpAuthMiddleware、DownloadTimeoutMiddleware、UserAgentMiddleware、RetryMiddleware、DefaultHeadersMiddleware、RedirectMiddleware、Cookies中间件、HttpCompressionMiddleware、ChunkedTransferMiddleware、DownloadersStats
2013-09-28 19:16:58-0400[scrapy]调试：启用的spider中间件：HttpErrorMiddleware、OffsiteMiddleware、RefererMiddleware、UrlLengthMiddleware、DepthMiddleware
2013-09-28 19:16:58-0400[scrapy]调试：启用项目pi