Python 刮擦和重定向

因此，我一直在用scrapy构建一个web刮板，并正在进行一些数据验证，以确保所有项目都被正确抓取

我用它来获取文章标题、名字、掌声、回复等的中间数据，这些都在一个html页面上。但是为了得到标签，我需要阅读每一篇文章。我得到的大多数标签都是链接到towardsdatascience.com而不是媒体网站的。例如，它在文章的链接处做了一个奇怪的小重定向

然后它重定向到：

现在我注意到，在重定向到towardsdatascience页面的文章中，它没有抓住页面的标签。标签css选择器与它获取的其他媒体文章完全相同

当我进入scrapy shell并尝试获取一篇链接到Datascience文章的文章时，我得到了这个响应

fetch("https://towardsdatascience.com/cython-a-speed-up-tool-for-your-python-function-9bab64364bfd? 
source=tag_archive---------1-----------------------")



**OUTPUT**

2020-02-16 11:52:31 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows 
NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36

2020-02-16 11:52:31 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET 
https://medium.com/m/global-identity?redirectUrl=https%3A%2F%2Ftowardsdatascience.com%2Fcython-a- 
speed-up-tool-for-your-python-function-9bab64364bfd%3Fsource%3Dtag_archive---------1----------------- 
------> from <GET https://towardsdatascience.com/cython-a-speed-up-tool-for-your-python-function- 
9bab64364bfd?source=tag_archive---------1----------------------->

2020-02-16 11:52:31 [scrapy.downloadermiddlewares.robotstxt] DEBUG: Forbidden by robots.txt: <GET 
https://medium.com/m/global-identity?redirectUrl=https%3A%2F%2Ftowardsdatascience.com%2Fcython-a- 
speed-up-tool-for-your-python-function-9bab64364bfd%3Fsource%3Dtag_archive---------1----------------- 
------>

我从这个网站上尝试了一些使用scrapy的重定向方法，但没有成功。下面是实际爬虫程序的代码

代码

 import scrapy
 from dateutil.parser import parse
 from medium.items import MediumItem
 from scrapy.spiders import CrawlSpider

 class DataSpider(CrawlSpider):

    name = 'data'
    allowed_domains = ['medium.com', 'towardsdatascience.com']
    start_urls = ['https://medium.com/tag/python/archive/2020/02/01']

    def parse(self,response):


    articles = response.xpath('//div[@class="postArticle postArticle--short js-postArticle js- 
    trackPostPresentation js-trackPostScrolls"]')

    for article in articles:

        item = MediumItem()

        if article.css("div > h3::text").extract_first():
             item['Title'] = article.css("div > h3::text").extract_first()

             item['Name'] = article.xpath('.//a[@class="ds-link ds-link--styleSubtle link link-- 
             darken link--accent u-accentColor--textNormal u-accentColor--  
             textDarken"]/text()').extract_first()

             item['Date'] = parse(article.css('time::text').extract_first()).date()

             item['Read'] = article.css("span::attr(title)").extract_first()

             item['Publication'] = article.xpath('.//a[@class="ds-link ds-link--styleSubtle link-- 
             darken link--accent u-accentColor--textNormal"]/text()').extract_first()

             item['Claps'] = articles.xpath('.//button[@class="button button--chromeless u-baseColor- 
             -buttonNormal js-multirecommendCountButton u- 
             disablePointerEvents"]/text()').extract_first()

             item['Responses'] = article.xpath('.//a[@class="button button--chromeless u-baseColor-- 
            buttonNormal"]/text()').extract_first()

             link = article.xpath('.//a[@class="button button--smaller button--chromeless u- 
             baseColor--buttonNormal"]/@href').extract_first()

            yield response.follow(link, callback=self.get_link, meta={'item':item})


  def get_link(self,response):
        item = response.meta['item']
        item['Tags'] = response.css("ul > li > a::text").getall()
        yield item

---

从这些页面（如链接的页面）获取标签的任何帮助都将非常有用。

感谢@furas comment。他们的答案是正确的，但我需要代表：）

---

设置如何？是的，Furas，这是绝对正确的！很抱歉迟了答复，你也回答了我的另一个问题。我真的很感谢你的帮助！那你能接受我的回答吗？我知道我有点偷了它（尽管我做了他的评论），但是如果你接受的话，任何其他搜索的人都可以很容易地看到答案。加上我是h03的代表。

 import scrapy
 from dateutil.parser import parse
 from medium.items import MediumItem
 from scrapy.spiders import CrawlSpider

 class DataSpider(CrawlSpider):

    name = 'data'
    allowed_domains = ['medium.com', 'towardsdatascience.com']
    start_urls = ['https://medium.com/tag/python/archive/2020/02/01']

    def parse(self,response):


    articles = response.xpath('//div[@class="postArticle postArticle--short js-postArticle js- 
    trackPostPresentation js-trackPostScrolls"]')

    for article in articles:

        item = MediumItem()

        if article.css("div > h3::text").extract_first():
             item['Title'] = article.css("div > h3::text").extract_first()

             item['Name'] = article.xpath('.//a[@class="ds-link ds-link--styleSubtle link link-- 
             darken link--accent u-accentColor--textNormal u-accentColor--  
             textDarken"]/text()').extract_first()

             item['Date'] = parse(article.css('time::text').extract_first()).date()

             item['Read'] = article.css("span::attr(title)").extract_first()

             item['Publication'] = article.xpath('.//a[@class="ds-link ds-link--styleSubtle link-- 
             darken link--accent u-accentColor--textNormal"]/text()').extract_first()

             item['Claps'] = articles.xpath('.//button[@class="button button--chromeless u-baseColor- 
             -buttonNormal js-multirecommendCountButton u- 
             disablePointerEvents"]/text()').extract_first()

             item['Responses'] = article.xpath('.//a[@class="button button--chromeless u-baseColor-- 
            buttonNormal"]/text()').extract_first()

             link = article.xpath('.//a[@class="button button--smaller button--chromeless u- 
             baseColor--buttonNormal"]/@href').extract_first()

            yield response.follow(link, callback=self.get_link, meta={'item':item})


  def get_link(self,response):
        item = response.meta['item']
        item['Tags'] = response.css("ul > li > a::text").getall()
        yield item

# settings.py

ROBOTSTXT_OBEY = False