Python 为什么Scrapy只能得到一些结果？（302重定向）

我写了一个Scrapy spider，从百度搜索结果中的一长串URL中获取文章内容

它一开始是有效的，但之后似乎就不再获取内容了——数千个URL中的所有其他URL都不会产生任何内容。它说，当我尝试这些URL时，Scrapy正在重定向（302）到这些URL，许多URL都有我试图获取的元素，但Scrapy没有得到这些数据

我花了几个小时想弄明白为什么。。。我尝试过调整Scrapy发出请求的速度，尝试过更改提取数据的代码，但仍然没有成功

以下是蜘蛛代码：

import json

class QuotesSpider(scrapy.Spider):
    name = "GET"

    def start_requests(self):
        data = []
        urls = []
        with open("/Users/.../PycharmProjects/GT/GT/links1.json") as f:
            for line in f:
                data.append(json.loads(line))

        for line in data:
            url = line['link']
            urls.append(url)

        global idx
        for idx, url in enumerate(urls):
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        for content in response.css("div.all-con"):
            yield {
                    'time': content.css("div.metadata-info p.time::text").get(),
                    'title': content.css("div.t-container-title h3::text").get(),
                    'text': "".join(content.css("div.l-container p::text").get()),
            }

我正在爬行的链接如下所示（大约有2500个）：

谢谢你的帮助

{"link": "http://www.baidu.com/link?url=g147bb1CGmddvvH2aoAEOqlFbyGcMACQpsc5ByxSZ_tF5hPcNXUdN3J3jP2XVcRDIDC4QFZ4hcqOFVSbshGpuq"}
{"link": "http://www.baidu.com/link?url=ptYcsVZSxMJ2YhLwSZuFjr_6ukftIdhl74589-olFOuqYe9rO-MxZhTLuPnyl55Lk2680r7rE59bEIMn1tHQ6a"}
{"link": "http://www.baidu.com/link?url=RsL4nmPDLYkCZ9KroyogLPgJ1t6ZnBX_N75OHlRLUXH40xJ2-0UfzLU0WtxzHcvOOuZUKtpBgaH0mzeUnSnvCK"}
{"link": "http://www.baidu.com/link?url=sqLCagxjItulhRLgg7hIqIl4HSnG5W9gCotVMEpMVDZtsFFY0cxvA0uVDFwOODlB520qOnLxIwqQ8_owlXmZr_"}
{"link": "http://www.baidu.com/link?url=euJcn2_Y5sbLzHlJKVtxFIiqipqv-THBEO60ZNm3eAKAQ9ccLfZhkmz_2bwltI3aXsFzVv4ZqiT7tGzg6C3vH_"}
{"link": "http://www.baidu.com/link?url=sbzqKKvvB91V1qpTXTsOwUO819odCOZgBIPj9g5xcttDRM_MiINnbQiMjyLhLhMd0ZJLqUqm90_3B9cUADprVk_mUtD5w8cFHQtxRnu5d33"}
{"link": "http://www.baidu.com/link?url=D5sm4ZzHLzO2zPGq-Oyi5_9K2Gp4vYU8DeLOOZ7ENj13te8eQGyyh6ugPCdkBIpgj1q4yi12KuZ5MzE6-iX_ddwpiGdZuvDe7YVSP5kcpPu"}
{"link": "http://www.baidu.com/link?url=B2o8aoFz1MTV6fqxx6puxTHN1PsFBzzkMba5wi0T1-ac_HD5z02qRuzAfsR5__pTuevcv1fGAUdD77M-U0rcIaUth0oHKFl2q-7136AuJ8e"}
{"link": "http://www.baidu.com/link?url=UiIg-ELqdzxPwxW5LrJYbBvQTWTe6nohtf6Mx85FAk2yV3iwIaKX3cI9yjub0AEQ-c3D6G47YsrBJ3h_yk_Sn1Vo3UGgqTcljyO9hJbcaCG"}
{"link": "http://www.baidu.com/link?url=JcKaRn6JAUCEvKhdh9w5jTPvuaUdh6e7mfktNdQmHZzGDmMtwn2rD6Nil0ret_y-IXbufdAucbZc5OTijj9kXtV87fmECYkxRt37-uwQPZy"}
{"link": "http://www.baidu.com/link?url=DR1uBZHMKGpcNLY44gKm_JIADKwQHr1aoh8QrfMP7P9IkRlQt2Ad10KtzkvvZS3VkSMA68oVDY2OpHysBNw3EumkLInsYB1JMpK55TzK7WW"}
{"link": "http://www.baidu.com/link?url=GR0tRhMRUQ8KpMIuK0oGzgujA8SglIkt9H7C9A1bc-Wlm8yxhdoGQ7ssdxGwvlK71xCFVBb9EQtDqtXU1aUUz_"}