如何使用Python获取Rottentomotos的观众评论？

我正在使用scrapy创建一个蜘蛛，从Rottontomatoes.com上获取细节。由于搜索页面是动态呈现的，所以我使用了RottomatoES API for eg:来获取搜索结果和URL。通过scrapy跟踪URL，我能够提取Tomotometer分数、观众分数、导演、演员等等。但是，我也想提取所有观众评论。问题是，观众评论页面（）使用分页，我无法从下一页提取数据，而且我也找不到使用API提取细节的方法。谁能帮我一下吗

    def parseRottenDetail(self, response):
    print("Reached Tomato Parser")
    try:
        if MoviecrawlSpider.current_parse <= MoviecrawlSpider.total_results:
            items = TomatoCrawlerItem()
            MoviecrawlSpider.parse_rotten_list[MoviecrawlSpider.current_parse]['tomatometerScore'] = response.css(
                '.mop-ratings-wrap__row .mop-ratings-wrap__half .mop-ratings-wrap__percentage::text').get().strip()
            MoviecrawlSpider.parse_rotten_list[MoviecrawlSpider.current_parse][
                'tomatoAudienceScore'] = response.css(
                '.mop-ratings-wrap__row .mop-ratings-wrap__half.audience-score .mop-ratings-wrap__percentage::text').get().strip()
            MoviecrawlSpider.parse_rotten_list[MoviecrawlSpider.current_parse][
                'tomatoCriticConsensus'] = response.css('p.mop-ratings-wrap__text--concensus::text').get()
            if MoviecrawlSpider.parse_rotten_list[MoviecrawlSpider.current_parse]["type"] == "Movie":
                MoviecrawlSpider.parse_rotten_list[MoviecrawlSpider.current_parse]['Director'] = response.xpath(
                    "//ul[@class='content-meta info']/li[@class='meta-row clearfix']/div[contains(text(),'Directed By')]/../div[@class='meta-value']/a/text()").get()
            else:
                MoviecrawlSpider.parse_rotten_list[MoviecrawlSpider.current_parse]['Director'] = response.xpath(
                    "//div[@class='tv-series__series-info-castCrew']/div/span[contains(text(),'Creator')]/../a/text()").get()
            reviews_page = response.css('div.mop-audience-reviews__view-all a[href*="reviews"]::attr(href)').get()
            if len(reviews_page) != 0:
                yield response.follow(reviews_page, callback=self.parseRottenReviews)
            else:
                for key in MoviecrawlSpider.parse_rotten_list[MoviecrawlSpider.current_parse].keys():
                    if "pageURL" not in key and "type" not in key:
                        items[key] = MoviecrawlSpider.parse_rotten_list[MoviecrawlSpider.current_parse][key]
                yield items
                if MoviecrawlSpider.current_parse <= MoviecrawlSpider.total_results:
                    MoviecrawlSpider.current_parse += 1
                    print("Parse Values are Current Parse " + str(
                        MoviecrawlSpider.current_parse) + "and Total Results " + str(MoviecrawlSpider.total_results))
                    yield response.follow(MoviecrawlSpider.parse_rotten_list[MoviecrawlSpider.current_parse]["pageURL"],
                                          callback=self.parseRottenDetail)
    except Exception as e:
        exc_type, exc_obj, exc_tb = sys.exc_info()
        print(e)
        print(exc_tb.tb_lineno)

---

当您转到下一页时，可以注意到它使用了该页的上一个结束光标值。您可以为第一次迭代设置带有空字符串的

endCursor

。还请注意，您需要电影id进行重新测试审查，该id可以从JS中嵌入的json中提取：

import requests
import re
import json

r = requests.get("https://www.rottentomatoes.com/m/inception/reviews?type=user")
data = json.loads(re.search('movieReview\s=\s(.*);', r.text).group(1))

movieId = data["movieId"]

def getReviews(endCursor):
    r = requests.get(f"https://www.rottentomatoes.com/napi/movie/{movieId}/reviews/user",
    params = {
        "direction": "next",
        "endCursor": endCursor,
        "startCursor": ""
    })
    return r.json()

reviews = []
result = {}
for i in range(0, 5):
    print(f"[{i}] request review")
    result = getReviews(result["pageInfo"]["endCursor"] if i != 0  else "")
    reviews.extend([t for t in result["reviews"]])

print(reviews)
print(f"got {len(reviews)} reviews")

---

请注意，您还可以在第一次迭代中删除html

因为我正在使用Scrapy，我正在寻找一种不使用requests模块执行此操作的方法。方法是相同的，但我发现页面在

标记中有一个对象

root.rottomatoos.context.fandangoData

，该对象有一个键“emsId”，该键具有电影的Id，该Id被传递给api以获取详细信息。通过查看每个分页事件的网络选项卡，我意识到他们使用startCursor和endCursor来过滤每个页面的结果

pattern = r'\broot.RottenTomatoes.context.fandangoData\s*=\s*(\{.*?\})\s*;\s*\n'
                    json_data = response.css('script::text').re_first(pattern)
                    movie_id = json.loads(json_data)["emsId"]
{SpiderClass}.movieId = movie_id
    next_page = "https://www.rottentomatoes.com/napi/movie/" + movie_id + "/reviews/user?direction=next&endCursor=&startCursor="
                    yield response.follow(next_page, callback=self.parseRottenReviews)

对于第一次迭代，可以将

startCursor

和

endCursor

参数留空。现在进入parse函数。您可以从当前响应中获取下一页的

startCursor

和

endCursor

参数。重复此操作，直到

hasNextPage

属性为false

def parseRottenReviews(self, response):
print("Reached Rotten Review Parser")
current_result = json.loads(response.text)
for review in current_result["reviews"]:
    {SpiderClass}.reviews.append(review) #Spider class memeber So that it could be shared among iterations
if current_result["pageInfo"]["hasNextPage"] is True:
    next_page = "https://www.rottentomatoes.com/napi/movie/" + \
                str({SpiderClass}.movieId) + "/reviews/user?direction=next&endCursor=" + str(
        current_result["pageInfo"][
            "endCursor"]) + "&startCursor=" + str(current_result["pageInfo"]["startCursor"])
    yield response.follow(next_page, callback=self.parseRottenReviews)

---

现在，

{SpiderClass}.reviews

数组中的reviews将非常有用，非常感谢：D

def parseRottenReviews(self, response):
print("Reached Rotten Review Parser")
current_result = json.loads(response.text)
for review in current_result["reviews"]:
    {SpiderClass}.reviews.append(review) #Spider class memeber So that it could be shared among iterations
if current_result["pageInfo"]["hasNextPage"] is True:
    next_page = "https://www.rottentomatoes.com/napi/movie/" + \
                str({SpiderClass}.movieId) + "/reviews/user?direction=next&endCursor=" + str(
        current_result["pageInfo"][
            "endCursor"]) + "&startCursor=" + str(current_result["pageInfo"]["startCursor"])
    yield response.follow(next_page, callback=self.parseRottenReviews)