Web scraping 如何删除IMDB评论

我想做一个从数据收集开始的情绪分析的端到端项目。为此，我从IMDB评论开始，特别是从本页开始：

我将为此使用scrapy，通过以下代码我可以获得评论和标题：

import requests
from scrapy.http import TextResponse
import urlparse
from urlparse import  urljoin

base_url = "http://www.imdb.com/title/tt2137109/reviews?start=0"
r = requests.get(base_url)

response = TextResponse(r.url, body=r.text, encoding='utf-8')

title = response.xpath('//*[contains(@id,"title")]//text()').re('".+"')[0]

reviews = response.xpath('//*[contains(@id,"1")]/p/text()').extract()

我遇到的问题是如何爬网站点以获得随机样本？ 我正在寻找一个10k标题的样本，我计划在5-10天内得到，以避免不必要的打击网站和被破坏

有一些起点，如前250名：

---

但是我在寻找一个随机样本。

这里是我能想到的可能的想法

    **#All U.S. Released Movies: 1972-2016 #**

    from requests import get
    from bs4 import BeautifulSoup
    import pandas as pd
    df=pd.DataFrame()
    for f in range(4,101):
      print(f)
      url = "https://www.imdb.com/list/ls057823854/?st_dt=&mode=detail&page="+str(f)+"&sort=release_date,desc"
      response = get(url)
      html_soup = BeautifulSoup(response.text, 'html.parser')  
      movie_containers = html_soup.find_all('div', class_ = 'lister-item mode-detail')
      names = []
      years = []
      imdb_ratings = []
      metascores = []
      votes = []
      plots=[]
      genre=[]
      lengths=[]
      collections=[]
      starss=[]
      directors=[]

      for container in movie_containers:

        if container.find('div', class_ = 'ratings-metascore') is not None or None:
          name = container.h3.a.text
          names.append(name)


          imdb =container.find('span', class_ = 'ipl-rating-star__rating').text
          imdb_ratings.append(imdb)


          year = container.h3.find('span', class_ = 'lister-item-year').text
          years.append(year)

          m_score = container.find('span', class_ = 'metascore').text
          metascores.append(int(m_score))

          b=container.find_all('span', attrs = {'name':'nv'})
          vote = b[0].text
          votes.append(vote)

          if len(b)==2:
            collection=b[1].text
            collections.append(collection)

          else:
            collections.append('0')


          par=container.find_all('p')
          length= par[0].find('span',class_='genre').text
          genre.append(length)

          length= par[0].find('span',class_='runtime').text
          lengths.append(length)

          plot= par[1].text
          plots.append(plot)
          stars_director=container.find_all('p')[2].text
          directors.append(stars_director)

      test_df = pd.DataFrame({'movie': names,
      'year': years,
      'imdb': imdb_ratings,
      'metascore': metascores,
      'votes': votes,
      'Plot':plots,
      'genre':genre,
      'duration':lengths,
      "revenue":collections,
      "directors":directors
      })

      df=pd.concat([df,test_df]) 

      if f%10==0:
        df.to_csv(str(f)+"page.csv")

并下载名为title.akas.tsv.gz的zip文件

它包含imdb数据库中moveis的所有平铺，并编写一个函数从文件中选择随机项，然后在列表中循环

通过将代码更改为

    import requests
    from scrapy.http import TextResponse
    import urlparse
    from urlparse import  urljoin
    def random_list():
        #this should read the file and give random tiles in a list form once called.

    for i in random_list():
        base_url = f"http://www.imdb.com/title/{i}/reviews?start=0"
        r = requests.get(base_url)

        response = TextResponse(r.url, body=r.text, encoding='utf-8')

        title = response.xpath('//*[contains(@id,"title")]//text()').re('".+"')[0]

        reviews = response.xpath('//*[contains(@id,"1")]/p/text()').extract()

imdb的文档如下所示

1）如果使用scrapy，为什么要使用请求？2） 提取链接并对随机样本使用随机函数1）我使用请求是因为我在笔记本上尝试XPath。我可以从哪里随机抽取样本？我没有单子，你试过了吗？