ESPN.com Python网页抓取问题

我试图为所有大学足球队的名单收集数据，因为我想根据他们的名单组成对球队表现进行一些分析

我的脚本在第一个页面上运行，它在每个团队中迭代，可以打开每个团队的“名册”链接，但是我在团队的“名册”页面上运行的漂亮的“汤”命令会不断抛出索引错误。当我查看HTML时，似乎我正在编写的命令应该可以工作，但当我从漂亮的Soup打印页面源代码时，我看不到我在Chrome开发工具中看到的内容。这是JS用来提供内容的实例吗？如果是这样的话，我想Selenium解决了这个问题

我的代码

import requests
import csv
from bs4 import BeautifulSoup
from selenium import webdriver

teams_driver = webdriver.Firefox()
teams_driver.get("http://www.espn.com/college-football/teams")
teams_html = teams_driver.page_source
teams_soup = BeautifulSoup(teams_html, "html5lib")

i = 0

for link_html in teams_soup.find_all('a'):
    if link_html.text == 'Roster':
        roster_link = 'https://www.espn.com' + link_html['href']

        roster_driver = webdriver.Firefox()
        roster_driver.get(roster_link)
        roster_html = teams_driver.page_source
        roster_soup = BeautifulSoup(roster_html, "html5lib")

        team_name_html = roster_soup.find_all('a', class_='sub-brand-title')[0]
        team_name = team_name_html.find_all('b')[0].text

        for player_html in roster_soup.find_all('tr', class_='oddrow'):
            player_name = player_html.find_all('a')[0].text
            player_pos = player_html.find_all('td')[2].text
            player_height = player_html.find_all('td')[3].text
            player_weight = player_html.find_all('td')[4].text
            player_year = player_html.find_all('td')[5].text
            player_hometown = player_html.find_all('td')[6].text

            print(team_name)
            print('\t', player_name)

        roster_driver.close()

teams_driver.close()

---

在for循环中，您使用的是第1页的html（

floster\u html=teams\u driver.page\u source

），因此当您尝试选择

team\u name\u html

的第1项时，会出现索引错误，因为

find\u all

返回一个空列表

另外，您不需要打开所有的

Firefox

实例，您可以在打开html时关闭驱动程序

teams_driver = webdriver.Firefox()
teams_driver.get("http://www.espn.com/college-football/teams")
teams_html = teams_driver.page_source
teams_driver.quit()

但是您不必为此任务使用

selenium

，您可以通过

请求和
bs4
获取所有数据

import requests
from bs4 import BeautifulSoup

r = requests.get("http://www.espn.com/college-football/teams")
teams_soup = BeautifulSoup(r.text, "html5lib")

for link_html in teams_soup.find_all('a'):
    if link_html.text == 'Roster':
        roster_link = 'https://www.espn.com' + link_html['href']
        r = requests.get(roster_link)
        roster_soup = BeautifulSoup(r.text, "html5lib")

        team_name = roster_soup.find('a', class_='sub-brand-title').find('b').text
        for player_html in roster_soup.find_all('tr', class_='oddrow'):
            player_name = player_html.find_all('a')[0].text
            player_pos = player_html.find_all('td')[2].text
            player_height = player_html.find_all('td')[3].text
            player_weight = player_html.find_all('td')[4].text
            player_year = player_html.find_all('td')[5].text
            player_hometown = player_html.find_all('td')[6].text
            print(team_name, player_name, player_pos, player_height, player_weight, player_year, player_hometown)

哇！复制和粘贴注定了我的命运。我已经看了几个晚上了，甚至“重构”了代码，尝试从不同的站点抓取。这根本不是网站。谢谢，任何人都会这样。有时它就在你面前，但你看不见它。