Python 3.x 使用Try:Except:in For循环进行网页抓取

我已经编写了下面的代码，试图使用Python、Pandas等来练习web抓取。一般来说，我尝试遵循四个步骤来实现所需的输出：

获取要附加到基本url的名称列表 创建特定于播放器的URL列表 使用播放器URL刮表 将玩家名称添加到我刮表中，以跟踪哪个玩家属于哪个统计数据-因此在表的每一行中添加一列，其中包含用于刮表的玩家名称 我能让1号和2号工作。3的组件似乎可以工作，但我相信我的尝试有问题：除了因为如果我只运行一行代码来刮取一个特定的playerUrl表DF会按预期填充。第一个被抓取的玩家没有数据，所以我认为我的错误捕捉失败了

4年来，我真的没有找到解决办法。在for循环中迭代时，如何将名称添加到列表中

感谢您的帮助

import requests
import pandas as pd
from bs4 import BeautifulSoup



### get the player data to create player specific urls

res = requests.get("https://www.mlssoccer.com/players?page=0")
soup = BeautifulSoup(res.content,'html.parser')
data = soup.find('div', class_ = 'item-list' )

names=[]


for player in data:
    name = data.find_all('div', class_ = 'name')
    for obj in name:
        names.append(obj.find('a').text.lower().lstrip().rstrip().replace(' ','-'))


### create a list of player specific urls
url = 'https://www.mlssoccer.com/players/'
playerUrl = []
x = 0
for name in (names):
    playerList = names
    newUrl = url + str(playerList[x])
    print("Gathering url..."+newUrl)
    playerUrl.append(newUrl)
    x +=1

### now take the list of urls and gather stats tables

tbls = []
i = 0
for url in (playerUrl):
    try:                                                        ### added the try, except, pass because some players have no stats table
        tables = pd.read_html(playerUrl[i], header = 0)[2]
        tbls.append(tables)
        i +=1
    except Exception:
        continue

---

您可以做几件事来改进代码并完成步骤3和步骤4

在名称循环中使用for name时，不需要显式使用索引，只需使用变量名即可。 你可以将玩家的名字和相应的URL保存为dict，其中名字是关键。然后在步骤3/4中，您可以使用该名称 iii为每个解析的HTML表构造一个数据框，并将播放器的名称附加到其中。单独保存此数据帧。 最后，将这些数据帧组成一个单独的数据帧

以下是使用上述建议更改修改的代码：

import requests
import pandas as pd
from bs4 import BeautifulSoup



### get the player data to create player specific urls

res = requests.get("https://www.mlssoccer.com/players?page=0")
soup = BeautifulSoup(res.content,'html.parser')
data = soup.find('div', class_ = 'item-list' )

names=[]

for player in data:
    name = data.find_all('div', class_ = 'name')
    for obj in name:
        names.append(obj.find('a').text.lower().lstrip().rstrip().replace(' ','-'))


### create a list of player specific urls
url = 'https://www.mlssoccer.com/players/'
playerUrl = {}
x = 0
for name in names:
    newUrl = url + str(name)
    print("Gathering url..."+newUrl)
    playerUrl[name] = newUrl

### now take the list of urls and gather stats tables

tbls = []
for name, url in playerUrl.items():
    try:                                                        
        tables = pd.read_html(url, header = 0)[2]
        df = pd.DataFrame(tables)
        df['Player'] = name
        tbls.append(df)
    except Exception as e:
        print(e)
        continue

result = pd.concat(tbls)
print(result.head())

---

您的脚本中存在大量冗余。您可以按照以下步骤清理它们。首先，我使用了select而不是find_all来消除冗长的内容。要摆脱该索引器，您可以使用continue关键字，如下所示：

import requests
import pandas as pd
from bs4 import BeautifulSoup

base_url = "https://www.mlssoccer.com/players?page=0"
url = 'https://www.mlssoccer.com/players/'

res = requests.get(base_url)
soup = BeautifulSoup(res.text,'lxml')
names = []
for player in soup.select('.item-list .name a'):
    names.append(player.get_text(strip=True).replace(" ","-"))

playerUrl = {}
for name in names:
    playerUrl[name] = f'{url}{name}'

tbls = []
for url in playerUrl.values():
    if len(pd.read_html(url))<=2:continue
    tables = pd.read_html(url, header=0)[2]
    tbls.append(tables)

print(tbls)

---

谢谢你！我不知道选择选项而不是查找所有。