Python 使用Selenium加载页面和抓取信息时出现问题

我正在尝试从Selenium中提取前200名艺术家，但在使用.click（）通过Selenium成功加载所有200名艺术家后，单击“加载更多”按钮和.implicit_wait（）函数，并将它们插入我的DB中，我只得到前50名艺术家

第一位艺术家有一个不同的css选择器，所以我通过这段代码收集了艺术家的其余部分

the_rest=soup.选择（“#site_wrap>div.l-page\u content.l-page\u charts.l-page\u charts--single.l-page\u charts--artists>div.l-section.l-section\u charts--single>div.c-content.c-content--charts>section.l-section\u charts--grid>section.l-section\u charts.c-charts--table--single code>

然后我用这个for循环来插入艺术家

for rest in the_rest:
    rest_ranking = rest.select_one(
        'div.c-chart__table--top > div.c-chart__table--rank').text
    # while rest_ranking <= '200':
    rest_img = rest.select_one('div.c-chart__table--top > img')['src']
    rest_name = rest.select_one('div.c-chart__table--top > div.c-chart__table--details > div').text
    doc = {
        'name': rest_name,
        'img_url': rest_img,
        'lyrics': [],
        'probability': 0
    }
    db.selen_top200.insert_one(doc)

对于_rest中的rest：
rest\u排名=rest.选择一个(
“div.c-chart\u table--top>div.c-chart\u table--rank”）。文本
#当rest\u排名时，我尝试用这个选择器单击load more按钮
div.load-more-link span
with
。使用循环执行脚本（…）

首先，请执行以下导入操作：

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

请尝试以下代码以获取排名、名称和图像源：

driver.get('https://www.rollingstone.com/charts/artists/')
 
target_artis = 200
list_artist = []
while True:
    if(len(driver.find_elements_by_css_selector('div.load-more-link span')) >0):
        load_more = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'div.load-more-link span')))
        driver.execute_script("arguments[0].click()", load_more)
 
    list_artist = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, '.l-section__charts')))
    if(len(list_artist)>=target_artis):
        break
 
print(len(list_artist))
 
list_rest_ranking = []
list_rest_img = []
list_rest_name = []
 
for x in range(target_artis):
    list_rest_ranking.append(driver.find_elements_by_css_selector('div.c-chart__table--rank')[x].text)
    list_rest_img.append(driver.find_elements_by_css_selector('img.c-chart__table--cover')[x].get_attribute('src'))
    list_rest_name.append(driver.find_elements_by_css_selector('div.c-chart__table--title p')[x].text)

print(list_rest_ranking)
print(list_rest_img)
print(list_rest_name)

控制台输出：

[u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'23', u'24', u'25', u'26', u'27', u'28', u'29', u'30', u'31', u'32', u'33', u'34', u'35', u'36', u'37', u'38', u'39', u'40', u'41', u'42', u'43', u'44', u'45', u'46', u'47', u'48', u'49', u'50', u'51', u'52', u'53', u'54', u'55', u'56', u'57', u'58', u'59', u'60', u'61', u'62', u'63', u'64', u'65', u'66', u'67', u'68', u'69', u'70', u'71', u'72', u'73', u'74', u'75', u'76', u'77', u'78', u'79', u'80', u'81', u'82', u'83', u'84', u'85', u'86', u'87', u'88', u'89', u'90', u'91', u'92', u'93', u'94', u'95', u'96', u'97', u'98', u'99', u'100', u'101', u'102', u'103', u'104', u'105', u'106', u'107', u'108', u'109', u'110', u'111', u'112', u'113', u'114', u'115', u'116', u'117', u'118', u'119', u'120', u'121', u'122', u'123', u'124', u'125', u'126', u'127', u'128', u'129', u'130', u'131', u'132', u'133', u'134', u'135', u'136', u'137', u'138', u'139', u'140', u'141', u'142', u'143', u'144', u'145', u'146', u'147', u'148', u'149', u'150', u'151', u'152', u'153', u'154', u'155', u'156', u'157', u'158', u'159', u'160', u'161', u'162', u'163', u'164', u'165', u'166', u'167', u'168', u'169', u'170', u'171', u'172', u'173', u'174', u'175', u'176', u'177', u'178', u'179', u'180', u'181', u'182', u'183', u'184', u'185', u'186', u'187', u'188', u'189', u'190', u'191', u'192', u'193', u'194', u'195', u'196', u'197', u'198', u'199', u'200']
[u'https://api.buzzanglemusic.com/images/artists/1374', u'https://api.buzzanglemusic.com/images/artists/1075289', u'https://api.buzzanglemusic.com/images/artists/2543301', u'https://api.buzzanglemusic.com/images/artists/4384780', u'https://api.buzzanglemusic.com/images/artists/4609765', u'https://api.buzzanglemusic.com/images/artists/5562543', u'https://api.buzzanglemusic.com/images/artists/1347', u'https://api.buzzanglemusic.com/images/artists/1350', u'https://api.buzzanglemusic.com/images/artists/6762453', u'https://api.buzzanglemusic.com/images/artists/1151925', u'https://api.buzzanglemusic.com/images/artists/1598398', u'https://api.buzzanglemusic.com/images/artists/6273541', u'https://api.buzzanglemusic.com/images/artists/197', u'https://api.buzzanglemusic.com/images/artists/4', u'https://api.buzzanglemusic.com/images/artists/6240779', u'https://api.buzzanglemusic.com/images/artists/10996', u'https://api.buzzanglemusic.com/images/artists/5019556', u'https://api.buzzanglemusic.com/images/artists/5330407', u'https://api.buzzanglemusic.com/images/artists/4878653', u'https://api.buzzanglemusic.com/images/artists/6462708', u'https://api.buzzanglemusic.com/images/artists/7501280', u'https://api.buzzanglemusic.com/images/artists/1340', u'https://api.buzzanglemusic.com/images/artists/931', u'https://api.buzzanglemusic.com/images/artists/4034234', u'https://api.buzzanglemusic.com/images/artists/4893654', u'https://api.buzzanglemusic.com/images/artists/6218616', u'https://api.buzzanglemusic.com/images/artists/3105901', u'https://api.buzzanglemusic.com/images/artists/5764159', u'https://api.buzzanglemusic.com/images/artists/1344', u'https://api.buzzanglemusic.com/images/artists/1380', u'https://api.buzzanglemusic.com/images/artists/202', u'https://api.buzzanglemusic.com/images/artists/162', u'https://api.buzzanglemusic.com/images/artists/804921', u'https://api.buzzanglemusic.com/images/artists/1345', u'https://api.buzzanglemusic.com/images/artists/19621', u'https://api.buzzanglemusic.com/images/artists/808927', u'https://api.buzzanglemusic.com/images/artists/223', u'https://api.buzzanglemusic.com/images/artists/1514', u'https://api.buzzanglemusic.com/images/artists/7', u'https://api.buzzanglemusic.com/images/artists/21', u'https://api.buzzanglemusic.com/images/artists/590329', u'https://api.buzzanglemusic.com/images/artists/7466212', u'https://api.buzzanglemusic.com/images/artists/263', u'https://api.buzzanglemusic.com/images/artists/594558', u'https://api.buzzanglemusic.com/images/artists/7520899', u'https://api.buzzanglemusic.com/images/artists/4529544', u'https://api.buzzanglemusic.com/images/artists/4304221', u'https://api.buzzanglemusic.com/images/artists/26', u'https://api.buzzanglemusic.com/images/artists/208', u'https://api.buzzanglemusic.com/images/artists/33', u'https://api.buzzanglemusic.com/images/artists/1079180', u'https://api.buzzanglemusic.com/images/artists/3305532', u'https://api.buzzanglemusic.com/images/artists/580367', u'https://api.buzzanglemusic.com/images/artists/54', u'https://api.buzzanglemusic.com/images/artists/4694249', u'https://api.buzzanglemusic.com/images/artists/1545', u'https://api.buzzanglemusic.com/images/artists/3391309', u'https://api.buzzanglemusic.com/images/artists/489706', u'https://api.buzzanglemusic.com/images/artists/185', u'https://api.buzzanglemusic.com/images/artists/812522', u'https://api.buzzanglemusic.com/images/artists/5681847', u'https://api.buzzanglemusic.com/images/artists/3195987', u'https://api.buzzanglemusic.com/images/artists/4179629', u'https://api.buzzanglemusic.com/images/artists/5212303', u'https://api.buzzanglemusic.com/images/artists/142', u'https://api.buzzanglemusic.com/images/artists/6384381', u'https://api.buzzanglemusic.com/images/artists/812157', u'https://api.buzzanglemusic.com/images/artists/192', u'https://api.buzzanglemusic.com/images/artists/692', u'https://api.buzzanglemusic.com/images/artists/1256797', u'https://api.buzzanglemusic.com/images/artists/5984690', u'https://api.buzzanglemusic.com/images/artists/5344432', u'https://api.buzzanglemusic.com/images/artists/2381', u'https://api.buzzanglemusic.com/images/artists/1338', u'https://api.buzzanglemusic.com/images/artists/28', u'https://api.buzzanglemusic.com/images/artists/6500275', u'https://api.buzzanglemusic.com/images/artists/4754086', u'https://api.buzzanglemusic.com/images/artists/189', u'https://api.buzzanglemusic.com/images/artists/715', u'https://api.buzzanglemusic.com/images/artists/11', u'https://api.buzzanglemusic.com/images/artists/690440', u'https://api.buzzanglemusic.com/images/artists/722962', u'https://api.buzzanglemusic.com/images/artists/6478745', u'https://api.buzzanglemusic.com/images/artists/1626', u'https://api.buzzanglemusic.com/images/artists/150', u'https://api.buzzanglemusic.com/images/artists/1356', u'https://api.buzzanglemusic.com/images/artists/221', u'https://api.buzzanglemusic.com/images/artists/6216355', u'https://api.buzzanglemusic.com/images/artists/5366988', u'https://api.buzzanglemusic.com/images/artists/771', u'https://api.buzzanglemusic.com/images/artists/610595', u'https://api.buzzanglemusic.com/images/artists/247467', u'https://api.buzzanglemusic.com/images/artists/5215781', u'https://api.buzzanglemusic.com/images/artists/7640', u'https://api.buzzanglemusic.com/images/artists/621107', u'https://api.buzzanglemusic.com/images/artists/4547105', u'https://api.buzzanglemusic.com/images/artists/143', u'https://api.buzzanglemusic.com/images/artists/4827213', u'https://api.buzzanglemusic.com/images/artists/769', u'https://api.buzzanglemusic.com/images/artists/1337', u'https://api.buzzanglemusic.com/images/artists/487', u'https://api.buzzanglemusic.com/images/artists/2643', u'https://api.buzzanglemusic.com/images/artists/8957464', u'https://api.buzzanglemusic.com/images/artists/807069', u'https://api.buzzanglemusic.com/images/artists/517460', u'https://api.buzzanglemusic.com/images/artists/662', u'https://api.buzzanglemusic.com/images/artists/6227591', u'https://api.buzzanglemusic.com/images/artists/330', u'https://api.buzzanglemusic.com/images/artists/2419669', u'https://api.buzzanglemusic.com/images/artists/7299117', u'https://api.buzzanglemusic.com/images/artists/630', u'https://api.buzzanglemusic.com/images/artists/38', u'https://api.buzzanglemusic.com/images/artists/743', u'https://api.buzzanglemusic.com/images/artists/295165', u'https://api.buzzanglemusic.com/images/artists/474', u'https://api.buzzanglemusic.com/images/artists/5085902', u'https://api.buzzanglemusic.com/images/artists/5433623', u'https://api.buzzanglemusic.com/images/artists/15510', u'https://api.buzzanglemusic.com/images/artists/1390', u'https://api.buzzanglemusic.com/images/artists/596038', u'https://api.buzzanglemusic.com/images/artists/465', u'https://api.buzzanglemusic.com/images/artists/503490', u'https://api.buzzanglemusic.com/images/artists/82', u'https://api.buzzanglemusic.com/images/artists/6427308', u'https://api.buzzanglemusic.com/images/artists/199', u'https://api.buzzanglemusic.com/images/artists/7144', u'https://api.buzzanglemusic.com/images/artists/44528', u'https://api.buzzanglemusic.com/images/artists/490', u'https://api.buzzanglemusic.com/images/artists/830802', u'https://api.buzzanglemusic.com/images/artists/178', u'https://api.buzzanglemusic.com/images/artists/20931', u'https://api.buzzanglemusic.com/images/artists/78', u'https://api.buzzanglemusic.com/images/artists/2599', u'https://api.buzzanglemusic.com/images/artists/31865', u'https://api.buzzanglemusic.com/images/artists/755031', u'https://api.buzzanglemusic.com/images/artists/5009248', u'https://api.buzzanglemusic.com/images/artists/7177348', u'https://api.buzzanglemusic.com/images/artists/9046744', u'https://api.buzzanglemusic.com/images/artists/472', u'https://api.buzzanglemusic.com/images/artists/6816051', u'https://api.buzzanglemusic.com/images/artists/603486', u'https://api.buzzanglemusic.com/images/artists/9518002', u'https://api.buzzanglemusic.com/images/artists/856682', u'https://api.buzzanglemusic.com/images/artists/3883529', u'https://api.buzzanglemusic.com/images/artists/173', u'https://api.buzzanglemusic.com/images/artists/236', u'https://api.buzzanglemusic.com/images/artists/225', u'https://api.buzzanglemusic.com/images/artists/4779775', u'https://api.buzzanglemusic.com/images/artists/723917', u'https://api.buzzanglemusic.com/images/artists/1823', u'https://api.buzzanglemusic.com/images/artists/881', u'https://api.buzzanglemusic.com/images/artists/7296610', u'https://api.buzzanglemusic.com/images/artists/3587186', u'https://api.buzzanglemusic.com/images/artists/678', u'https://api.buzzanglemusic.com/images/artists/866831', u'https://api.buzzanglemusic.com/images/artists/4231', u'https://api.buzzanglemusic.com/images/artists/218', u'https://api.buzzanglemusic.com/images/artists/22', u'https://api.buzzanglemusic.com/images/artists/1339', u'https://api.buzzanglemusic.com/images/artists/1352', u'https://api.buzzanglemusic.com/images/artists/181', u'https://api.buzzanglemusic.com/images/artists/806032', u'https://api.buzzanglemusic.com/images/artists/1364', u'https://api.buzzanglemusic.com/images/artists/2611', u'https://api.buzzanglemusic.com/images/artists/828', u'https://api.buzzanglemusic.com/images/artists/885', u'https://api.buzzanglemusic.com/images/artists/36569', u'https://api.buzzanglemusic.com/images/artists/4466817', u'https://api.buzzanglemusic.com/images/artists/13', u'https://api.buzzanglemusic.com/images/artists/227', u'https://api.buzzanglemusic.com/images/artists/2506', u'https://api.buzzanglemusic.com/images/artists/730102', u'https://api.buzzanglemusic.com/images/artists/9284021', u'https://api.buzzanglemusic.com/images/artists/9371339', u'https://api.buzzanglemusic.com/images/artists/164', u'https://api.buzzanglemusic.com/images/artists/1336', u'https://api.buzzanglemusic.com/images/artists/5722286', u'https://api.buzzanglemusic.com/images/artists/4351620', u'https://api.buzzanglemusic.com/images/artists/166', u'https://api.buzzanglemusic.com/images/artists/148', u'https://api.buzzanglemusic.com/images/artists/680', u'https://api.buzzanglemusic.com/images/artists/5880', u'https://api.buzzanglemusic.com/images/artists/335779', u'https://api.buzzanglemusic.com/images/artists/184', u'https://api.buzzanglemusic.com/images/artists/4375959', u'https://api.buzzanglemusic.com/images/artists/646', u'https://api.buzzanglemusic.com/images/artists/289', u'https://api.buzzanglemusic.com/images/artists/502', u'https://api.buzzanglemusic.com/images/artists/596334', u'https://api.buzzanglemusic.com/images/artists/4786404', u'https://api.buzzanglemusic.com/images/artists/224', u'https://api.buzzanglemusic.com/images/artists/379', u'https://api.buzzanglemusic.com/images/artists/29', u'https://api.buzzanglemusic.com/images/artists/9224269', u'https://api.buzzanglemusic.com/images/artists/6221', u'https://api.buzzanglemusic.com/images/artists/590503', u'https://api.buzzanglemusic.com/images/artists/804250', u'https://api.buzzanglemusic.com/images/artists/322', u'https://api.buzzanglemusic.com/images/artists/89', u'https://api.buzzanglemusic.com/images/artists/6944241']
[u'Drake', u'Lil Baby', u'DaBaby', u'Post Malone', u'Lil Uzi Vert', u'YoungBoy Never Broke Again', u'Future', u'The Weeknd', u'Juice Wrld', u'Gunna', u'Luke Combs', u'Polo G', u'Eminem', u'Taylor Swift', u'Rod Wave', u'Travis Scott', u'Xxxtentacion', u'Bad Bunny', u'Billie Eilish', u'Roddy Ricch', u'Pop Smoke', u'Kanye West', u'The Beatles', u'Harry Styles', u'A Boogie wit da Hoodie', u'Megan Thee Stallion', u'Morgan Wallen', u'Trippie Redd', u'Lil Wayne', u'J. Cole', u'Lady Gaga', u'Chris Brown', u'Kevin Gates', u'Kendrick Lamar', u'Ariana Grande', u'Khalid', u'Ed Sheeran', u'Mac Miller', u'Jason Aldean', u'Luke Bryan', u'Jhen\xe9 Aiko', u'NLE Choppa', u'Justin Bieber', u'Lil Durk', u'Iann Dior', u'Moneybagg Yo', u'Kodak Black', u'Florida Georgia Line', u'Beyonc\xe9', u'Kenny Chesney', u'Tory Lanez', u'Doja Cat', u'Young Thug', u'Thomas Rhett', u'Dua Lipa', u'Meek Mill', u'Kehlani', u'Sam Hunt', u'Rihanna', u'BTS', u'Lil Peep', u'NF', u'Kane Brown', u'Chloe x Halle', u'Maroon 5', u'YNW Melly', u'Migos', u'Imagine Dragons', u'Queen', u'Halsey', u'Lewis Capaldi', u'$uicideboy$', u'twenty one pilots', u'Frank Ocean', u'Blake Shelton', u'Lil Mosey', u'G Herbo', u'One Direction', u'Fleetwood Mac', u'Zac Brown Band', u'Maren Morris', u'Nav', u'Don Toliver', u'Panic! at the Disco', u'Bruno Mars', u'2Pac', u'Nicki Minaj', u'6IX9INE', u'SAINt JHN', u'Eagles', u'George Strait', u'Chris Stapleton', u'Russ', u'Tyler, The Creator', u'Eric Church', u'Anuel AA', u'Katy Perry', u'Jack Harlow', u'Elton John', u'Wiz Khalifa', u'AC/DC', u'Jonas Brothers', u'Lil Baby, 42 Dugg', u'Blackbear', u'H.E.R', u'Michael Jackson', u'Lil Skies', u'Red Hot Chili Peppers', u'Bazzi', u'Lil Tjay', u'The Rolling Stones', u'Tim McGraw', u'Tom Petty And The Heartbreakers', u'Summer Walker', u'Led Zeppelin', u'Playboi Carti', u'Surfaces', u'Childish Gambino', u'Gucci Mane', u'Melanie Martinez', u'Five Finger Death Punch', u'Old Dominion', u'Miranda Lambert', u'JayDaYoungan', u'Linkin Park', u'Nipsey Hussle', u'Coldplay', u'Metallica', u'Dan + Shay', u'John Mayer', u'Sam Smith', u'Jon Pardi', u'Kid Cudi', u'Logic', u'5 Seconds of Summer', u'Lil Yachty', u'Natanael Cano', u'StaySolidRocky', u'Pink Floyd', u'Lil Tecca', u'Brett Young', u'6Ix9Ine & Nicki Minaj', u'Tyler Childers', u'Ozuna', u'Adele', u'Jack Johnson', u'Usher', u'21 Savage', u'Lizzo', u'Machine Gun Kelly', u'Bob Marley & The Wailers', u'Sleepy hallow', u'Shawn Mendes', u'Billy Joel', u'Run The Jewels', u'YG', u'Green Day', u'Carrie Underwood', u'Trey Songz', u'A$AP Rocky', u'JAY-Z', u'G-Eazy', u'Notorious B.I.G.', u'Selena Gomez', u'Fall Out Boy', u'Creedence Clearwater Revival', u'Camila Cabello', u'Joji', u'Keith Urban', u'Demi Lovato', u'Frank Sinatra', u'Chance the Rapper', u'Chris Brown & Young Thug', u'Lady Gaga, Ariana Grande', u'Lana Del Rey', u'DJ Khaled', u'Key Glock', u'Brent Faiyaz', u'The Lumineers', u'Pink', u'Journey', u'Tame Impala', u'Jordan Davis', u'OneRepublic', u'YFN Lucci', u'Lynyrd Skynyrd', u'Sublime', u'Nirvana', u'The Chainsmokers', u'Cardi B', u'Calvin Harris', u'Johnny Cash', u'Lee Brice', u'Travis Scott, The Scotts, Kid Cudi', u'Alan Jackson', u'J. Balvin', u'PARTYNEXTDOOR', u'50 Cent', u'Garth Brooks', u'Powfu']

谢谢你知道为什么我的for循环只包括前50名艺术家吗？在调试器中，“theu rest”是一个包含205个元素的数组，for循环遍历了所有元素，但它只包含前50名美工。另外，您如何解决由于chrome崩溃而导致代码无法工作的错误？