Python+selenium在单击最后一个next按钮时抛出错误
我已经用python和selenium编写了一些代码来解析站点的名称。该网站有“下一步”按钮,可以进入“下一页”。我已经试着管理它来完美地运行我的脚本。然而,目前我面临两个问题: 在执行时,scraper进入下一页并从那里进行解析,因为我无法修复逻辑,所以没有截取起始页。 当它找不到最后一个灰显的“下一步”按钮时,会抛出一个错误来破坏代码。 以下是我到目前为止所做的尝试:Python+selenium在单击最后一个next按钮时抛出错误,python,python-3.x,selenium,selenium-webdriver,web-scraping,Python,Python 3.x,Selenium,Selenium Webdriver,Web Scraping,我已经用python和selenium编写了一些代码来解析站点的名称。该网站有“下一步”按钮,可以进入“下一页”。我已经试着管理它来完美地运行我的脚本。然而,目前我面临两个问题: 在执行时,scraper进入下一页并从那里进行解析,因为我无法修复逻辑,所以没有截取起始页。 当它找不到最后一个灰显的“下一步”按钮时,会抛出一个错误来破坏代码。 以下是我到目前为止所做的尝试: from selenium import webdriver from selenium.webdriver.common.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get("https://www.yellowpages.com/search?search_terms=pizza&geo_location_terms=San%20Francisco%2C%20CA&page=10")
while True:
wait.until(EC.visibility_of_element_located((By.XPATH, '//li/a[contains(@class,"next")]')))
item = driver.find_element_by_xpath('//li/a[contains(@class,"next")]')
if not driver.find_element_by_xpath('//li/a[contains(@class,"next")]'):
break
item.click()
wait.until(EC.visibility_of_element_located((By.XPATH, '//div[@class="info"]')))
for items in driver.find_elements_by_xpath('//div[@class="info"]'):
name = items.find_element_by_xpath('.//span[@itemprop="name"]').text
print(name)
driver.quit()
以下是灰显“下一步”按钮的元素:
<div class="pagination"><p><span>Showing</span>361-388
of 388<span>results</span></p><ul><li><a href="/search?search_terms=pizza&geo_location_terms=San%20Francisco%2C%20CA&page=12" data-page="12" data-analytics="{"click_id":132}" data-remote="true" class="prev ajax-page" data-impressed="1">Previous</a></li><li><a href="/search?search_terms=pizza&geo_location_terms=San%20Francisco%2C%20CA&page=9" data-page="9" data-analytics="{"click_id":132,"module":1,"listing_page":9}" data-remote="true" data-impressed="1">9</a></li><li><a href="/search?search_terms=pizza&geo_location_terms=San%20Francisco%2C%20CA&page=10" data-page="10" data-analytics="{"click_id":132,"module":1,"listing_page":10}" data-remote="true" data-impressed="1">10</a></li><li><a href="/search?search_terms=pizza&geo_location_terms=San%20Francisco%2C%20CA&page=11" data-page="11" data-analytics="{"click_id":132,"module":1,"listing_page":11}" data-remote="true" data-impressed="1">11</a></li><li><a href="/search?search_terms=pizza&geo_location_terms=San%20Francisco%2C%20CA&page=12" data-page="12" data-analytics="{"click_id":132,"module":1,"listing_page":12}" data-remote="true" data-impressed="1">12</a></li><li><span class="disabled">13</span></li></ul></div>
显然,您应该尝试切换刮削页面并单击“下一步”按钮。您也可以使用try/except来避免制动代码:
while True:
# Scraping required elements first
items = wait.until(EC.visibility_of_all_elements_located((By.XPATH, '//div[@class="info"]')))
for item in items:
name = item.find_element_by_xpath('.//span[@itemprop="name"]').text
print(name)
# ...and then try to click 'Next' button
try:
driver.find_element_by_xpath('//li/a[contains(@class,"next")]').click()
except:
break
感谢安德森爵士提供了宝贵的解决方案。它总是摇摆不定。顺便说一句,将所有刮板元件更换为刮板平稳运行的元件