如何使用Selenium Python从网站中提取产品标题_Selenium_Xpath_Web Scraping_Css Selectors_Webdriverwait

如何使用Selenium Python从网站中提取产品标题

selenium xpath web-scraping

如何使用Selenium Python从网站中提取产品标题,selenium,xpath,web-scraping,css-selectors,webdriverwait,Selenium,Xpath,Web Scraping,Css Selectors,Webdriverwait,我试图从一个网站上刮取标题，但它只返回1个标题。我怎样才能得到所有的头衔下面是我尝试使用xpath获取的元素之一（以开头）：输出： UNDERCOVER x Nike SFB Mountain Pack有发布日期我期待更多的标题，但只返回一个结果。如果定位器找到多个元素，则find\u elemnt返回第一个元素查找元素返回定位器找到的所有元素的列表。然后可以迭代列表并获取所有元素如果您试图查找的所有元素都有类邮箱，则可以按类名查找元素。如果定位器找到多个元素，则查找元素返回第一个元素

我试图从一个网站上刮取标题，但它只返回1个标题。我怎样才能得到所有的头衔

下面是我尝试使用xpath获取的元素之一（以开头）：

输出：

UNDERCOVER x Nike SFB Mountain Pack有发布日期

我期待更多的标题，但只返回一个结果。

如果定位器找到多个元素，则

find\u elemnt

返回第一个元素<代码>查找元素返回定位器找到的所有元素的列表。
然后可以迭代列表并获取所有元素

如果您试图查找的所有元素都有类

邮箱

，则可以按类名查找元素。

如果定位器找到多个元素，则

查找元素

返回第一个元素<代码>查找元素返回定位器找到的所有元素的列表。
然后可以迭代列表并获取所有元素

如果您试图查找的所有元素都有类

邮箱

，则您可以按类名查找元素。

要从中提取产品标题，因为所需元素是已启用的元素，您需要诱导WebDriverWait以查看所有元素（）您可以使用以下任一选项：

XPATH

：

driver.get('https://hypebeast.com/search?s=nike+undercover')
print([element.text for element in WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//h2/span")))])

CSS\u选择器

：

driver.get('https://hypebeast.com/search?s=nike+undercover')
print([element.text for element in WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "h2>span")))])

控制台输出：

['The UNDERCOVER x Nike SFB Mountain Pack Gets a Release Date', 'The UNDERCOVER x Nike SFB Mountain Surfaces in "Dark Obsidian/University Red"', 'A First Look at UNDERCOVER’s Nike SFB Mountain Collaboration', "Here's Where to Buy the UNDERCOVER x Gyakusou Nike Running Models", 'Take Another Look at the Upcoming UNDERCOVER x Nike Daybreak', "Take an Official Look at GYAKUSOU's SS19 Footwear and Apparel Range", 'UNDERCOVER x Nike Daybreak Expected to Hit Shelves This Summer', "The 10 Best Sneakers From Paris Fashion Week's FW19 Runways", "UNDERCOVER FW19 Debuts 'A Clockwork Orange' Theme, Nike & Valentino Collabs", 'These Are the Best Sneakers of 2018']

要从中提取产品标题，因为所需的元素是启用的元素，您需要诱导WebDriverWait，以实现所有元素的可见性（），并且您可以使用以下任一选项：

XPATH

：

driver.get('https://hypebeast.com/search?s=nike+undercover')
print([element.text for element in WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//h2/span")))])

CSS\u选择器

：

driver.get('https://hypebeast.com/search?s=nike+undercover')
print([element.text for element in WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "h2>span")))])

控制台输出：

['The UNDERCOVER x Nike SFB Mountain Pack Gets a Release Date', 'The UNDERCOVER x Nike SFB Mountain Surfaces in "Dark Obsidian/University Red"', 'A First Look at UNDERCOVER’s Nike SFB Mountain Collaboration', "Here's Where to Buy the UNDERCOVER x Gyakusou Nike Running Models", 'Take Another Look at the Upcoming UNDERCOVER x Nike Daybreak', "Take an Official Look at GYAKUSOU's SS19 Footwear and Apparel Range", 'UNDERCOVER x Nike Daybreak Expected to Hit Shelves This Summer', "The 10 Best Sneakers From Paris Fashion Week's FW19 Runways", "UNDERCOVER FW19 Debuts 'A Clockwork Orange' Theme, Nike & Valentino Collabs", 'These Are the Best Sneakers of 2018']

请求

数据标题

import requests
from bs4 import BeautifulSoup as bs

r = requests.get('https://hypebeast.com/search?s=nike+undercover')
soup = bs(r.content, 'lxml')
titles = [item['data-title'] for item in soup.select('[data-title]')]
print(titles)

from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://hypebeast.com/search?s=nike+undercover')
titles = [item.get_attribute('data-title') for item in driver.find_elements_by_css_selector('[data-title]')]
print(titles)

请求

数据标题

import requests
from bs4 import BeautifulSoup as bs

r = requests.get('https://hypebeast.com/search?s=nike+undercover')
soup = bs(r.content, 'lxml')
titles = [item['data-title'] for item in soup.select('[data-title]')]
print(titles)

from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://hypebeast.com/search?s=nike+undercover')
titles = [item.get_attribute('data-title') for item in driver.find_elements_by_css_selector('[data-title]')]
print(titles)

element.get_attribute('ATTRIBUTE-NAME')

element.get_attribute('ATTRIBUTE-NAME')

查找元素

查找元素

查找元素

查找元素

find_elements

未找到任何属性

elements

find_元素

find_elements

未找到任何属性

elements

find_元素