Python 使用Beautifulsoup从网站中提取数据
我试图提取模型名称和有关模型的任何其他细节。当我试图获取文本时,我找不到任何可以用于获取数据的特殊内容 有人知道如何从这些URL获取数据吗 或 我想得到的模型和任何财产的清单,这是像价格 到目前为止,我尝试使用以下方法:Python 使用Beautifulsoup从网站中提取数据,python,pandas,selenium,beautifulsoup,Python,Pandas,Selenium,Beautifulsoup,我试图提取模型名称和有关模型的任何其他细节。当我试图获取文本时,我找不到任何可以用于获取数据的特殊内容 有人知道如何从这些URL获取数据吗 或 我想得到的模型和任何财产的清单,这是像价格 到目前为止,我尝试使用以下方法: from selenium import webdriver from bs4 import BeautifulSoup import pandas as pd from urllib.request import urlopen u
from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd
from urllib.request import urlopen
url = "https://www.opel.de/auswahlhilfe/modelle.html"
html = urlopen(url)
text = soup.get_text()
但是我没有得到任何有用的东西。。这里有专家吗?我不确定您是否承诺使用BeautifulSoup,但您可以用一些简单的旧硒解决这个问题 使用您提供的URL,这些示例可以帮助您开始
from selenium import webdriver
from time import sleep
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
# start the driver
driver = webdriver.Chrome()
# navigate to the URL
driver.get("https://www.opel.de/auswahlhilfe/modelle.html")
# invoke WebDriverWait to wait for page to load
# get the model names
model_names = WebDriverWait(driver, 10).until(
EC.presence_of_all_elements_located((By.XPATH, "//a/div/div/div/span[contains(@class, 'q-carline')]")))
for model in model_names:
print(model.text)
要获取价格列表,请执行以下操作:
prices = driver.find_elements_by_xpath("//a/div/div/div/span[contains(@class, 'q-price')]/span[contains(@class, 'q-value')]");
for price in prices:
print(price.text)
使用
请求
而不是urllib.request
import requests
from bs4 import BeautifulSoup
url = "https://www.opel.de/auswahlhilfe/modelle.html"
r = requests.get(url)
soup = BeautifulSoup(r.text, 'html.parser')
text = soup.get_text()
print(text)
如果您转到网络选项卡,您将获得以下链接,该链接以json格式返回值。 你不需要硒来做到这一点
https://www.opel.de/apps/atomic/getVehicleTeasers.path=L2NvbnRlbnQvb3BlbC93b3JsZHdpZGUvZ2VybWFueS9kZS9pbmRleC9iYXNlYmFsbC1jYXJkcy9iYmMtY29sbGVjdGlvbnMvdmVoaWNsZXMtb25seS1jb2xsZWN0aW9uPuGlIfE.feefoEnabled=false.expandingMenuEnabled=false.json
试试下面的代码
from bs4 import BeautifulSoup
import requests
import pandas as pd
url='https://www.opel.de/apps/atomic/getVehicleTeasers.path=L2NvbnRlbnQvb3BlbC93b3JsZHdpZGUvZ2VybWFueS9kZS9pbmRleC9iYXNlYmFsbC1jYXJkcy9iYmMtY29sbGVjdGlvbnMvdmVoaWNsZXMtb25seS1jb2xsZWN0aW9uPuGlIfE.feefoEnabled=false.expandingMenuEnabled=false.json'
rs=requests.get(url).json()
html=''.join(rs['bbcTeaser'])
soup=BeautifulSoup(html,'html.parser')
car_name=[]
car_price=[]
for name, price in zip(soup.select('.q-carline'),soup.select('.q-value')):
car_name.append(name.text)
car_price.append(price.text)
df = pd.DataFrame({"car_name":car_name,"car_price":car_price})
print(df)
输出:
car_name car_price
0 ADAM € 14.120,00 nur Lagerfahrzeuge
1 ADAM ROCKS € 16.475,00 nur Lagerfahrzeuge
2 ADAM ROCKS S € 20.430,00
3 ADAM S € 19.330,00
4 Ampera-e € 42.990,00
5 Astra 5-Türer € 19.990,00
6 Astra Sports Tourer € 20.990,00
7 Cascada € 33.995,00 nur Lagerfahrzeuge
8 Combo Life € 21.645,00
9 Neuer Corsa € 13.990,00
10 Corsa-e € 29.900,00
11 Corsa 3-Türer € 13.255,00
12 Corsa 5-Türer € 14.055,00
13 Crossland X € 18.750,00
14 Grandland X € 24.700,00
15 Insignia Grand Sport € 28.505,00
16 Insignia GSi € 46.695,00
17 Insignia Sports Tourer € 29.505,00
18 Insignia Country Tourer € 41.385,00
19 KARL € 13.350,00
20 KARL ROCKS € 12.965,00
21 Mokka X € 20.495,00
22 Zafira € 28.495,00
23 Zafira Life € 34.780,00
24 Combo Cargo € 20.230,00
25 Movano Cargo € 27.925,00
26 Movano Doppelkabine € 38.288,25
27 Movano Fahrgestell Normalkabine € 34.777,75
28 Movano Fahrgestell Doppelkabine € 35.967,75
29 Movano Plattformfahrgestell € 34.777,75
30 Movano Kofferaufbau € 46.320,75
31 Movano Pritsche Normalkabine € 37.574,25
32 Movano Pritsche Doppelkabine € 38.764,25
33 Movano Kofferaufbau € 46.320,75
34 Movano Pritsche Normalkabine € 37.574,25
35 Movano Pritsche Doppelkabine € 38.764,25
36 Movano Kipper Normalkabine € 39.894,75
37 Movano Kipper Doppelkabine € 44.178,75
38 Vivaro Cargo € 29.750,00
39 Vivaro Doppelkabine € 33.082,00
40 Vivaro Kombi € 31.237,50
41 Grandland X Hybrid4 € 51.165,00
42 Movano Kombi € 30.905,00
快照:
car_name car_price
0 ADAM € 14.120,00 nur Lagerfahrzeuge
1 ADAM ROCKS € 16.475,00 nur Lagerfahrzeuge
2 ADAM ROCKS S € 20.430,00
3 ADAM S € 19.330,00
4 Ampera-e € 42.990,00
5 Astra 5-Türer € 19.990,00
6 Astra Sports Tourer € 20.990,00
7 Cascada € 33.995,00 nur Lagerfahrzeuge
8 Combo Life € 21.645,00
9 Neuer Corsa € 13.990,00
10 Corsa-e € 29.900,00
11 Corsa 3-Türer € 13.255,00
12 Corsa 5-Türer € 14.055,00
13 Crossland X € 18.750,00
14 Grandland X € 24.700,00
15 Insignia Grand Sport € 28.505,00
16 Insignia GSi € 46.695,00
17 Insignia Sports Tourer € 29.505,00
18 Insignia Country Tourer € 41.385,00
19 KARL € 13.350,00
20 KARL ROCKS € 12.965,00
21 Mokka X € 20.495,00
22 Zafira € 28.495,00
23 Zafira Life € 34.780,00
24 Combo Cargo € 20.230,00
25 Movano Cargo € 27.925,00
26 Movano Doppelkabine € 38.288,25
27 Movano Fahrgestell Normalkabine € 34.777,75
28 Movano Fahrgestell Doppelkabine € 35.967,75
29 Movano Plattformfahrgestell € 34.777,75
30 Movano Kofferaufbau € 46.320,75
31 Movano Pritsche Normalkabine € 37.574,25
32 Movano Pritsche Doppelkabine € 38.764,25
33 Movano Kofferaufbau € 46.320,75
34 Movano Pritsche Normalkabine € 37.574,25
35 Movano Pritsche Doppelkabine € 38.764,25
36 Movano Kipper Normalkabine € 39.894,75
37 Movano Kipper Doppelkabine € 44.178,75
38 Vivaro Cargo € 29.750,00
39 Vivaro Doppelkabine € 33.082,00
40 Vivaro Kombi € 31.237,50
41 Grandland X Hybrid4 € 51.165,00
42 Movano Kombi € 30.905,00
您可以从json响应中获取一些数据。只需仿冒出你能从那里得到的东西:
import requests
from bs4 import BeautifulSoup
url = 'https://www.opel.de/apps/atomic/getVehicleTeasers.path=L2NvbnRlbnQvb3BlbC93b3JsZHdpZGUvZ2VybWFueS9kZS9pbmRleC9iYXNlYmFsbC1jYXJkcy9iYmMtY29sbGVjdGlvbnMvdmVoaWNsZXMtb25seS1jb2xsZWN0aW9uPuGlIfE.feefoEnabled=false.expandingMenuEnabled=false.json'
jsonData = requests.get(url).json()['bbcTeaser']
for each in jsonData:
soup = BeautifulSoup(each)
carline = soup.find('span', {'class':'q-carline'}).text
price = soup.find('span', {'class':'q-value'}).text
print (carline, price)
输出:
ADAM € 14.120,00 nur Lagerfahrzeuge
ADAM ROCKS € 16.475,00 nur Lagerfahrzeuge
ADAM ROCKS S € 20.430,00
ADAM S € 19.330,00
Ampera-e € 42.990,00
Astra 5-Türer € 19.990,00
Astra Sports Tourer € 20.990,00
Cascada € 33.995,00 nur Lagerfahrzeuge
Combo Life € 21.645,00
Neuer Corsa € 13.990,00
Corsa-e € 29.900,00
Corsa 3-Türer € 13.255,00
Corsa 5-Türer € 14.055,00
Crossland X € 18.750,00
Grandland X € 24.700,00
Insignia Grand Sport € 28.505,00
Insignia GSi € 46.695,00
Insignia Sports Tourer € 29.505,00
Insignia Country Tourer € 41.385,00
KARL € 13.350,00
KARL ROCKS € 12.965,00
Mokka X € 20.495,00
Zafira € 28.495,00
Zafira Life € 34.780,00
Combo Cargo € 20.230,00
Movano Cargo € 27.925,00
Movano Doppelkabine € 38.288,25
Movano Fahrgestell Normalkabine € 34.777,75
Movano Fahrgestell Doppelkabine € 35.967,75
Movano Plattformfahrgestell € 34.777,75
Movano Kofferaufbau € 46.320,75
Movano Pritsche Normalkabine € 37.574,25
Movano Pritsche Doppelkabine € 38.764,25
Movano Kofferaufbau € 46.320,75
Movano Pritsche Normalkabine € 37.574,25
Movano Pritsche Doppelkabine € 38.764,25
Movano Kipper Normalkabine € 39.894,75
Movano Kipper Doppelkabine € 44.178,75
Vivaro Cargo € 29.750,00
Vivaro Doppelkabine € 33.082,00
Vivaro Kombi € 31.237,50
Grandland X Hybrid4 € 51.165,00
Movano Kombi € 30.905,00
谢谢你的回答。是否因为使用了“驱动程序”而使用了其他库?
未定义名称“驱动程序”
只是出现错误。驱动程序是Selenium Webdriver。你必须首先启动一个驱动程序。更新了我的代码示例。谢谢。但仍然出现异常错误:WebDriverException:消息:“chromedriver”可执行文件需要位于路径中。请看https://sites.google.com/a/chromium.org/chromedriver/home
您需要完全按照错误消息的指示执行操作——将chromedriver.exe添加到Path环境变量中。有大量的指南和教程向您展示如何做到这一点。这里有一个:谢谢@Moshe,但我如何生成列表?我正在获取文本及其干净的文本,但不确定如何从中创建模型列表…@MuhammadSalmanShahid请参阅KunduK的答案。。。他完成了工作!太神了这就是我想要的:)谢谢@KunduK@KunduK我试图在frm网络中找到这个json url,但找不到那个特定的url。你能再解释一下吗?一旦你进入网络标签,点击XHR标签,然后点击ctlr+R,你就会看到一个链接,当你点击response标签时,它会以json的形式返回数据。