如何使用Python刮取网站数据？
我正在尝试从该网页中获取实时商品价值，它有一个iframe地址： 这是我用BeautifulSoup尝试的：如何使用Python刮取网站数据？,python,selenium,web-scraping,beautifulsoup,Python,Selenium,Web Scraping,Beautifulsoup,我正在尝试从该网页中获取实时商品价值，它有一个iframe地址： 这是我用BeautifulSoup尝试的： from bs4 import BeautifulSoup #import time import urllib data = [] url=urllib.urlopen("http://213.136.84.136:8000/") html=url.read() url.close() soup = BeautifulSoup(html,"html.parser") span=sou
from bs4 import BeautifulSoup
#import time
import urllib
data = []
url=urllib.urlopen("http://213.136.84.136:8000/")
html=url.read()
url.close()
soup = BeautifulSoup(html,"html.parser")
span=soup.find('table', attrs={'class':'table2'})
table_body = span.find('tbody')
rows = table_body.find_all('tr')
for row in rows:
cols = row.find_all('td')
cols = [ele.text.strip() for ele in cols]
data.append([ele for ele in cols if ele]) # Get rid of empty values
print([data])
[[[], [u'INTERNATIONAL MARKET'], [u'SPOT Gold'], [u'SPOT Silver'], [u'CrudeOil'], [u'Copper'], [u'NaturalGas'], [u'Dow Jones'], [u'Bank Nifty'], [u'INDIAN MARKET'], [u'MCXGold'], [u'MCXSilver'], [u'MCXCrudeOil'], [u'MCXCopper'], [u'MCXLead'], [u'MCXNickel'], [u'MCXZinc'], [u'MCXNaturalGas'], [u'MCXAluminium'], [u'MCXMenthaOil'], [u'USDINR'], [], [u"Disclaimer: We can't assure any guarantee about the accuaracy of the data."]]]
<table class="table2" border="1">
<tbody>
<tr style="background-color: #DFD9D9; font-size: 16px; ">
<th style="border:4px solid white; text-align:left; ">SYMBOL</th>
<th style="border:4px solid white; text-align:right;">LTP</th>
<th style="border:4px solid white; text-align:right;">HIGH</th>
<th style="border:4px solid white; text-align:right;">LOW</th>
</tr>
<tr style="font-size: 13px; color: black;">
<td colspan="4" style="color:black; text-align:center; font-size:initial; background-color:lightyellow;">INTERNATIONAL MARKET</td>
</tr>
<tr style="font-size: 15px; color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">SPOT Gold</td>
<td id="goldsell" style="text-align:right;"></td>
<td id="goldhigh" style="text-align:right;"></td>
<td id="goldlow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px; color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">SPOT Silver</td>
<td id="Silversell" style="text-align:right;"></td>
<td id="Silverhigh" style="text-align:right;"></td>
<td id="Silverlow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px; color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">CrudeOil</td>
<td id="CrudeOilsell" style="text-align:right;"></td>
<td id="CrudeOilhigh" style="text-align:right;"></td>
<td id="CrudeOillow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px; color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">Copper</td>
<td id="Coppersell" style="text-align:right;"></td>
<td id="Copperhigh" style="text-align:right;"></td>
<td id="Copperlow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px; color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">NaturalGas</td>
<td id="NaturalGassell" style="text-align:right;"></td>
<td id="NaturalGashigh" style="text-align:right;"></td>
<td id="NaturalGaslow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px; color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">Dow Jones</td>
<td id="DJsell" style="text-align:right;"></td>
<td id="DJhigh" style="text-align:right;"></td>
<td id="DJlow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px; color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">Bank Nifty</td>
<td id="BNsell" style="text-align:right;"></td>
<td id="BNhigh" style="text-align:right;"></td>
<td id="BNlow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 13px; color: black;">
<td colspan="4" style="color:black; text-align:center; font-size:initial; background-color:lightyellow">INDIAN MARKET</td>
</tr>
<tr style="font-size: 15px; color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">MCXGold</td>
<td id="MCXGoldsell" style="text-align:right;"></td>
<td id="MCXGoldhigh" style="text-align:right;"></td>
<td id="MCXGoldlow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px; color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">MCXSilver</td>
<td id="MCXSilversell" style="text-align:right;"></td>
<td id="MCXSilverhigh" style="text-align:right;"></td>
<td id="MCXSilverlow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px;color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">MCXCrudeOil</td>
<td id="MCXCrudeOilsell" style="text-align:right;"></td>
<td id="MCXCrudeOilhigh" style="text-align:right;"></td>
<td id="MCXCrudeOillow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px;color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">MCXCopper</td>
<td id="MCXCoppersell" style="text-align:right;"></td>
<td id="MCXCopperhigh" style="text-align:right;"></td>
<td id="MCXCopperlow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px;color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">MCXLead</td>
<td id="MCXLeadsell" style="text-align:right;"></td>
<td id="MCXLeadhigh" style="text-align:right;"></td>
<td id="MCXLeadlow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px;color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">MCXNickel</td>
<td id="MCXNickelsell" style="text-align:right;"></td>
<td id="MCXNickelhigh" style="text-align:right;"></td>
<td id="MCXNickellow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px;color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">MCXZinc</td>
<td id="MCXZincsell" style="text-align:right;"></td>
<td id="MCXZinchigh" style="text-align:right;"></td>
<td id="MCXZinclow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px;color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">MCXNaturalGas</td>
<td id="MCXNaturalGassell" style="text-align:right;"></td>
<td id="MCXNaturalGashigh" style="text-align:right;"></td>
<td id="MCXNaturalGaslow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px;color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">MCXAluminium</td>
<td id="MCXAluminiumsell" style="text-align:right;"></td>
<td id="MCXAluminiumhigh" style="text-align:right;"></td>
<td id="MCXAluminiumlow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px;color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">MCXMenthaOil</td>
<td id="MCXMenthaOilsell" style="text-align:right;"></td>
<td id="MCXMenthaOilhigh" style="text-align:right;"></td>
<td id="MCXMenthaOillow" style="text-align:right;"></td>
</tr>
<tr style="font-size: 15px;color: black;">
<td style="color:#3399ff; background-color:#E0F5D6;">USDINR</td>
<td id="USDINRsell" style="text-align:right;"></td>
<td id="USDINRhigh" style="text-align:right;"></td>
<td id="USDINRlow" style="text-align:right;"></td>
</tr>
<tr>
<td id="lastupdate" colspan="4" style="color:grey; text-align:center; background-color:lavender;">
</td>
</tr>
<tr>
<td id="disclaimer" colspan="4" style="color:red; text-align:center; background-color:lavender;">
Disclaimer: We can't assure any guarantee about the accuaracy of the data.
</td>
</tr>
</tbody>
</table>
象征
LTP
高
低
国际市场
现货金
现货银
原油
铜
自然藻类
道琼斯公司
银行俏皮
印度市场
MCXGold
麦克斯西尔弗
MCXCrudeOil
麦克斯考珀
麦克斯莱德
MCX镍
MCXZIN
MCXNaturalGas
MCXAluminium
麦克斯门塔乌尔
美元卢比
免责声明：我们不能保证数据的准确性。
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# De-comment next two lines and add chrome_options=chrome_options to ChromeOptions() run Chrome as headless
# chrome_options = webdriver.ChromeOptions()
# chrome_options.add_argument('--headless')
driver = webdriver.Chrome()
url = "http://www.mcxliverates.in/"
driver.get(url)
xpath = '/html/body/div[1]/div[4]/div[1]/iframe' # this is the where the table that holds our data is located
elem = driver.find_element_by_xpath(xpath)
driver.switch_to.frame(elem)
wait = WebDriverWait(driver, 10)
xpath = '/html/body/div/div/div/table/tbody'
wait.until(EC.presence_of_element_located((By.XPATH, xpath))) # waits until the data is loaded
html = driver.page_source
driver.close()
soup = BeautifulSoup(html, "html.parser")
table = soup.find('table', class_='table2') # BeautifulSoup's built-in way of specifying class
table_body = table.find('tbody')
data = []
rows = table_body.find_all('tr')
for row in rows:
cols = row.find_all('td')
cols = [ele.text.strip() for ele in cols]
data.append([ele for ele in cols if ele]) # Get rid of empty values
print([data])
[[[], ['INTERNATIONAL MARKET'], ['SPOT Gold', '1330.09', '1335.92', '1320.63'], ['SPOT Silver', '16.585', '16.734', '16.338'], ['CrudeOil', '62.82', '63.02', '60.71'], ['Copper', '3.2300', '3.2304', '3.1430'], ['NaturalGas', '2.660', '2.684', '2.638'], ['Dow Jones', '25034', '25250', '24572'], ['Bank Nifty', '24955.0', '24957.0', '24763.0'], ['INDIAN MARKET'], ['MCXGold', '30564', '30575', '30400'], ['MCXSilver', '38713', '38755', '38250'], ['MCXCrudeOil', '4092', '4104', '3974'], ['MCXCopper', '464.85', '464.85', '453.30'], ['MCXLead', '166.00', '166.30', '161.65'], ['MCXNickel', '895.00', '896.30', '871.10'], ['MCXZinc', '231.60', '231.60', '226.00'], ['MCXNaturalGas', '172.00', '173.10', '170.90'], ['MCXAluminium', '142.10', '142.35', '140.65'], ['MCXMenthaOil', '1252.00', '1299.80', '1246.00'], ['USDINR', '64.9360', '65.1040', '64.8780'], ['Last Updated on : 11:45:57 22-Feb-2018'], ["Disclaimer: We can't assure any guarantee about the accuaracy of the data."]]]
欢迎来到精彩世界您提供的URL访问受限，因此如果您正在寻求帮助，最好在问题中包含HTML片段。感谢您提供HTML片段。看起来这只是一个模板，实际的报价数据是用JavaScript添加到DOM中的。这就是数据提供程序提供更新的方式。看到了吗<代码>在
td