用python刮表_Python_Web Scraping_Beautifulsoup_Html Table

用python刮表

python web-scraping

用python刮表,python,web-scraping,beautifulsoup,html-table,Python,Web Scraping,Beautifulsoup,Html Table,我正在尝试用python将表刮取并转换为data.tables，但我在美国几乎没有选举数据。这是我想要刮取的数据的html <tr class="type-republican"> <th class="results-name" scope="row"><a href="xxxxx"><span class="name-combo"><span class="token token-party"><abbr title="

我正在尝试用python将表刮取并转换为data.tables，但我在美国几乎没有选举数据。这是我想要刮取的数据的html

<tr class="type-republican">
<th class="results-name" scope="row"><a href="xxxxx"><span class="name-combo"><span    class="token token-party"><abbr title="Republican">R</abbr></span> <span    class="token token-winner"><b aria-hidden="true" class="icon icon-check"></b>   <span class="icon-text">Winner</span></span> D. Trump</span></a></th>
<td class="results-percentage"><span class="percentage-combo"><span  class="number">62.9%</span><span class="graph"><span class="bar"><span class="index" style="width:62.9%;"></span></span></span></span></td>
<td class="results-popular">1,306,925</td>
<td class="delegates-cell">9</td>
</tr>
<tr class="type-democrat">
<th class="results-name" scope="row"><a href="xxxxxx"><span class="name-combo"><span   class="token token-party"><abbr title="Democratic">D</abbr></span> H.   Clinton</span></a></th>
<td class="results-percentage"><span class="percentage-combo"><span class="number">34.6%</span><span class="graph"><span class="bar"><span class="index" style="width:34.6%;"></span></span></span></span></td>
<td class="results-popular">718,084</td>
<td class="delegates-cell"></td>
</tr>
<tr class="type-independent">
<th class="results-name" scope="row"><span class="name-combo"><span class="token token-party"><abbr title="Independent">I</abbr></span> G. Johnson</span></th>
<td class="results-percentage"><span class="percentage-combo"><span class="number">2.1%</span><span class="graph"><span class="bar"><span class="index" style="width:2.1%;"></span></span></span></span></td>
<td class="results-popular">43,869</td>
<td class="delegates-cell"></td>
</tr>
<tr class="type-independent">
<th class="results-name" scope="row"><span class="name-combo"><span class="token token-party"><abbr title="Independent">I</abbr></span> J. Stein</span></th>
<td class="results-percentage"><span class="percentage-combo"><span class="number">0.4%</span><span class="graph"><span class="bar"><span class="index" style="width:0.4%;"></span></span></span></span></td>
<td class="results-popular">9,287</td>
<td class="delegates-cell"></td>
</tr>
</tbody>
</table>, <table class="results-table">
<tbody>
<tr class="type-republican">
<th class="results-name" scope="row"><a href="xxxxx"><span class="name-combo"><span class="token token-party"><abbr title="Republican">R</abbr></span> D. Trump</span></a></th>
<td class="results-percentage"><span class="percentage-combo"><span class="number">73.4%</span><span class="graph"><span class="bar"><span class="index" style="width:73.4%;"></span></span></span></span></td>
<td class="results-popular">18,110</td>
</tr>
<tr class="type-democrat">
<th class="results-name" scope="row"><a href="xxxxxx"><span class="name-combo"><span class="token token-party"><abbr title="Democratic">D</abbr></span> H. Clinton</span></a></th>
<td class="results-percentage"><span class="percentage-combo"><span class="number">24.0%</span><span class="graph"><span class="bar"><span class="index" style="width:24.0%;"></span></span></span></span></td>
<td class="results-popular">5,908</td>
</tr>
<tr class="type-independent">
<th class="results-name" scope="row"><span class="name-combo"><span class="token token-party"><abbr title="Independent">I</abbr></span> G. Johnson</span></th>
<td class="results-percentage"><span class="percentage-combo"><span class="number">2.2%</span><span class="graph"><span class="bar"><span class="index" style="width:2.2%;"></span></span></span></span></td>
<td class="results-popular">538</td>
</tr>
<tr class="type-independent">
<th class="results-name" scope="row"><span class="name-combo"><span class="token token-party"><abbr title="Independent">I</abbr></span> J. Stein</span></th>
<td class="results-percentage"><span class="percentage-combo"><span class="number">0.4%</span><span class="graph"><span class="bar"><span class="index" style="width:0.4%;"></span></span></span></span></td>
<td class="results-popular">105</td>
</tr>
</tbody>

但我在这里得到的是来自少数几张表的信息，但不是全部。如何从所有表中获取信息？为什么我只从几张表中获取信息

我希望你能理解这个问题

HTML真的很大，所以我给网站添加了链接。我想搜集阿拉巴马州每个县2016年的美国选举数据

import requests, bs4

r = requests.get('http://www.politico.com/2016-election/results/map/president/alabama/')
soup = bs4.BeautifulSoup(r.text, 'lxml')
contents = soup.find(class_='contrast-white')
for table in contents.find_all(class_='results-group'):
    title = table.find(class_='title').text
    for tr in table.find_all('tr'):
        _, name, percentage, popular = [td for td in tr.stripped_strings]
        print(title, name, percentage, popular)

输出：

其余的都是空的，里面什么都没有。

所以过了一段时间，我设法从这个网站上搜集了所有数据。所以主要的问题是，该网站嵌入了JavaScript，所以我无法使用Beautifulsoup。所以我使用selenium+beautifulsoup4，将页面转换为html并将其删除

from selenium import webdriver
import time
import os
from bs4 import BeautifulSoup
chrome_path = r"C:\Users\Desktop\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get('http://www.politico.com/2016-election/primary/results/map/president/arizona/')
time.sleep(80)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(5)
html = driver.page_source
soup = BeautifulSoup(html,'html.parser')
for posts in soup.findAll('table',{'class':'results-table'}):
for tr in posts.findAll('tr'):
    popular = [td for td in tr.stripped_strings]
    print(popular)

因为它是动态网页，我需要用selenium模拟一些东西。比如向下滚动页面。我使用time.sleep（60）来加载页面。它的加载速度非常慢，所以我将时间设置为60秒。希望它对某人有所帮助。

此处的数据中不存在“content alpha”类。你能更新你想要刮取的数据和预期结果吗？如果你提供你想要刮取的url，我们会更容易帮助你。我添加了网站链接。谢谢你的回答。我是python新手，所以我有同样的问题，为什么它只刮取页面的一部分直到切诺基县？所以没有办法刮取其余的县？@Extria因为页面中没有信息，我们不能从无到有地刮取。但是如果数据显示在网站上，应该有办法刮取还是没有？

Autauga County D. Trump 73.4% 18,110
Autauga County H. Clinton 24.0% 5,908
Autauga County G. Johnson 2.2% 538
Autauga County J. Stein 0.4% 105
Baldwin County D. Trump 77.4% 72,780
Baldwin County H. Clinton 19.6% 18,409
Baldwin County G. Johnson 2.6% 2,448
Baldwin County J. Stein 0.5% 453
Barbour County D. Trump 52.3% 5,431
Barbour County H. Clinton 46.7% 4,848
Barbour County G. Johnson 0.9% 93
Barbour County J. Stein 0.2% 18
Bibb County D. Trump 77.0% 6,733
Bibb County H. Clinton 21.4% 1,874
Bibb County G. Johnson 1.4% 124
Bibb County J. Stein 0.2% 17
Blount County D. Trump 89.9% 22,808
Blount County H. Clinton 8.5% 2,150
Blount County G. Johnson 1.3% 337
Blount County J. Stein 0.4% 89
Bullock County H. Clinton 75.1% 3,530
Bullock County D. Trump 24.2% 1,139
Bullock County G. Johnson 0.5% 22
Bullock County J. Stein 0.2% 10
Butler County D. Trump 56.3% 4,891
Butler County H. Clinton 42.8% 3,716
Butler County G. Johnson 0.7% 65
Butler County J. Stein 0.1% 13
Calhoun County D. Trump 69.2% 32,803
Calhoun County H. Clinton 27.9% 13,197
Calhoun County G. Johnson 2.4% 1,114
Calhoun County J. Stein 0.6% 262
Chambers County D. Trump 56.6% 7,803
Chambers County H. Clinton 41.8% 5,763
Chambers County G. Johnson 1.2% 168
Chambers County J. Stein 0.3% 44
Cherokee County D. Trump 83.9% 8,809
Cherokee County H. Clinton 14.5% 1,524
Cherokee County G. Johnson 1.4% 145
Cherokee County J. Stein 0.2% 25

from selenium import webdriver
import time
import os
from bs4 import BeautifulSoup
chrome_path = r"C:\Users\Desktop\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get('http://www.politico.com/2016-election/primary/results/map/president/arizona/')
time.sleep(80)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(5)
html = driver.page_source
soup = BeautifulSoup(html,'html.parser')
for posts in soup.findAll('table',{'class':'results-table'}):
for tr in posts.findAll('tr'):
    popular = [td for td in tr.stripped_strings]
    print(popular)