使用python刮取google resultstats
我想从谷歌获得一个关键字的估计结果数。我正在使用Python3.3，并尝试使用BeautifulSoup和urllib.request来完成这项任务。这是迄今为止我的简单代码使用python刮取google resultstats,python,beautifulsoup,urllib2,scrape,Python,Beautifulsoup,Urllib2,Scrape,我想从谷歌获得一个关键字的估计结果数。我正在使用Python3.3，并尝试使用BeautifulSoup和urllib.request来完成这项任务。这是迄今为止我的简单代码 def numResults(): try: page_google = '''http://www.google.de/#output=search&sclient=psy-ab&q=pokerbonus&oq=pokerbonus&gs_l=hp.3..0i10l2j0i10i3
def numResults():
try:
page_google = '''http://www.google.de/#output=search&sclient=psy-ab&q=pokerbonus&oq=pokerbonus&gs_l=hp.3..0i10l2j0i10i30l2.16503.18949.0.20819.10.9.0.1.1.0.413.2110.2-6j1j1.8.0....0...1c.1.19.psy-ab.FEBvxrgi0KU&pbx=1&bav=on.2,or.r_qf.&bvm=bv.48705608,d.Yms&'''
req_google = Request(page_google)
req_google.add_header('User Agent', 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20120427 Firefox/15.0a1')
html_google = urlopen(req_google).read()
soup = BeautifulSoup(html_google)
scounttext = soup.find('div', id='resultStats')
except URLError as e:
print(e)
return scounttext
非常感谢 如果您还没有解决这个问题，那么BeautifulSoup找不到任何东西的原因似乎是resultStats从未出现在soup中-您的请求（谷歌页面）只是返回JavaScript，而不是JavaScript动态加载的任何搜索结果。您可以通过添加
print(soup)
import sys
from urllib2 import Request, urlopen
import urllib
from bs4 import BeautifulSoup
query = 'pokerbonus'
url = "http://www.google.de/search?q=%s" % urllib.quote_plus(query)
req_google = Request(url)
req_google.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3')
html_google = urlopen(req_google).read()
soup = BeautifulSoup(html_google)
scounttext = soup.find('div', id='resultStats')
print(scounttext)
<div class="sd" id="resultStats">Ungefähr 1.060.000 Ergebnisse</div>
Ungefähr 1.060.000 Ergebnisse