Python Web抓取-urlopen错误[Errno-2]名称或服务未知
我正试图从我的项目链接中提取数据。我能够获得我需要的页面链接。但当我试图打开链接时，我得到“urlopen error[Errno-2]名称或服务未知” web抓取python代码：Python Web抓取-urlopen错误[Errno-2]名称或服务未知,python,web-scraping,beautifulsoup,Python,Web Scraping,Beautifulsoup,我正试图从我的项目链接中提取数据。我能够获得我需要的页面链接。但当我试图打开链接时，我得到“urlopen error[Errno-2]名称或服务未知” web抓取python代码： from bs4 import BeautifulSoup from urlparse import urlparse, parse_qs import re import urllib2 import pdb base_url = "http://civiccommons.org" url
from bs4 import BeautifulSoup
from urlparse import urlparse, parse_qs
import re
import urllib2
import pdb
base_url = "http://civiccommons.org"
url = "http://civiccommons.org/apps"
page = urllib2.urlopen(url)
soup = BeautifulSoup(page.read())
list_of_links = []
for link_tag in soup.findAll('a', href=re.compile('^/civic-function.*')):
string_temp_link = base_url+link_tag.get('href')
list_of_links.append(string_temp_link)
list_of_links = list(set(list_of_links))
list_of_next_pages = []
for categorized_apps_url in list_of_links:
categorized_apps_page = urllib2.urlopen(categorized_apps_url)
categorized_apps_soup = BeautifulSoup(categorized_apps_page.read())
last_page_tag = categorized_apps_soup.find('a', title="Go to last page")
if last_page_tag:
last_page_url = base_url+last_page_tag.get('href')
index_value = last_page_url.find("page=") + 5
base_url_for_next_page = last_page_url[:index_value]
for pageno in xrange(0, int(parse_qs(urlparse(last_page_url).query)['page'][0]) + 1):
list_of_next_pages.append(base_url_for_next_page+str(pageno))
else:
list_of_next_pages.append(categorized_apps_url)
urllib2.urlopen(categorized_apps_url)
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno -2] Name or service not known>
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
raise URLError(err)
我还尝试在scraperwiki中运行代码，并成功完成。但很少有url丢失（与mac相比）。这些行为有什么原因吗？代码在我的Mac电脑和你的朋友的Mac电脑上都有效。它在Ubuntu 12.04服务器的虚拟机实例上运行良好。很明显，在您的特定环境中，您的操作系统（Ubuntu桌面？）或网络导致了它的崩溃。例如，我的家庭路由器的默认设置会在x秒内限制对同一域的呼叫数——如果我没有关闭它，可能会导致此类问题。可能有很多事情 在这个阶段，我建议重构代码以捕获
URLErrorurlopenurlopentime.sleep（1）最近，我跳进一个很久没玩过的旧ubuntu盒子后，遇到了同样的问题。此问题实际上是由您计算机上的DNS设置引起的。我强烈建议您检查您的DNS设置（/etc/resolv.conf并添加nameserver 8.8.8.8），然后再试一次，您应该会成功。在出现错误时，
categorized\u apps\u urlurllib2