Python 我们可以将XPath与BeautifulSoup一起使用吗？
我正在使用BeautifulSoup刮取一个URL，我有以下代码Python 我们可以将XPath与BeautifulSoup一起使用吗？,python,xpath,beautifulsoup,urllib,Python,Xpath,Beautifulsoup,Urllib,我正在使用BeautifulSoup刮取一个URL，我有以下代码 import urllib import urllib2 from BeautifulSoup import BeautifulSoup url = "http://www.example.com/servlet/av/ResultTemplate=AVResult.html" req = urllib2.Request(url) response = urllib2.urlopen(req) the_pa
import urllib
import urllib2
from BeautifulSoup import BeautifulSoup
url = "http://www.example.com/servlet/av/ResultTemplate=AVResult.html"
req = urllib2.Request(url)
response = urllib2.urlopen(req)
the_page = response.read()
soup = BeautifulSoup(the_page)
soup.findAll('td',attrs={'class':'empformbody'})
现在在上面的代码中，我们可以使用
findAll.xpath（）try:
# Python 2
from urllib2 import urlopen
except ImportError:
from urllib.request import urlopen
from lxml import etree
url = "http://www.example.com/servlet/av/ResultTemplate=AVResult.html"
response = urlopen(url)
htmlparser = etree.HTMLParser()
tree = etree.parse(response, htmlparser)
tree.xpath(xpathselector)
响应lxml请求stream=True响应。原始CSSSelectortd.empformbodyfrom lxml.cssselect import CSSSelector
td_empformbody = CSSSelector('td.empformbody')
for elem in td_empformbody(tree):
# Do something with these table cells.
我可以确认Beautiful Soup中不支持XPath。BeautifulSoup有一个名为current element directed childern的函数，因此：
father.findNext('div',{'class':'class_value'}).findNext('div',{'id':'id_value'}).findAll('a')
div[class=class_value]/div[id=id_value]
正如其他人所说，BeautifulSoup不支持xpath。从xpath获取内容可能有多种方法，包括使用Selenium。但是，这里有一个在Python 2或Python 3中都可以使用的解决方案：
from lxml import html
import requests
page = requests.get('http://econpy.pythonanywhere.com/ex/001.html')
tree = html.fromstring(page.content)
#This will create a list of buyers:
buyers = tree.xpath('//div[@title="buyer-name"]/text()')
#This will create a list of prices
prices = tree.xpath('//span[@class="item-price"]/text()')
print('Buyers: ', buyers)
print('Prices: ', prices)
我用作参考。这是一个非常古老的线程，但现在有一个变通解决方案，它可能当时不在BeautifulSoup中 这是我所做的一个例子。我使用“requests”模块读取RSS提要，并在一个名为“RSS_text”的变量中获取其文本内容。这样，我就可以通过BeautifulSoup运行它，搜索xpath/rss/channel/title，并检索其内容。它并不完全是XPath（通配符、多路径等），但如果您只需要找到一个基本路径，它就可以工作
from bs4 import BeautifulSoup
rss_obj = BeautifulSoup(rss_text, 'xml')
cls.title = rss_obj.rss.channel.title.get_text()
使用lxml时，所有操作都很简单：
tree = lxml.html.fromstring(html)
i_need_element = tree.xpath('//a[@class="shared-components"]/@href')
- 首先删除“/”和“@”
- 第二个-在“=”之前添加星号
soup = BeautifulSoup(html, "lxml")
i_need_element = soup.select ('a[class*="shared-components"]')
如您所见，这不支持子标记，因此我删除了“/@href”部分也许您可以在不使用XPath的情况下尝试以下操作
from simplified_scrapy.simplified_doc import SimplifiedDoc
html = '''
<html>
<body>
<div>
<h1>Example Domain</h1>
<p>This domain is for use in illustrative examples in documents. You may use this
domain in literature without prior coordination or asking for permission.</p>
<p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>
'''
# What XPath can do, so can it
doc = SimplifiedDoc(html)
# The result is the same as doc.getElementByTag('body').getElementByTag('div').getElementByTag('h1').text
print (doc.body.div.h1.text)
print (doc.div.h1.text)
print (doc.h1.text) # Shorter paths will be faster
print (doc.div.getChildren())
print (doc.div.getChildren('p'))
从simplified_scrapy.simplified_文档导入SimplifiedDoc
html=“”
示例域
此域用于文档中的示例。你可以用这个
未经事先协调或请求许可的文献中的域
'''
#XPath能做什么，它也能做什么
doc=SimplifiedDoc（html）
#结果与doc.getElementByTag（'body'）.getElementByTag（'div'）.getElementByTag（'h1'）.text相同
打印（doc.body.div.h1.text）
打印（doc.div.h1.text）
打印（doc.h1.text）#路径越短速度越快
打印（doc.div.getChildren（））
打印（doc.div.getChildren（'p'））
上面使用了Soup对象与lxml的组合，可以使用xpath提取值。使用
Soup.find（class='myclass'）urllib2urllib.requestselect（）soup = BeautifulSoup(html, "lxml")
i_need_element = soup.select ('a[class*="shared-components"]')
from simplified_scrapy.simplified_doc import SimplifiedDoc
html = '''
<html>
<body>
<div>
<h1>Example Domain</h1>
<p>This domain is for use in illustrative examples in documents. You may use this
domain in literature without prior coordination or asking for permission.</p>
<p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>
'''
# What XPath can do, so can it
doc = SimplifiedDoc(html)
# The result is the same as doc.getElementByTag('body').getElementByTag('div').getElementByTag('h1').text
print (doc.body.div.h1.text)
print (doc.div.h1.text)
print (doc.h1.text) # Shorter paths will be faster
print (doc.div.getChildren())
print (doc.div.getChildren('p'))
from lxml import etree
from bs4 import BeautifulSoup
soup = BeautifulSoup(open('path of your localfile.html'),'html.parser')
dom = etree.HTML(str(soup))
print dom.xpath('//*[@id="BGINP01_S1"]/section/div/font/text()')