Python 全力以赴<；a>；来自<；部门>；具有特定类的标记_Python_Web Scraping_Beautifulsoup

Python 全力以赴<；a>；来自<；部门>；具有特定类的标记

python web-scraping

Python 全力以赴<；a>；来自<；部门>；具有特定类的标记,python,web-scraping,beautifulsoup,Python,Web Scraping,Beautifulsoup,我正在调试并获得“用python自动完成无聊的工作”中的lucky.py代码。这里的主要问题是作者的代码不起作用（可能已经过时）。该代码旨在在执行python脚本时传递命令行参数。该脚本在新选项卡中打开参数的前五个（或更少）谷歌搜索结果。现在，原始代码使用“r”类提取所有标记。然而，现在，google不再使用“r”类来搜索结果超链接，而是简单地将同一标签与“r”类封装在一个div中这就是原始代码所做的 res = requests.get('http://google.com/search?q

我正在调试并获得“用python自动完成无聊的工作”中的lucky.py代码。这里的主要问题是作者的代码不起作用（可能已经过时）。该代码旨在在执行python脚本时传递命令行参数。该脚本在新选项卡中打开参数的前五个（或更少）谷歌搜索结果。现在，原始代码使用“r”类提取所有标记。然而，现在，google不再使用“r”类来搜索结果超链接，而是简单地将同一标签与“r”类封装在一个div中

这就是原始代码所做的

res = requests.get('http://google.com/search?q=' +' '.join(sys.argv[1:]))
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, 'lxml')

linkElems = soup.select('.r a')
numOpen = min(5, len(linkElems))
for i in range(numOpen):
    webbrowser.open('http://google.com' + linkElems[i].get('href'))

我已经尝试获取直接封装在div中的所有标记，但是我找不到任何方法来提取直接封装在'r'类标记中的所有标记

以下是我想到的一些事情，但它们不能正常工作

linkElems = soup.select('.r div > a')

这是因为我想要的所有标记都有以“\url”开头的ping属性

 linkElems = soup.select('a')
 for link in linkElems:
     if link.attrs.hget('ping').startswith('\\url'):
         ...

这个代码对我有用

soup = BeautifulSoup(res.text, "html.parser")
for div in soup.find_all("div", {"class": "class name"}):
    for a in div.find_all("a", {"class": "r"}):
        print(a.attrs['href'])

您可以使用

find_all（）

函数获取所有

tags name

，如果您想获取具有特定

属性的所有tags
，您应该发送另一个dict
作为find_all（）
函数的输入。
是的，这篇文章似乎过时了。没有任何带有classr
（至少在我的例子中）的标记，但您仍然可以通过href
属性选择链接
要选择以/url
开头的href
属性的所有
标记，可以使用CSS选择器a[href^=“/url”]

import bs4
import requests

search_term = 'tree'

res = requests.get('http://google.com/search?q=' + search_term)
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, 'lxml')

for link in soup.select('a[href^="/url"]'):
    print(link['href'])

印刷品：
/url?q=https://en.wikipedia.org/wiki/Tree&sa=U&ved=2ahUKEwj4iMW3rN7jAhWJxMQBHag1Cr4QFjAGegQIBxAB&usg=AOvVaw3paXH3cMIxBpu9X0bAY3mR
/url?q=https://en.wikipedia.org/wiki/Tree_line&sa=U&ved=2ahUKEwj4iMW3rN7jAhWJxMQBHag1Cr4Q0gIwBnoECAcQAg&usg=AOvVaw3ynJgH_Bbw1mSqAL8ovO7e
/url?q=https://en.wikipedia.org/wiki/Tree_(disambiguation)&sa=U&ved=2ahUKEwj4iMW3rN7jAhWJxMQBHag1Cr4Q0gIwBnoECAcQAw&usg=AOvVaw1Dcz4l8mkB9jZHqeJKT9B9
/url?q=https://en.wikipedia.org/wiki/Portal:Trees&sa=U&ved=2ahUKEwj4iMW3rN7jAhWJxMQBHag1Cr4Q0gIwBnoECAcQBA&usg=AOvVaw0mZS3EU93_a96SpiqfFG-R
/url?q=https://en.wikipedia.org/wiki/I-Tree&sa=U&ved=2ahUKEwj4iMW3rN7jAhWJxMQBHag1Cr4Q0gIwBnoECAcQBQ&usg=AOvVaw2lq87vNdcDmw0tCZxeIs_E

... and so on.

/url?q=https://en.wikipedia.org/wiki/Tree&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjAGegQIAxAB&usg=AOvVaw213y4pDofhSr3-AzbeN6Xe
/url?q=https://en.wikipedia.org/wiki/Tree_line&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwBnoECAMQAg&usg=AOvVaw0qQCjrcrP6YHGLeeSvYkN1
/url?q=https://en.wikipedia.org/wiki/Tree_(disambiguation)&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwBnoECAMQAw&usg=AOvVaw2OSqEJ_jRM_ByhjfvMSzjC
/url?q=https://en.wikipedia.org/wiki/Portal:Trees&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwBnoECAMQBA&usg=AOvVaw1Xh2A4mp3beT6zQNzS8aJD
/url?q=https://en.wikipedia.org/wiki/I-Tree&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwBnoECAMQBQ&usg=AOvVaw1ARsOn-3cMHsILu_-1AF-Q
/url?q=https://simple.wikipedia.org/wiki/Tree&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjAHegQICBAB&usg=AOvVaw3J9VoAcyvn01DK6VQjQOcJ
/url?q=https://simple.wikipedia.org/wiki/Tree%23Parts_of_trees&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwB3oECAgQAg&usg=AOvVaw3uiAZjYQTYR02__Da6xkHi
/url?q=https://simple.wikipedia.org/wiki/Tree%23Records&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwB3oECAgQAw&usg=AOvVaw2jexFkOqkPQ3bHZ1q1KdKj
/url?q=https://simple.wikipedia.org/wiki/Tree%23Tree_value_estimation&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwB3oECAgQBA&usg=AOvVaw3URu63Yk-j0o-G75SSaeW3
/url?q=https://simple.wikipedia.org/wiki/Tree%23Tree_climbing&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwB3oECAgQBQ&usg=AOvVaw2YmeOvTuDS2cacWiM7Fzj6
/url?q=https://www.royalparks.org.uk/parks/the-regents-park/things-to-see-and-do/gardens-and-landscapes/tree-map/why-trees-are-important&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjAIegQIARAB&usg=AOvVaw0uk4ZAk22_zyuVRPmGGEae
/url?q=https://www.homedepot.com/b/Outdoors-Garden-Center-Trees-Bushes/N-5yc1vZc8rq&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjAJegQIAhAB&usg=AOvVaw1v36Vzsvx9s-0BPWGp3QrH
/url?q=https://www.britannica.com/plant/tree&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjAKegQIABAB&usg=AOvVaw101wIJj19V4TEj57BCA7Xe
/url?q=https://www.nparks.gov.sg/trees&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjALegQIBBAB&usg=AOvVaw3CDs1obwYNKnMwtMK2RBbG
/url?q=https://en.wiktionary.org/wiki/tree&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjAMegQIBxAB&usg=AOvVaw3AJJuZ5vY3I8TqOSfKtVa4
/url?q=https://www.bbc.com/news/uk-england-47541491&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjANegQIBRAB&usg=AOvVaw1d2QTAZ5JYAB9t6f11VY-_
/url?q=https://www.theguardian.com/world/2019/jul/29/ethiopia-plants-250m-trees-in-a-day-to-help-tackle-climate-crisis&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjAOegQIBhAB&usg=AOvVaw0c6bDr70Km_E8v3wmey124

编辑：要过滤IMG链接和内部帐户，您可以执行以下操作：
for link in soup.select('a[href^="/url"]'):
    if link.find('img'):
        continue
    if 'accounts.google.com' in link['href']:
        continue
    print(link['href'])

印刷品：
/url?q=https://en.wikipedia.org/wiki/Tree&sa=U&ved=2ahUKEwj4iMW3rN7jAhWJxMQBHag1Cr4QFjAGegQIBxAB&usg=AOvVaw3paXH3cMIxBpu9X0bAY3mR
/url?q=https://en.wikipedia.org/wiki/Tree_line&sa=U&ved=2ahUKEwj4iMW3rN7jAhWJxMQBHag1Cr4Q0gIwBnoECAcQAg&usg=AOvVaw3ynJgH_Bbw1mSqAL8ovO7e
/url?q=https://en.wikipedia.org/wiki/Tree_(disambiguation)&sa=U&ved=2ahUKEwj4iMW3rN7jAhWJxMQBHag1Cr4Q0gIwBnoECAcQAw&usg=AOvVaw1Dcz4l8mkB9jZHqeJKT9B9
/url?q=https://en.wikipedia.org/wiki/Portal:Trees&sa=U&ved=2ahUKEwj4iMW3rN7jAhWJxMQBHag1Cr4Q0gIwBnoECAcQBA&usg=AOvVaw0mZS3EU93_a96SpiqfFG-R
/url?q=https://en.wikipedia.org/wiki/I-Tree&sa=U&ved=2ahUKEwj4iMW3rN7jAhWJxMQBHag1Cr4Q0gIwBnoECAcQBQ&usg=AOvVaw2lq87vNdcDmw0tCZxeIs_E

... and so on.

/url?q=https://en.wikipedia.org/wiki/Tree&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjAGegQIAxAB&usg=AOvVaw213y4pDofhSr3-AzbeN6Xe
/url?q=https://en.wikipedia.org/wiki/Tree_line&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwBnoECAMQAg&usg=AOvVaw0qQCjrcrP6YHGLeeSvYkN1
/url?q=https://en.wikipedia.org/wiki/Tree_(disambiguation)&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwBnoECAMQAw&usg=AOvVaw2OSqEJ_jRM_ByhjfvMSzjC
/url?q=https://en.wikipedia.org/wiki/Portal:Trees&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwBnoECAMQBA&usg=AOvVaw1Xh2A4mp3beT6zQNzS8aJD
/url?q=https://en.wikipedia.org/wiki/I-Tree&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwBnoECAMQBQ&usg=AOvVaw1ARsOn-3cMHsILu_-1AF-Q
/url?q=https://simple.wikipedia.org/wiki/Tree&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjAHegQICBAB&usg=AOvVaw3J9VoAcyvn01DK6VQjQOcJ
/url?q=https://simple.wikipedia.org/wiki/Tree%23Parts_of_trees&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwB3oECAgQAg&usg=AOvVaw3uiAZjYQTYR02__Da6xkHi
/url?q=https://simple.wikipedia.org/wiki/Tree%23Records&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwB3oECAgQAw&usg=AOvVaw2jexFkOqkPQ3bHZ1q1KdKj
/url?q=https://simple.wikipedia.org/wiki/Tree%23Tree_value_estimation&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwB3oECAgQBA&usg=AOvVaw3URu63Yk-j0o-G75SSaeW3
/url?q=https://simple.wikipedia.org/wiki/Tree%23Tree_climbing&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQ0gIwB3oECAgQBQ&usg=AOvVaw2YmeOvTuDS2cacWiM7Fzj6
/url?q=https://www.royalparks.org.uk/parks/the-regents-park/things-to-see-and-do/gardens-and-landscapes/tree-map/why-trees-are-important&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjAIegQIARAB&usg=AOvVaw0uk4ZAk22_zyuVRPmGGEae
/url?q=https://www.homedepot.com/b/Outdoors-Garden-Center-Trees-Bushes/N-5yc1vZc8rq&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjAJegQIAhAB&usg=AOvVaw1v36Vzsvx9s-0BPWGp3QrH
/url?q=https://www.britannica.com/plant/tree&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjAKegQIABAB&usg=AOvVaw101wIJj19V4TEj57BCA7Xe
/url?q=https://www.nparks.gov.sg/trees&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjALegQIBBAB&usg=AOvVaw3CDs1obwYNKnMwtMK2RBbG
/url?q=https://en.wiktionary.org/wiki/tree&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjAMegQIBxAB&usg=AOvVaw3AJJuZ5vY3I8TqOSfKtVa4
/url?q=https://www.bbc.com/news/uk-england-47541491&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjANegQIBRAB&usg=AOvVaw1d2QTAZ5JYAB9t6f11VY-_
/url?q=https://www.theguardian.com/world/2019/jul/29/ethiopia-plants-250m-trees-in-a-day-to-help-tackle-climate-crisis&sa=U&ved=2ahUKEwj9m9KPsN7jAhXwxcQBHb7eDcIQFjAOegQIBhAB&usg=AOvVaw0c6bDr70Km_E8v3wmey124

TLDR：从python脚本运行时，Google会发送不同的HTML响应
如果您实际打印linkElems
变量，您将看到它是空的。我认为这是因为Gooogle基于大量HTTP头更改了HTML。用外行的话说，这意味着您在浏览器中看到的HTML不是在运行Python的get请求时得到的HTML
现在，您可以使用linkElems=soup.select（'.jfp3ef>a'）
这样就可以了。它将选择所有。但既然你这样做是为了学习，我上面提到的黑客应该没问题
代码：
import bs4
import requests

res = requests.get('http://google.com/search?q=test')
soup = bs4.BeautifulSoup(res.text, 'html.parser')
linkElems = soup.select('.jfp3ef > a')
numOpen = min(5, len(linkElems))
for i in range(numOpen):
    print('http://google.com' + linkElems[i].get('href'))

输出：
http://google.com/url?q=https://www.speedtest.net/&sa=U&ved=2ahUKEwjP9eumr97jAhX2GLkGHbGoDuoQFjAKegQIChAB&usg=AOvVaw0mhIK0jUq5fUfhEJTuA90h
http://google.com/url?q=https://fast.com/&sa=U&ved=2ahUKEwjP9eumr97jAhX2GLkGHbGoDuoQFjALegQICRAB&usg=AOvVaw3WERIy0Wo_UNyqmNAVBCeZ
http://google.com/url?q=https://openspeedtest.com/Get-widget.php&sa=U&ved=2ahUKEwjP9eumr97jAhX2GLkGHbGoDuoQFjAMegQICBAB&usg=AOvVaw1161mhQBhD75gfmsIzzg4n
http://google.com/url?q=https://www.meter.net/&sa=U&ved=2ahUKEwjP9eumr97jAhX2GLkGHbGoDuoQFjANegQIBxAB&usg=AOvVaw2Z3xTSmhoxz6VS7MYAaS2x
http://google.com/url?q=https://speedtest.telstra.com/&sa=U&ved=2ahUKEwjP9eumr97jAhX2GLkGHbGoDuoQFjAOegQIARAB&usg=AOvVaw36SosexF66e8fQUWIG14mZ

这不是正确的答案，因为它将为您提供给定页面上的所有超链接，其中一些甚至可能不是搜索results@DollarAkshay你可以过滤更多的结果，看我的编辑。它仍然不太正确。前4个结果指向同一个域。如果你在浏览器中检查它，它是谷歌提供的4个域内超链接。@DollarAkshay，但这取决于用例-你可以根据域过滤更多的链接。或者不，如果你想要所有的链接。