Html 不能刮<;div id=";搜索容器“&燃气轮机;来自pdb数据库网站
我是网络垃圾新手,尝试访问pdb网页上的每个pdb ID,例如:Html 不能刮<;div id=";搜索容器“&燃气轮机;来自pdb数据库网站,html,web-scraping,beautifulsoup,urllib,pdb,Html,Web Scraping,Beautifulsoup,Urllib,Pdb,我是网络垃圾新手,尝试访问pdb网页上的每个pdb ID,例如: url= https://www.rcsb.org/search?request=%7B%22query%22%3A%7B%22parameters%22%3A%7B%22attribute%22%3A%22rcsb_entity_source_organism.rcsb_gene_name.value%22%2C%22operator%22%3A%22exact_match%22%2C%22value%22%3A%22MCF3%
url= https://www.rcsb.org/search?request=%7B%22query%22%3A%7B%22parameters%22%3A%7B%22attribute%22%3A%22rcsb_entity_source_organism.rcsb_gene_name.value%22%2C%22operator%22%3A%22exact_match%22%2C%22value%22%3A%22MCF3%22%7D%2C%22type%22%3A%22terminal%22%2C%22service%22%3A%22text%22%2C%22node_id%22%3A0%7D%2C%22return_type%22%3A%22entry%22%2C%22request_options%22%3A%7B%22pager%22%3A%7B%22start%22%3A0%2C%22rows%22%3A100%7D%2C%22scoring_strategy%22%3A%22combined%22%2C%22sort%22%3A%5B%7B%22sort_by%22%3A%22score%22%2C%22direction%22%3A%22desc%22%7D%5D%7D%2C%22request_info%22%3A%7B%22src%22%3A%22ui%22%2C%22query_id%22%3A%22c75bbf18a058a812f5384f297528d4b6%22%7D%7D
在这里,我试图获得像“3ZBF”
和“4UXL”
这样的ID
我写了下面的代码:
url='https://www.rcsb.org/search?request=%7B%22query%22%3A%7B%22parameters%22%3A%7B%22attribute%22%3A%22rcsb_entity_source_organism.rcsb_gene_name.value%22%2C%22operator%22%3A%22exact_match%22%2C%22value%22%3A%22MCF3%22%7D%2C%22type%22%3A%22terminal%22%2C%22service%22%3A%22text%22%2C%22node_id%22%3A0%7D%2C%22return_type%22%3A%22entry%22%2C%22request_options%22%3A%7B%22pager%22%3A%7B%22start%22%3A0%2C%22rows%22%3A100%7D%2C%22scoring_strategy%22%3A%22combined%22%2C%22sort%22%3A%5B%7B%22sort_by%22%3A%22score%22%2C%22direction%22%3A%22desc%22%7D%5D%7D%2C%22request_info%22%3A%7B%22src%22%3A%22ui%22%2C%22query_id%22%3A%22c75bbf18a058a812f5384f297528d4b6%22%7D%7D'
page = requests.get(url)
data = page.text
soup = BeautifulSoup(data, "html.parser")
tex_tag= soup.find('div',{"class":"container","id":"maincontentcontainer"})
new_line=tex_tag.find("div",{"id":"search-container"})
print(new_line)
print(tex_tag.prettify())
在这里,我看不到的内部。我已经检查了网页上的html文件,pdb id位于
中
你能给我一个解决方案,或者给我一个如何解决这个问题的见解吗
先谢谢你 此站点使用API在呈现结果之前获取结果。它来自以下url:
此脚本将根据您的搜索URL打印所有标识符:
感谢您的回复,我的目的是找出pdb ID,逐个输入基因名称。我不知道pdb ID,我只想输入基因名来提取它们。因此,我尝试使用网页抓取。你有什么建议给我吗?提前谢谢,非常感谢。你帮了大忙。
先谢谢你 此站点使用API在呈现结果之前获取结果。它来自以下url:
POST https://www.rcsb.org/search/gql
在JSON输入中传递标识符列表时:
import requests
ids = ["3ZBF","4UXL"]
r = requests.post("https://www.rcsb.org/search/gql",
json = {
"attributes": None,
"identifiers": ids,
"returnType": "entry",
"report": "search_summary"
})
print(r.json())
此脚本将根据您的搜索URL打印所有标识符:
import json
import requests
import urllib.parse
url = 'https://www.rcsb.org/search?request=%7B%22query%22%3A%7B%22parameters%22%3A%7B%22attribute%22%3A%22rcsb_entity_source_organism.rcsb_gene_name.value%22%2C%22operator%22%3A%22exact_match%22%2C%22value%22%3A%22MCF3%22%7D%2C%22type%22%3A%22terminal%22%2C%22service%22%3A%22text%22%2C%22node_id%22%3A0%7D%2C%22return_type%22%3A%22entry%22%2C%22request_options%22%3A%7B%22pager%22%3A%7B%22start%22%3A0%2C%22rows%22%3A100%7D%2C%22scoring_strategy%22%3A%22combined%22%2C%22sort%22%3A%5B%7B%22sort_by%22%3A%22score%22%2C%22direction%22%3A%22desc%22%7D%5D%7D%2C%22request_info%22%3A%7B%22src%22%3A%22ui%22%2C%22query_id%22%3A%22c75bbf18a058a812f5384f297528d4b6%22%7D%7D'
search_link = 'https://www.rcsb.org/search/data'
json_data = url.split('=')[-1]
json_data = json.loads(urllib.parse.unquote(json_data))
data = requests.post(search_link, json=json_data).json()
# uncomment this to print all data:
# print(json.dumps(data, indent=4))
for r in data['result_set']:
print(r['identifier'])
印刷品:
3ZBF
4UXL
感谢您的回复,我的目的是找出pdb ID,逐个输入基因名称。我不知道pdb ID,我只想输入基因名来提取它们。因此,我尝试使用网页抓取。你有什么建议给我吗?提前谢谢,非常感谢。你帮了大忙。