Python 从特定表元素中刮取特定文本时返回错误数据
多亏了SO和@QHarr,下面的代码可以很好地处理URL,例如Python 从特定表元素中刮取特定文本时返回错误数据,python,html,python-3.x,web-scraping,beautifulsoup,Python,Html,Python 3.x,Web Scraping,Beautifulsoup,多亏了SO和@QHarr,下面的代码可以很好地处理URL,例如 https://www.amazon.com/dp/B00FSCBQV2 但它不适用于这样的URL- https://www.amazon.com/dp/B01N1ZD912/ 我的结果是—— 'R1_NO' :'.zg_hrsr { margin: 0; padding: 0; list-style-type: none;}\n.zg_hrsr_item { margin: 0 0 0 10px; }\n.zg_hrsr_r
https://www.amazon.com/dp/B00FSCBQV2
但它不适用于这样的URL-
https://www.amazon.com/dp/B01N1ZD912/
我的结果是——
'R1_NO' :'.zg_hrsr { margin: 0; padding: 0; list-style-type:
none;}\n.zg_hrsr_item { margin: 0 0 0 10px; }\n.zg_hrsr_rank {
display:inline-block; width: 80px; text-align: right; }'}'
它应该回来了
R1_NO = 42553
R1_CAT = Baby Care Products
R2_NO = 6452
R2_CAT = Baby Bathing Products (Health & Household)
这是因为排名数据不在第一行。需要做些什么才能获得预期的结果?该脚本是否可以精简/更高效
我试着用bs4 select.one处理它,获取文本条,但我所做的一切都不起作用。请帮帮我
fields = ['Amazon Best Sellers Rank']
temp_dict = {}
for field in fields:
element = soup.select_one('li:contains("' + field + '")')
if element is None:
temp_dict[field] = 'NA'
else:
if field == 'Amazon Best Sellers Rank':
item='NA'
item = [re.sub('#|\(','', string).strip() for string in soup.select_one('li:contains("' + field + '")').stripped_strings][1].split(' in ')
temp_dict[field] = item
else:
item = [string for string in element.stripped_strings][1]
temp_dict[field] = item.replace('(', '').strip()
ranks = soup.select('.zg_hrsr_rank')
ladders = soup.select('.zg_hrsr_ladder')
if ranks:
cat_nos = [item.text.split('#')[1] for item in ranks]
else:
cat_nos = ['NA']
if ladders:
cats = [item.text.split('\xa0')[1] for item in soup.select('.zg_hrsr_ladder')]
else:
cats = ['NA']
rankings = dict(zip(cat_nos, cats))
map_dict = {'Amazon Best Sellers Rank': ['R1_NO','R1_CAT']}
final_dict = {}
final_dict['R2_NO'] = 'NA'
final_dict['R2_CAT'] = 'NA'
final_dict['R3_NO'] = 'NA'
final_dict['R3_CAT'] = 'NA'
final_dict['R4_NO'] = 'NA'
final_dict['R4_CAT'] = 'NA'
for k,v in temp_dict.items():
if k == 'Amazon Best Sellers Rank' and v!= 'NA':
item = dict(zip(map_dict[k],v))
final_dict = {**final_dict, **item}
elif k == 'Amazon Best Sellers Rank' and v == 'NA':
item = dict(zip(map_dict[k], [v, v]))
final_dict = {**final_dict, **item}
else:
final_dict[map_dict[k]] = v
for k,v in enumerate(rankings):
#print(k + 1, v, rankings[v])
prefix = 'R' + str(k + 2) + '_'
final_dict[prefix + 'NO'] = v
final_dict[prefix + 'CAT'] = rankings[v]
我希望它能够处理并返回问题中发布的两个URL的值,因此由于html布局的不同,剥离的字符串导致返回内联css。你可以尝试缩短和使用正则表达式。你可以收紧正则表达式,但我会等着看你是否先找到失败的例子
import requests
from bs4 import BeautifulSoup as bs
import re
links = ['https://www.amazon.com/dp/B00FSCBQV2?th=1','https://www.amazon.com/dp/B01N1ZD912/','https://www.amazon.com/Professional-Dental-Guard-Remoldable-Customizable/dp/B07L4YHBQ4', 'https://www.amazon.com/dp/B0040ODFK4/?tag=stackoverfl08-20']
map_dict = {'Product Dimensions': 'dimensions', 'Shipping Weight': 'weight', 'Item model number': 'Item_No', 'Amazon Best Sellers Rank': ['R1_NO','R1_CAT']}
# This handles when a ranking is from 1 to x,xxx,xxx
p = re.compile(r'#([0-9][0-9,]*)+[\n\s]+in[\n\s]+([A-Za-z&\s]+)')
with requests.Session() as s:
for link in links:
r = s.get(link, headers = {'User-Agent': 'Mozilla\5.0'})
soup = bs(r.content, 'lxml')
fields = ['Product Dimensions', 'Shipping Weight', 'Item model number', 'Amazon Best Sellers Rank']
final_dict = {}
for field in fields:
element = soup.select_one('li:contains("' + field + '")')
if element is None:
if field == 'Amazon Best Sellers Rank':
item = dict(zip(map_dict[field], ['N/A','N/A']))
final_dict = {**final_dict, **item}
else:
final_dict[map_dict[field]] = 'N/A'
else:
if field == 'Amazon Best Sellers Rank':
text = element.text
i = 1
for x,y in p.findall(text):
prefix = 'R' + str(i) + '_'
final_dict[prefix + 'NO'] = x
final_dict[prefix + 'CAT'] = y.strip()
i+=1
else:
item = [string for string in element.stripped_strings][1]
final_dict[map_dict[field]] = item.replace('(', '').strip()
print(final_dict)
除了只返回R1s,而不返回R2或R3之外,它适用于所有情况。我想这与p=re.compile(r'#(\d+,?\d+)有关,因为我用了它,让它显示R2和R3,但它没有显示R1。我看一下,我通过将(\d+,?\d+)更改为(\d{0,20},\d+)解决了这个问题我相信这是因为排名为“1”,并且让正则表达式调用零行。如果你知道更好的方法,请修改你的答案。我编辑了它,但如果需要,一定要这样做。如果你高兴,那么我很高兴…直到下一个链接打破它;-)你修改的正则表达式对最初使用的每个链接都有效吗?