Python 抓取亚马逊评论,不能排除付费评论
我正试图从每一个评论员给一个产品的星星数中剔除。我注意到有些评论员是“藤蔓之声”或付费评论员。他们很少给出4颗星,大部分是5颗星。因此,我想排除它们 我这样做的方式是,如果评论标记为“a-color-success a-text-bold”,则将其标记为“付费”或“未付费” 我似乎无法在vine变量中添加任何“Paid”标记。为什么 只有那些由Vine Voice撰写的评论才有标签,那些没有“付费”标签的评论才没有标签 这就是我目前得到的。 评论2和评论3是Vine Voice,但它们被标记为未付费,但应该付费Python 抓取亚马逊评论,不能排除付费评论,python,html,web-scraping,beautifulsoup,Python,Html,Web Scraping,Beautifulsoup,我正试图从每一个评论员给一个产品的星星数中剔除。我注意到有些评论员是“藤蔓之声”或付费评论员。他们很少给出4颗星,大部分是5颗星。因此,我想排除它们 我这样做的方式是,如果评论标记为“a-color-success a-text-bold”,则将其标记为“付费”或“未付费” 我似乎无法在vine变量中添加任何“Paid”标记。为什么 只有那些由Vine Voice撰写的评论才有标签,那些没有“付费”标签的评论才没有标签 这就是我目前得到的。 评论2和评论3是Vine Voice,但它们被标记为未
0 5.0 out of 5 stars 2019-09-18 Not-paid
1 4.0 out of 5 stars 2019-09-13 Not-paid
2 5.0 out of 5 stars 2019-09-12 Not-paid
3 5.0 out of 5 stars 2019-09-11 Not-paid
4 5.0 out of 5 stars 2019-09-10 Not-paid
...
将元素与元素进行比较,这就是为什么它总是处于else状态的原因。 我已经做了更改,并将文本与文本进行了比较,它工作正常。请检查下面的代码
import requests
from bs4 import BeautifulSoup
import pandas as pd
import time
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36'}
rating_list = []
date_list = []
vine = []
for num in range(1,12):
url = "https://www.amazon.com/Jabra-Wireless-Noise-Canceling-Headphones-Built/product-reviews/B07RS8B5HV/ref=cm_cr_arp_d_paging_btm_next_2?ie=UTF8&reviewerType=all_reviews&pageNumber={}&sortBy=recent".format(num)
r = requests.get(url, headers = headers)
soup = BeautifulSoup(r.content, 'lxml')
for ratings in soup.find_all("div", attrs={"data-hook": "review"}):
submission_date = ratings.find("span", {'data-hook':'review-date'}).text
rating = ratings.find('i', attrs={"data-hook": "review-star-rating"}).text
paid = ratings.find("span", attrs={"class": "a-color-success a-text-bold"})
if paid:
if paid.text in ratings.text:
vine.append("Paid")
date_list.append(submission_date)
rating_list.append(rating)
data = {'Rating': rating_list, 'Date': date_list, "Paid": vine}
else:
vine.append("Not-paid")
date_list.append(submission_date)
rating_list.append(rating)
data = {'Rating':rating_list, 'Date':date_list, "Paid":vine}
time.sleep(2)
df = pd.DataFrame(data)
df["Date"] = pd.to_datetime(df["Date"])
df = df.sort_values(by="Date", ascending=False)
print(df)
输出:
Date Paid Rating
0 2019-09-18 Not-paid 5.0 out of 5 stars
1 2019-09-13 Not-paid 4.0 out of 5 stars
2 2019-09-12 Paid 5.0 out of 5 stars
3 2019-09-11 Paid 5.0 out of 5 stars
4 2019-09-10 Not-paid 5.0 out of 5 stars
5 2019-09-10 Not-paid 2.0 out of 5 stars
6 2019-09-10 Paid 5.0 out of 5 stars
7 2019-09-09 Paid 5.0 out of 5 stars
8 2019-09-09 Not-paid 2.0 out of 5 stars
9 2019-09-08 Paid 5.0 out of 5 stars
10 2019-09-05 Paid 5.0 out of 5 stars
11 2019-09-01 Not-paid 2.0 out of 5 stars
12 2019-08-31 Paid 5.0 out of 5 stars
13 2019-08-25 Paid 5.0 out of 5 stars
14 2019-08-24 Not-paid 4.0 out of 5 stars
15 2019-08-22 Not-paid 5.0 out of 5 stars
16 2019-08-21 Paid 5.0 out of 5 stars
17 2019-08-20 Not-paid 5.0 out of 5 stars
18 2019-08-20 Paid 5.0 out of 5 stars
19 2019-08-18 Paid 5.0 out of 5 stars
20 2019-08-17 Not-paid 5.0 out of 5 stars
21 2019-08-17 Not-paid 5.0 out of 5 stars
22 2019-08-14 Not-paid 4.0 out of 5 stars
23 2019-08-12 Paid 5.0 out of 5 stars
24 2019-08-05 Paid 5.0 out of 5 stars
25 2019-08-05 Paid 4.0 out of 5 stars
26 2019-08-04 Paid 5.0 out of 5 stars
27 2019-08-04 Paid 4.0 out of 5 stars
29 2019-08-03 Paid 5.0 out of 5 stars
28 2019-08-03 Paid 4.0 out of 5 stars
.. ... ... ...
80 2019-07-08 Paid 5.0 out of 5 stars
81 2019-07-08 Paid 5.0 out of 5 stars
82 2019-07-08 Paid 5.0 out of 5 stars
85 2019-07-07 Paid 5.0 out of 5 stars
83 2019-07-07 Paid 5.0 out of 5 stars
84 2019-07-07 Paid 5.0 out of 5 stars
87 2019-07-06 Paid 5.0 out of 5 stars
86 2019-07-06 Paid 4.0 out of 5 stars
88 2019-07-05 Not-paid 4.0 out of 5 stars
89 2019-07-05 Paid 5.0 out of 5 stars
90 2019-07-05 Paid 5.0 out of 5 stars
91 2019-07-05 Paid 5.0 out of 5 stars
92 2019-07-04 Paid 5.0 out of 5 stars
93 2019-07-04 Paid 4.0 out of 5 stars
94 2019-07-04 Paid 5.0 out of 5 stars
95 2019-07-04 Paid 5.0 out of 5 stars
96 2019-07-04 Paid 5.0 out of 5 stars
98 2019-07-03 Not-paid 3.0 out of 5 stars
97 2019-07-03 Paid 5.0 out of 5 stars
99 2019-07-01 Paid 5.0 out of 5 stars
100 2019-07-01 Paid 3.0 out of 5 stars
101 2019-07-01 Paid 5.0 out of 5 stars
102 2019-06-30 Paid 5.0 out of 5 stars
103 2019-06-29 Paid 5.0 out of 5 stars
104 2019-06-29 Paid 5.0 out of 5 stars
105 2019-06-28 Not-paid 1.0 out of 5 stars
106 2019-06-27 Paid 4.0 out of 5 stars
107 2019-06-27 Paid 5.0 out of 5 stars
108 2019-06-26 Paid 5.0 out of 5 stars
109 2019-06-26 Paid 5.0 out of 5 stars
[110 rows x 3 columns]
我认为更好的方法(使用bs4.7.1+)是使用:has和:而不是预先进行排除。这样就不需要exclude字段/标志。在下面的示例中,我打印了审阅者姓名作为一个视觉检查(您将看到付费审阅者姓名不会出现)。我还将调整循环以使其正常工作,并使用会话
提高效率。我还使用更短更健壮的选择器
css选择器比find
更快,因此我可能会将find
行更改为:
submission_date = review.select_one('[data-hook=review-date]').text
rating = review.select_one('[data-hook=review-star-rating]').text
派克
不需要工作。但我也注意到“如果支付”缩进发生了什么?如果元素出现在父标记下,那么检查内部的if item文本val或转到else,这就是如果支付:全部。但是我运行了代码,它给了我未支付和支付选项。奇怪。我不知道为什么我的电脑在窃听。与我自己的代码相比,我只得到了20条评论,都是非付费的。我自己的代码给了我所有110条评论。你添加了输出,它看起来是正确的。我买的是20排的。不知道为什么我们得到如此不同的结果?我做到了。谢谢你的帮助。我想故障出在键盘语言上。所以当我手动输入你的代码时,它工作了!我对这个伪选择器非常敬畏。
:has()。该脚本能够绕过任何形式的禁令,除了最初出现的验证码。它能够抓取几乎所有的网站,只要内容是静态的,而不是谷歌。总有一天我会把它寄给你的。@SIM我期待着看到它。很高兴看到您使用:has。看起来很有趣。明天将对此进行测试,并将报告!
submission_date = review.select_one('[data-hook=review-date]').text
rating = review.select_one('[data-hook=review-star-rating]').text
import requests
from bs4 import BeautifulSoup
import pandas as pd
headers = {'User-Agent': 'Mozilla/5.0'}
rating_list = []
date_list = []
with requests.Session() as s:
for num in range(1,12):
url = "https://www.amazon.com/Jabra-Wireless-Noise-Canceling-Headphones-Built/product-reviews/B07RS8B5HV/ref=cm_cr_arp_d_paging_btm_next_2?ie=UTF8&reviewerType=all_reviews&pageNumber={}&sortBy=recent".format(num)
r = s.get(url, headers = headers)
soup = BeautifulSoup(r.content, 'lxml')
for review in soup.select('.review:not(:has(.a-color-success))'):
submission_date = review.select_one('[data-hook=review-date]').text
rating = review.select_one('[data-hook=review-star-rating]').text
date_list.append(submission_date)
rating_list.append(rating)
print(review.select_one('.a-profile-name').text) #check
data = {'Rating':rating_list, 'Date':date_list}
df = pd.DataFrame(data)
df["Date"] = pd.to_datetime(df["Date"])
df = df.sort_values(by="Date", ascending=False)
print(df)