Python MapReduce如何添加条件语句
我是MapReduce的新手,我试图在MovieLens 100k数据集中找到电影的平均电影评论。我有一个工作程序,可以找到每部电影的平均评论,但我只想对评论超过100的电影这样做。如何添加条件语句来执行此操作Python MapReduce如何添加条件语句,python,if-statement,hadoop,mapreduce,Python,If Statement,Hadoop,Mapreduce,我是MapReduce的新手,我试图在MovieLens 100k数据集中找到电影的平均电影评论。我有一个工作程序,可以找到每部电影的平均评论,但我只想对评论超过100的电影这样做。如何添加条件语句来执行此操作 from mrjob.job import MRJob class PopularMovieAvgReview(MRJob): def mapper(self, key, line): (userID, movieID, rating, timestamp) =
from mrjob.job import MRJob
class PopularMovieAvgReview(MRJob):
def mapper(self, key, line):
(userID, movieID, rating, timestamp) = line.split('\t')
yield movieID, float(rating)
def reducer(self, movieID, rating):
total = 0
numElements = 0
for x in rating:
total += x
numElements += 1
yield movieID, total / numElements
if __name__ == '__main__':
PopularMovieAvgReview.run()
如果我理解正确,您希望根据给定的评级数量限制输出
def reducer(self, movieID, rating):
total = 0
numElements = 0
for x in rating:
total += x
numElements += 1
if numElements > 100:
yield movieID, total / numElements
或者,使用PySpark,您可以聚合,然后过滤评级金额