Python 获取Hadoop Mapreduce中出现的最大字数字数
因此,我一直在关注这个网站上的Mapreduce python代码,它从文本文件返回一个单词计数,即单词及其在文本中出现的次数。但是,我想知道如何返回出现的最大单词数。映射器和还原器如下所示:Python 获取Hadoop Mapreduce中出现的最大字数字数,python,hadoop,mapreduce,hadoop-streaming,Python,Hadoop,Mapreduce,Hadoop Streaming,因此,我一直在关注这个网站上的Mapreduce python代码,它从文本文件返回一个单词计数,即单词及其在文本中出现的次数。但是,我想知道如何返回出现的最大单词数。映射器和还原器如下所示: #Mapper import sys # input comes from STDIN (standard input) for line in sys.stdin: # remove leading and trailing whitespace line = line.strip(
#Mapper
import sys
# input comes from STDIN (standard input)
for line in sys.stdin:
# remove leading and trailing whitespace
line = line.strip()
# split the line into words
words = line.split()
# increase counters
for word in words:
# write the results to STDOUT (standard output);
# what we output here will be the input for the
# Reduce step, i.e. the input for reducer.py
#
# tab-delimited; the trivial word count is 1
print '%s\t%s' % (word, 1)
#Reducer
from operator import itemgetter
import sys
current_word = None
current_count = 0
word = None
# input comes from STDIN
for line in sys.stdin:
# remove leading and trailing whitespace
line = line.strip()
# parse the input we got from mapper.py
word, count = line.split('\t', 1)
# convert count (currently a string) to int
try:
count = int(count)
except ValueError:
# count was not a number, so silently
# ignore/discard this line
continue
# this IF-switch only works because Hadoop sorts map output
# by key (here: word) before it is passed to the reducer
if current_word == word:
current_count += count
else:
if current_word:
# write result to STDOUT
print '%s\t%s' % (current_word, current_count)
current_count = count
current_word = word
# do not forget to output the last word if needed!
if current_word == word:
print '%s\t%s' % (current_word, current_count)
因此,我知道我需要在减速机的末尾添加一些内容,但我不完全确定是什么。您只需要设置一个减速机来聚合所有值—numreductasks 1 您的reduce应该是这样的:
max_count = 0
max_word = None
for line in sys.stdin:
# remove leading and trailing whitespace
line = line.strip()
# parse the input we got from mapper.py
word, count = line.split('\t', 1)
# convert count (currently a string) to int
try:
count = int(count)
except ValueError:
# count was not a number, so silently
# ignore/discard this line
continue
# this IF-switch only works because Hadoop sorts map output
# by key (here: word) before it is passed to the reducer
if current_word == word:
current_count += count
else:
# check if new word greater
if current_count > max_count:
max_count= current_count
max_word = current_word
current_count = count
current_word = word
# do not forget to check last word if needed!
if current_count > max_count:
max_count= current_count
max_word = current_word
print '%s\t%s' % (max_word, max_count)
但是由于只有一个减速器,您会失去并行化,因此如果在第一个减速器之后运行此作业,可能会更快,而不是相反。这样,您的映射器将与reducer相同。因此,您只需找到计数最大的单词,并准确地输出它。计数最大的单词和计数本身。我猜在减缩器的末尾有一点代码要添加,但我尝试了无效。