Perl 为什么我的减速机坏了?(Hadoop)
所以我写了两个Perl脚本来练习MapReduce。这个程序应该计算我放在一个目录中的一堆文本文件中的所有单词 这是我的mapper.plPerl 为什么我的减速机坏了?(Hadoop),perl,apache,hadoop,mapreduce,Perl,Apache,Hadoop,Mapreduce,所以我写了两个Perl脚本来练习MapReduce。这个程序应该计算我放在一个目录中的一堆文本文件中的所有单词 这是我的mapper.pl #!/usr/bin/perl use 5.010; use strict; use warnings; while(my $line = <>) { my @words = split(' ', $line); foreach my $word(@words) { print "$word \t 1\n";
#!/usr/bin/perl
use 5.010;
use strict;
use warnings;
while(my $line = <>) {
my @words = split(' ', $line);
foreach my $word(@words) {
print "$word \t 1\n";
}
}
我得到以下错误:
13/07/19 11:36:33 INFO streaming.StreamJob: map 0% reduce 0%
13/07/19 11:36:39 INFO streaming.StreamJob: map 9% reduce 0%
13/07/19 11:36:40 INFO streaming.StreamJob: map 64% reduce 0%
13/07/19 11:36:41 INFO streaming.StreamJob: map 73% reduce 0%
13/07/19 11:36:44 INFO streaming.StreamJob: map 82% reduce 0%
13/07/19 11:36:45 INFO streaming.StreamJob: map 100% reduce 0%
13/07/19 11:36:49 INFO streaming.StreamJob: map 100% reduce 11%
13/07/19 11:36:53 INFO streaming.StreamJob: map 100% reduce 0%
13/07/19 11:37:02 INFO streaming.StreamJob: map 100% reduce 17%
13/07/19 11:37:03 INFO streaming.StreamJob: map 100% reduce 33%
13/07/19 11:37:06 INFO streaming.StreamJob: map 100% reduce 17%
13/07/19 11:37:08 INFO streaming.StreamJob: map 100% reduce 0%
13/07/19 11:37:16 INFO streaming.StreamJob: map 100% reduce 33%
13/07/19 11:37:21 INFO streaming.StreamJob: map 100% reduce 0%
13/07/19 11:37:31 INFO streaming.StreamJob: map 100% reduce 33%
13/07/19 11:37:35 INFO streaming.StreamJob: map 100% reduce 17%
13/07/19 11:37:38 INFO streaming.StreamJob: map 100% reduce 100%
13/07/19 11:37:38 INFO streaming.StreamJob: To kill this job, run:
13/07/19 11:37:38 INFO streaming.StreamJob: /usr/local/hadoop/libexec/../bin/hadoop job -Dmapred.job.tracker=shiv0:54311 -kill job_201307031312_0065
13/07/19 11:37:38 INFO streaming.StreamJob: Tracking URL: http://shiv0:50030/jobdetails.jsp?jobid=job_201307031312_0065
13/07/19 11:37:38 ERROR streaming.StreamJob: Job not successful. Error: # of failed Reduce Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201307031312_0065_r_000001
13/07/19 11:37:38 INFO streaming.StreamJob: killJob... Streaming Command Failed!
一段时间以来,我一直在试图找出我做错了什么,我一直在挠头。有人对我如何诊断这个问题有什么建议吗?bin/hadoop-jar contrib/streaming/hadoop-streaming-1.1.2.jar-file/home/hduser/countWords/mapper.py-mapper/home/hduser/countWords/mapper.py-file/home/hduser/countWords/reducer.py-reducer/home/hduser/countWords/reducer.py-input/user/hduser/testData/*-output/用户/hduser/testData/output/*
为什么要调用.py文件?你不应该调用perl文件,即reducer.pl而不是reducer.py吗?我犯了一个极其愚蠢的错误。。reducer.pl的shbang行不正确。我有
#!/bin/usr/perl
而不是
#!/usr/bin/perl
您的命令行显示了
mapper.py
和reducer.py
。是的,对不起,我刚刚修复了它。我试图运行一个python版本的程序,我在网上找到了这个程序是的,这是我的错。我试图运行一个在python online中找到的单词计数应用程序,看看它是否也能运行。我在上面修好了
#!/bin/usr/perl
#!/usr/bin/perl