Java Hadoop Map Reduce wordcount Shuffle错误：超过最大值\u失败\u唯一\u获取；纾困_Java_Hadoop

Java Hadoop Map Reduce wordcount Shuffle错误：超过最大值\u失败\u唯一\u获取；纾困

java hadoop

Java Hadoop Map Reduce wordcount Shuffle错误：超过最大值\u失败\u唯一\u获取；纾困,java,hadoop,Java,Hadoop,我已经使用manul从以下站点安装并配置hadoop作为单节点我已经编译了wordcount示例并运行了它，但它需要很长的时间并生成Shuffle错误：超过了MAX\u FAILED\u UNIQUE\u FETCHES；纾困。 hduser@aptunix0043:/usr/local/hadoop/src$ hadoop jar WordCount.jar org/apache/hadoop/examples/WordCount input ot ****hdfs://localho

我已经使用manul从以下站点安装并配置hadoop作为单节点

我已经编译了wordcount示例并运行了它，但它需要很长的时间并生成

Shuffle错误：超过了MAX\u FAILED\u UNIQUE\u FETCHES；纾困。

hduser@aptunix0043:/usr/local/hadoop/src$ hadoop jar WordCount.jar org/apache/hadoop/examples/WordCount input  ot

****hdfs://localhost:54310/user/hduser/input
12/07/03 02:52:35 INFO input.FileInputFormat: Total input paths to process : 1
12/07/03 02:52:36 INFO mapred.JobClient: Running job: job_201207030248_0002
12/07/03 02:52:37 INFO mapred.JobClient:  map 0% reduce 0%
12/07/03 02:52:52 INFO mapred.JobClient:  map 100% reduce 0%
12/07/03 03:21:26 INFO mapred.JobClient: Task Id :attempt_201207030248_0002_r_000000_0, Status : FAILED 
Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.

12/07/03 03:21:47 WARN mapred.JobClient: Error reading task outputConnection timed out
12/07/03 03:22:08 WARN mapred.JobClient: Error reading task outputConnection timed out
 /user/hduser/input/*12/07/03 03:50:01 INFO mapred.JobClient: Task Id :      attempt_201207030248_0002_r_000000_1, Status : FAILED
Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
12/07/03 03:50:22 WARN mapred.JobClient: Error reading task outputConnection timed out
12/07/03 03:50:43 WARN mapred.JobClient: Error reading task outputConnection timed out
12/07/03 04:18:35 INFO mapred.JobClient: Task Id :  attempt_201207030248_0002_r_000000_2, Status : FAILED
Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
12/07/03 04:18:56 WARN mapred.JobClient: Error reading task outputConnection timed out
12/07/03 04:19:17 WARN mapred.JobClient: Error reading task outputConnection timed out
12/07/03 04:47:15 INFO mapred.JobClient: Job complete: job_201207030248_0002
12/07/03 04:47:15 INFO mapred.JobClient: Counters: 23
12/07/03 04:47:15 INFO mapred.JobClient:   Job Counters
12/07/03 04:47:15 INFO mapred.JobClient:     Launched reduce tasks=4
12/07/03 04:47:15 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=12948
12/07/03 04:47:15 INFO mapred.JobClient:     Total time spent by all reduces waiting  after reserving slots (ms)=0
12/07/03 04:47:15 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/07/03 04:47:15 INFO mapred.JobClient:     Launched map tasks=1
12/07/03 04:47:15 INFO mapred.JobClient:     Data-local map tasks=1
12/07/03 04:47:15 INFO mapred.JobClient:     Failed reduce tasks=1
12/07/03 04:47:15 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=16469
12/07/03 04:47:15 INFO mapred.JobClient:   FileSystemCounters
12/07/03 04:47:15 INFO mapred.JobClient:     HDFS_BYTES_READ=661744
12/07/03 04:47:15 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=288616
12/07/03 04:47:15 INFO mapred.JobClient:   File Input Format Counters
12/07/03 04:47:15 INFO mapred.JobClient:     Bytes Read=661630
12/07/03 04:47:15 INFO mapred.JobClient:   Map-Reduce Framework
12/07/03 04:47:15 INFO mapred.JobClient:     Map output materialized bytes=267085
12/07/03 04:47:15 INFO mapred.JobClient:     Combine output records=18040
12/07/03 04:47:15 INFO mapred.JobClient:     Map input records=12761
12/07/03 04:47:15 INFO mapred.JobClient:     Physical memory (bytes) snapshot=183209984
12/07/03 04:47:15 INFO mapred.JobClient:     Spilled Records=18040
12/07/03 04:47:15 INFO mapred.JobClient:     Map output bytes=1086716
12/07/03 04:47:15 INFO mapred.JobClient:     CPU time spent (ms)=1940
12/07/03 04:47:15 INFO mapred.JobClient:     Total committed heap usage  (bytes)=162856960
12/07/03 04:47:15 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=393482240
12/07/03 04:47:15 INFO mapred.JobClient:     Combine input records=109844
12/07/03 04:47:15 INFO mapred.JobClient:     Map output records=109844
12/07/03 04:47:15 INFO mapred.JobClient:     SPLIT_RAW_BYTES=114

有什么线索吗？

我以前有过这个错误，它是由DNS问题引起的。您是否在基于Linux的发行版中运行？如果是这样，请确保所有/etc/hosts都是同步的，在我的例子中，我为每个节点“slave1192.168.1.23”等使用了一个别名。。。但这与框名不匹配，所以我不得不更改它，或者您可以更改框名以匹配hadoop配置中对应于“奴隶”的框名。

为了方便像我一样在internet上搜索并访问此页面的用户，您可能会遇到两个问题

DNS解析-确保在安装hadoop时为每个主机使用完全限定的域名

防火墙-防火墙可能会根据您的hadoop发行版阻止端口5006050030和其他几个端口（cloudera为71827180）

你检查过tasktracker的日志了吗？它包含更多信息。我是hadoop新手。所以我不知道是什么造成了这个问题。您可以指定我可以查看日志文件的任何术语/短语。打开web浏览器并输入以下URL以访问您的JobTracker:，从那里您应该可以找到失败的作业（搜索作业id job_20120703248_0002），单击超链接以查看作业详细信息，然后单击表示任务失败的超链接，然后单击日志链接-最后将日志粘贴到pastebin或原始问题中