Hadoop 有没有办法找到为什么减速器被AM杀死

Hadoop 有没有办法找到为什么减速器被AM杀死,hadoop,mapreduce,yarn,Hadoop,Mapreduce,Yarn,我正在尝试使用1GB的输入在Hadoop上运行一些图形处理作业。但是我的reduce任务被应用程序主程序杀死了。 这是输出 14/06/29 16:15:02 INFO mapreduce.Job: map 100% reduce 53% 14/06/29 16:15:03 INFO mapreduce.Job: map 100% reduce 57% 14/06/29 16:15:04 INFO mapreduce.Job: map 100% reduce 60% 14/06/29 16

我正在尝试使用1GB的输入在Hadoop上运行一些图形处理作业。但是我的reduce任务被应用程序主程序杀死了。 这是输出

14/06/29 16:15:02 INFO mapreduce.Job:  map 100% reduce 53%
14/06/29 16:15:03 INFO mapreduce.Job:  map 100% reduce 57%
14/06/29 16:15:04 INFO mapreduce.Job:  map 100% reduce 60%
14/06/29 16:15:05 INFO mapreduce.Job:  map 100% reduce 63%
14/06/29 16:15:05 INFO mapreduce.Job: Task Id : attempt_1404050864296_0002_r_000003_0, Status : FAILED
Container killed on request. Exit code is 137
Container exited with a non-zero exit code 137
Killed by external signal
最后作业失败,输出如下

14/06/29 16:11:58 INFO mapreduce.Job: Job job_1404050864296_0001 failed with state FAILED due to: Task failed task_1404050864296_0001_r_000001
Job failed as tasks failed. failedMaps:0 failedReduces:1

14/06/29 16:11:58 INFO mapreduce.Job: Counters: 38
        File System Counters
                FILE: Number of bytes read=1706752372
                FILE: Number of bytes written=3414132444
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=1319319669
                HDFS: Number of bytes written=0
                HDFS: Number of read operations=30
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=0
        Job Counters
                Failed reduce tasks=7
                Killed reduce tasks=1
                Launched map tasks=10
                Launched reduce tasks=8
                Data-local map tasks=10
                Total time spent by all maps in occupied slots (ms)=12527776
                Total time spent by all reduces in occupied slots (ms)=1256256
                Total time spent by all map tasks (ms)=782986
                Total time spent by all reduce tasks (ms)=78516
                Total vcore-seconds taken by all map tasks=782986
                Total vcore-seconds taken by all reduce tasks=78516
                Total megabyte-seconds taken by all map tasks=6263888000
                Total megabyte-seconds taken by all reduce tasks=628128000
        Map-Reduce Framework
                Map input records=85331845
                Map output records=170663690
                Map output bytes=1365309520
                Map output materialized bytes=1706637020
                Input split bytes=980
                Combine input records=0
                Spilled Records=341327380
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=2573
                CPU time spent (ms)=820310
                Physical memory (bytes) snapshot=18048614400
                Virtual memory (bytes) snapshot=72212246528
                Total committed heap usage (bytes)=28289007616
        File Input Format Counters
                Bytes Read=1319318689
Exception in thread "main" java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836)
        at pegasus.DegDist.run(DegDist.java:201)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at pegasus.DegDist.main(DegDist.java:158)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

我已经检查了日志,但没有关于reduce任务被终止的信息。是否有办法找出此reduce任务被终止的原因。我对终止reduce作业的具体原因感兴趣。

您有一个IOException。看起来您用一条无用的“作业失败!”消息吞并了异常。将其替换为
e.printStackTrace()
,再次运行,然后查看结果。我正在考虑其他方法,而不仅仅是更改hadoop/Pegasus代码。无论如何,谢谢。