Hadoop映射任务初始化花费的时间太长

最近，在我运行的任何Hadoop流中，我都会遇到某个映射节点（主节点作为从节点工作）有3分钟10秒的延迟。在初始化延迟之后，它会恢复正常并立即执行

例如，运行QuasiMonteCarlo时，例如：

Task Id                                 Start Time  Finish Time <br>
attempt_201204101957_0006_m_000003_0    10/04 20:14:54  10/04 20:18:05 (3mins, 10sec)   /default-rack/master

2012-04-10 20:18:04,470 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library<br>2012-04-10 20:18:04,646 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=<br>
2012-04-10 20:18:04,647 WARN org.apache.hadoop.conf.Configuration: user.name is deprecated. Instead, use mapreduce.job.user.name<br>
2012-04-10 20:18:04,751 INFO org.apache.hadoop.mapreduce.util.ProcessTree: setsid exited with exit code 0<br>
2012-04-10 20:18:04,754 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@79ee2c2c<br>
2012-04-10 20:18:04,912 INFO org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)<br>
2012-04-10 20:18:04,912 INFO org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100<br>
2012-04-10 20:18:04,912 INFO org.apache.hadoop.mapred.MapTask: soft limit at 83886080<br>
2012-04-10 20:18:04,912 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600<br>
2012-04-10 20:18<br>:04,912 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600
2012-04-10 20:18:04,939 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output<br>
2012-04-10 20:18:04,940 INFO org.apache.hadoop.mapred.MapTask: Spilling map output<br>
2012-04-10 20:18:04,940 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 18; bufvoid = 104857600<br>
2012-04-10 20:18:04,940 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600<br>
2012-04-10 20:18:04,972 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0<br>
2012-04-10 20:18:04,975 INFO org.apache.hadoop.mapred.Task: Task:attempt_201204101957_0006_m_000003_0 is done. And is in the process of commiting<br>
2012-04-10 20:18:05,058 INFO org.apache.hadoop.mapred.Task: Task 'attempt_201204101957_0006_m_000003_0' done.<br>

任务Id开始时间完成时间

尝试\u 201204101957\u 0006\u m\u000003\u 0 10/04 20:14:54 10/04 20:18:05（3分钟10秒）/默认机架/主机
2012-04-10 20:18:04470 INFO org.apache.hadoop.util.NativeCodeLoader:加载本机hadoop库
2012-04-10 20:18:04646 INFO org.apache.hadoop.metrics.JvmMetrics:使用processName=MAP、sessionId=

2012-04-10 20:18:04647警告org.apache.hadoop.conf.Configuration:user.name已弃用。而是使用mapreduce.job.user.name

2012-04-10 20:18:04751 INFO org.apache.hadoop.mapreduce.util.ProcessTree:setsid已退出，退出代码为0

2012-04-10 20:18:04754 INFO org.apache.hadoop.mapred.Task:使用ResourceCalculatorPlugin:org.apache.hadoop.mapreduce.util。LinuxResourceCalculatorPlugin@79ee2c2c

2012-04-10 20:18:04912 INFO org.apache.hadoop.mapred.MapTask：（赤道）0千伏26214396（104857584）

2012-04-10 20:18:04912 INFO org.apache.hadoop.mapred.MapTask:mapreduce.task.io.sort.mb:100

2012-04-10 20:18:04912 INFO org.apache.hadoop.mapred.MapTask：83886080的软限制

2012-04-10 20:18:04912 INFO org.apache.hadoop.mapred.MapTask:bufstart=0；bufvoid=104857600

2012-04-10 20:18
：04912 INFO org.apache.hadoop.mapred.MapTask:kvstart=26214396；长度=6553600
2012-04-10 20:18:04939 INFO org.apache.hadoop.mapred.MapTask:开始刷新地图输出

2012-04-10 20:18:04940 INFO org.apache.hadoop.mapred.MapTask：溢出映射输出

2012-04-10 20:18:04940 INFO org.apache.hadoop.mapred.MapTask:bufstart=0；bufend=18；bufvoid=104857600

2012-04-10 20:18:04940 INFO org.apache.hadoop.mapred.MapTask:kvstart=26214396（104857584）；kvend=26214392（104857568）；长度=5/6553600

2012-04-10 20:18:04972 INFO org.apache.hadoop.mapred.MapTask:Finished spill 0

2012-04-10 20:18:04975 INFO org.apache.hadoop.mapred.Task：任务：完成尝试。并且正在提交

2012-04-10 20:18:05058 INFO org.apache.hadoop.mapred.Task：任务“尝试”完成。

任务跟踪日志更能说明问题：

2012-04-10 **20:14:54,615** INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 1 and trying to launch attempt_201204101957_0006_m_000003_0 which needs 1 slots<br>
2012-04-10 20:14:54,685 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204101957_0006_m_377512887 spawned.<br>
2012-04-10 20:16:34,041 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 1<br>
2012-04-10 **20:18:04,433** INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204101957_0006_m_377512887 given task: attempt_201204101957_0006_m_000003_0<br>
2012-04-10 20:18:04,938 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204101957_0006_m_000003_0 0.0% <br>
2012-04-10 20:18:05,056 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204101957_0006_m_000003_0 0.667% Generated 1000 samples. <br>

2012-04-10**20:14:54615**INFO org.apache.hadoop.mapred.TaskTracker:在TaskLauncher中，当前可用插槽：1，正在尝试启动需要1个插槽的尝试\u 201204101957\u 0006\u m\u000003\u 0

2012-04-10 20:14:54685 INFO org.apache.hadoop.mapred.JvmManager:JVM运行程序JVM_201204101957_0006_m_377512887生成。

2012-04-10 20:16:34041 INFO org.apache.hadoop.mapred.TaskTracker:addFreeSlot:current free slot:1

2012-04-10**20:18:04433**INFO org.apache.hadoop.mapred.TaskTracker:ID为JVM的JVM\u 201204101957\u 0006\u m\u 377512887给定任务：尝试\u 201204101957\u 0006\u m\u 000003\u 0

2012-04-10 20:18:04938 INFO org.apache.hadoop.mapred.TaskTracker:trust_201204101957_0006_m_000003_0.0%

2012-04-10 20:18:05056 INFO org.apache.hadoop.mapred.TaskTracker:trust_201204101957_0006_m_000003_0.667%生成了1000个样本

排序
2012-04-10 20:18:05058 INFO org.apache.hadoop.mapred.TaskTracker:任务尝试\u 201204101957\u 0006\u m\u000003\u 0已完成。
2012-04-10 20:18:05058 INFO org.apache.hadoop.mapred.TaskTracker:尝试的报告输出大小为28
2012-04-10 20:18:05058 INFO org.apache.hadoop.mapred.TaskTracker:addFreeSlot:current free slot:2
2012-04-10 20:18:05213 INFO org.apache.hadoop.mapreduce.util.ProcessTree:向进程组的所有成员发送信号-23030:SIGTERM。退出代码1
2012-04-10 20:18:08478 INFO org.apache.hadoop.mapred.TaskTracker:发送了28个字节以从映射中减少0:trust_201204101957_0006_m_000003_0给定28/24
2012-04-10 20:18:08478 INFO org.apache.hadoop.mapred.TaskTracker:Shuffled 1map（mapIds=trust_201204101957_0006_m_000003_0）在29秒内减少0
2012-04-10 20:18:08478 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace:src:147.102.7.173:50060，dest:147.102.7.175:57289，maps:1，op:mapred_shufle，reduceID:0，duration:29
2012-04-10 20:18:10217 INFO org.apache.hadoop.mapred.JvmManager:JVM:JVM_201204101957_0006_m_377512887已退出，退出代码为0。它运行的任务数：1

---

我怀疑这里有网络问题，但我可以ping和ssh，没有问题。

将master作为slave

运行任务跟踪器和name节点（在同一个节点上）吗。此外，您的日志文件显示从开始到结束的时间刚好超过每秒1/2。节点的tasktracker日志显示了什么？更新了问题
我总共有三个节点，主节点都是jobtracker/namenode/datanode。问题在于初始化所花费的时间超过了它应该花费的时间，如问题中的日志所示。它可能是JVM的东西吗？我是否也应该获得堆栈跟踪？您使用的是哪个版本的hadoop？