Hadoop:跟踪本地作业的进度
我目前正在尝试做类似的事情。不同之处在于,我的目标作业是本地作业(这是在单元测试中运行的)。以下是我正在做的事情(我对mapProgress感兴趣): 我认为JobClient不是为了获取本地工作的信息而设计的。有没有一种方法可以为本地运行的作业执行类似的操作Hadoop:跟踪本地作业的进度,hadoop,Hadoop,我目前正在尝试做类似的事情。不同之处在于,我的目标作业是本地作业(这是在单元测试中运行的)。以下是我正在做的事情(我对mapProgress感兴趣): 我认为JobClient不是为了获取本地工作的信息而设计的。有没有一种方法可以为本地运行的作业执行类似的操作 final String job = this.config.get("mapreduce.job.id"); final JobClient client = new JobClient(this.con
final String job = this.config.get("mapreduce.job.id");
final JobClient client = new JobClient(this.config);
final RunningJob running = client.getJob(JobID.forName(job));
if (running == null) {
Logger.warn(
this, String.format("Job id does not exist: %s", job)
);
throw new IllegalStateException(
String.format("No running job found: %s", job)
);
}
while (running.mapProgress() < 1.0) {
TimeUnit.SECONDS.sleep(1);
}
[←[0;33mWARN←[m] Thread-6 org.apache.hadoop.mapred.LocalJobRunner: job_local204744704_0002
java.lang.Exception: java.lang.IllegalStateException: No running job found: job_local204744704_0002
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.lang.IllegalStateException: No running job found: job_local204744704_0002
at com.xockets.hadoop.XsInput.run(XsInput.java:150)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)