Warning: file_get_contents(/data/phpspider/zhask/data//catemap/9/java/364.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Java 即使在EMR作业完成时,获取EMR结果的调用也始终返回空状态集合_Java_Amazon Web Services_Mapreduce - Fatal编程技术网

Java 即使在EMR作业完成时,获取EMR结果的调用也始终返回空状态集合

Java 即使在EMR作业完成时,获取EMR结果的调用也始终返回空状态集合,java,amazon-web-services,mapreduce,Java,Amazon Web Services,Mapreduce,由于IP的原因,我无法发布完整的源代码。然而,我打电话提交了一个Amazon弹性地图缩减作业(EMR),该作业现在已经运行完成。以前它失败了,基本上是一个找不到文件的错误 RunJobFlowResult result=emr.runJobFlow(request); 成功,我可以从中获取作业流ID 稍后,我将循环轮询第一个用户的状态 DescribeJobFlowsRequest请求=新的DescribeJobFlowsRequest(jobFlowIdArray); 我通过调用 reque

由于IP的原因,我无法发布完整的源代码。然而,我打电话提交了一个Amazon弹性地图缩减作业(EMR),该作业现在已经运行完成。以前它失败了,基本上是一个找不到文件的错误

RunJobFlowResult result=emr.runJobFlow(request);
成功,我可以从中获取作业流ID

稍后,我将循环轮询第一个用户的状态 DescribeJobFlowsRequest请求=新的DescribeJobFlowsRequest(jobFlowIdArray); 我通过调用 request.getJobFlowState()

不幸的是,无论作业是否正在运行、失败或成功,该调用始终返回空集合。我怎样才能至少知道发生了什么

AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
AmazonElasticMapReduceClient client = new AmazonElasticMapReduceClient(credentials);
client.setEndPoint("elasticmapreduce.us-east-1.amazonaws.com");

StepFactory stepFactory = new StepFactory();
StepConfig enableDebugging = new StepConfig()
.withActionOnFailure("TERMINATE_JOB_FLOW")
.withHadoopjJarStep(stepFactory.newEnableDebuggingStep());


String[] arguments={...} // Custom jar arguments

HadoopJarStepConfig jarConfig = new HadoopJarStepConfig();
jarConfig.setJar(JAR_NAME);
jarConfig.setArgs(Arrays.asList(arguments));

StepConfig runJar = new     StepConfig(JAR_NAME.substring(JAR_NAME.indexOf('/')+1),jarConfig);

RunJobFlowRequest request = new RunJobFlowRequest()
.withName("...")
.withSteps(runJar)
.withLogUri("...")
.withInstances(
     new JobFlowInstancesCOnfig()
       .withHadoopVersion("1.0.3")
        .withInstanceCount(5)
         .withKeepJobFlowAliveWhenNoSteps(false)
         .withMasterInstanceType("m1.small")
          .withSlaveInstanceType("m1.small");

RunJobFlowResult result = client.runJobFlow(request);
String jobFlowID=result.getJobFlowID();
List<String> describeJobFlowIdList=new ArrayList<String>(1);
describeJobFlowIdList.add(jobFlowID);

String lastState="";
boolean jobMonitoringNotDone=true;
while(jobMonitoringNotDone){
    SescribeJobFlowsRequest describeJobFlowsRequest=
        new DescribeJobFlowsRequest(describeJobFlowIdList);
    // Call to describeJobFlowsRequest.getJobFlowStates() always returns
    // empty list even when job succeeds or fails.
    for(String state : describeJobFlowsRequest.getJobFlowStates()){
        if(DONE_STATES.contains(state)){
            jobMonitoringNotDone=false;
        } else if(!lastState.equals(state)){
            lastState = state;
            System.out.println("Job "+state + " at "+ new Date().toString());
        }
    }
    try {
    Thread.sleep(10000);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}
AWSCredentials凭证=新的基本凭证(accessKey、secretKey);
AmazonElasticMapReduceClient=新AmazonElasticMapReduceClient(凭据);
client.setEndPoint(“elasticmapreduce.us-east-1.amazonaws.com”);
StepFactory StepFactory=新的StepFactory();
StepConfig enableDebugging=新的StepConfig()
.withActionOnFailure(“终止作业流”)
.withHadoopjJarStep(stepFactory.newEnableDebuggingStep());
字符串[]参数={…}//自定义jar参数
HadoopJarStepConfig jarConfig=新的HadoopJarStepConfig();
jarConfig.setJar(JAR_名称);
jarConfig.setArgs(Arrays.asList(arguments));
StepConfig runJar=newstepconfig(JAR\u NAME.substring(JAR\u NAME.indexOf('/')+1),jarConfig);
RunJobFlowRequest=新的RunJobFlowRequest()
.withName(“…”)
.withSteps(runJar)
.withLogUri(“…”)
.举例说明(
新的JobFlowInstanceConfig()
.WithHadoop版本(“1.0.3”)
.带实例计数(5)
.在关闭时保持作业流有效(false)
.withMasterInstanceType(“m1.small”)
.带有LaveInstanceType(“m1.small”);
RunJobFlowResult=client.runJobFlow(请求);
字符串jobFlowID=result.getJobFlowID();
List describeJobFlowIdList=新数组列表(1);
describeJobFlowIdList.add(jobFlowID);
字符串lastState=“”;
布尔值jobMonitoringNotDone=true;
while(作业监视未完成){
SescribeJobFlowsRequest描述作业流Request=
新建DescribeJobFlowRequest(describeJobFlowIdList);
//调用describeJobFlowsRequest.getJobFlowStates()始终返回
//即使作业成功或失败,列表也为空。
for(字符串状态:describeJobFlowsRequest.GetJobFlowsState()){
if(完成)_STATES.contains(状态)){
jobMonitoringNotDone=false;
}else如果(!lastState.equals(state)){
lastState=状态;
System.out.println(“作业”+状态+”位于“+新日期().toString());
}
}
试一试{
睡眠(10000);
}捕捉(中断异常e){
e、 printStackTrace();
}
}

上面的代码缺少类似的调用

DescribeJobFlowsResult describeJobFlowsResult =      client.describeJobFlows(describeJobFlowsRequest);

这给了我一个可行的解决方案,但不幸的是,Amazon不赞成这种方法,但没有提供替代方案。我希望我有一个不反对的解决方案,所以这只是一个部分答案。

你甚至不能创建一个最小功能的示例来重现你的问题吗?我对它进行了更新,以提供我试图做的事情的最小视图。