以编程方式在hadoop中获取总映射并减少时间
我试图通过MR代码中的所有任务计算映射、洗牌和减少的总次数 我需要帮助检索每个MapReduce作业的信息以编程方式在hadoop中获取总映射并减少时间,hadoop,mapreduce,mapper,reducers,Hadoop,Mapreduce,Mapper,Reducers,我试图通过MR代码中的所有任务计算映射、洗牌和减少的总次数 我需要帮助检索每个MapReduce作业的信息 有人可以发布任何进行该计算的代码片段吗?您需要使用JobClient API,如下所示: 然而,API有一些古怪之处。试试看,让我知道我会帮你的 JobClient client = null; Configuration configuration = new Configuration(); configuration.set("mapre
有人可以发布任何进行该计算的代码片段吗?您需要使用JobClient API,如下所示: 然而,API有一些古怪之处。试试看,让我知道我会帮你的
JobClient client = null;
Configuration configuration = new Configuration();
configuration.set("mapred.job.tracker", jobTrackerURL);
client = new JobClient(new JobConf(configuration));
while (true) {
List<JobStatus> jobEntries = getTrackerEntries(jobName,
client);
for (JobStatus jobStatus : jobEntries) {
JobID jobId = jobStatus.getJobID();
String trackerJobName = client.getJob(jobId)
.getJobName();
TaskReport[] mapReports = client
.getMapTaskReports(jobId);
TaskReport[] reduceReports = client
.getReduceTaskReports(jobId);
client.getJob(jobId).getJobStatus().getStartTime();
int jobMapper = mapReports.length;
mapNumber = +jobMapper;
int jobReducers = reduceReports.length;
reduceNumber = +jobReducers;
}
}
JobClient=null;
配置=新配置();
set(“mapred.job.tracker”,jobTrackerURL);
client=newjobclient(newjobconf(配置));
while(true){
List jobEntries=getTrackerEntries(jobName,
客户);
对于(JobStatus JobStatus:jobEntries){
JobID JobID=jobStatus.getJobID();
字符串trackerJobName=client.getJob(jobId)
.getJobName();
TaskReport[]mapReports=客户端
.getMapTaskReports(作业ID);
TaskReport[]reduceReports=客户端
.getReduceTaskReports(作业ID);
getJob(jobId).getJobStatus().getStartTime();
int jobMapper=mapReports.length;
mapNumber=+作业映射器;
int jobReducers=reduceReports.length;
reduceEnumber=+减速机;
}
}