Java Hadoop mapreduce输入路径不存在(MR Job在寻找什么文件系统?)
我正在尝试在Hadoop集群上运行MR作业。有一次我尝试成功,但从昨天开始,我无法运行相同的命令,因为它找不到输入路径 以下是跟踪:Java Hadoop mapreduce输入路径不存在(MR Job在寻找什么文件系统?),java,hadoop,mapreduce,hdfs,yarn,Java,Hadoop,Mapreduce,Hdfs,Yarn,我正在尝试在Hadoop集群上运行MR作业。有一次我尝试成功,但从昨天开始,我无法运行相同的命令,因为它找不到输入路径 以下是跟踪: [esb99usr@SBT-IPO-078 ~]$ yarn jar /tmp/mihaylov-temp/MR.jar ru.sbt.hadoop.mr.data.index.DayDataIndexStart -D ru.sbt.hadoop.mr.map.mb=3072 -D ru.sbt.hadoop.mr.reduce.count=8 -D ru.sb
[esb99usr@SBT-IPO-078 ~]$ yarn jar /tmp/mihaylov-temp/MR.jar
ru.sbt.hadoop.mr.data.index.DayDataIndexStart -D ru.sbt.hadoop.mr.map.mb=3072 -D ru.sbt.hadoop.mr.reduce.count=8 -D ru.sbt.hadoop.mr.map.count=80 /nameservices/supercluster/esbLogs/year/2015/months/7/days/27/hrs/18/ /nameservices/supercluster/esbLogs/year/2015/months/7/days/27/sort/ 1440622800966 1440709200966
15/09/03 14:33:37 INFO client.RMProxy: Connecting to ResourceManager at SBT-IPO-077.ca.sbrf.ru/10.116.6.195:8032
15/09/03 14:33:38 INFO mapreduce.JobSubmitter: Cleaning up the staging area /nameservices/supercluster/user/esb99usr/.staging/job_1439318575767_0078
15/09/03 14:33:38 WARN security.UserGroupInformation: PriviledgedActionException as:esb99usr (auth:SIMPLE) cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: viewfs://cluster16/nameservices/supercluster/esbLogs/year/2015/months/7/days/27/hrs/18
Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: viewfs://cluster16/nameservices/supercluster/esbLogs/year/2015/months/7/days/27/hrs/18
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:321)
..........
[esb99usr@SBT-IPO-078 ~]$ hdfs dfs -ls /nameservices/supercluster/esbLogs/year/2015/months/7/days/27/hrs/18/
Found 2 items
-rw-r--r-- 3 wasadmin supergroup 1417233459 2015-08-27 21:39 /nameservices/supercluster/esbLogs/year/2015/months/7/days/27/hrs/18/log1.seq
-rw-r--r-- 3 wasadmin supergroup 6020096 2015-08-27 21:39 /nameservices/supercluster/esbLogs/year/2015/months/7/days/27/hrs/18/log1.seq.filter
所以当我运行MR作业时,它声称输入路径不存在,但我可以通过
hadoop dfs-ls/…
看到它。Cloudera说,它一直工作到昨天,并没有证据表明集群已经停止运行,集群的健康状况良好。如何解决此问题?运行作业的命令是
]$ yarn jar /tmp/mihaylov-temp/MR.jar (Main Class Name) (input file path) (output directoru path)
你能检查一下
esb99usr
是否属于supergroup
group???@Mr.Chowdary是的,确实如此。@Mr.Chowdary顺便说一句,文件权限允许任何用户读取hdfs文件,所以我怀疑,这个chmod就是问题所在,你可以试试这个纱线jar/tmp/mihaylov temp/MR.jar ru.sbt.hadoop.MR.data.index.DayDataIndexStart-Dru.sbt.hadoop.MR.map.mb=3072-Dru.sbt.hadoop.MR.reduce.count=8-Dru.sbt.hadoop.MR.map.count=80/nameservices/supercluster/esbLogs/year/2015/months/7/days/27/hrs/18//nameservices/supercluster/esbLogs/year/2015/months/7/days/27/sort/1440622800966 1440709200966
@Mr.Chowdary不幸的是,这没有帮助