Warning: file_get_contents(/data/phpspider/zhask/data//catemap/0/hadoop/6.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181

Warning: file_get_contents(/data/phpspider/zhask/data//catemap/1/database/8.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Java 如何在正常权限下运行MR作业_Java_Hadoop_Hadoop Streaming_Hadoop2 - Fatal编程技术网

Java 如何在正常权限下运行MR作业

Java 如何在正常权限下运行MR作业,java,hadoop,hadoop-streaming,hadoop2,Java,Hadoop,Hadoop Streaming,Hadoop2,我已经安装了Hadoop 2.3.0,能够成功地执行MR作业。但当我试图在没有管理员权限的情况下以正常权限执行MR jobs时,意味着job get失败,出现以下异常。 我尝试了WordCount.jar示例 14/10/28 09:16:12 INFO mapreduce.Job: Task Id : attempt_1414467725299_0002_r_000 000_1, Status : FAILED Error: java.lang.NullPointerException

我已经安装了Hadoop 2.3.0,能够成功地执行MR作业。但当我试图在没有管理员权限的情况下以正常权限执行MR jobs时,意味着job get失败,出现以下异常。 我尝试了WordCount.jar示例

14/10/28 09:16:12 INFO mapreduce.Job: Task Id : attempt_1414467725299_0002_r_000
000_1, Status : FAILED
Error: java.lang.NullPointerException
        at org.apache.hadoop.mapred.Task.getFsStatistics(Task.java:347)
        at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(Re
duceTask.java:478)
        at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:414
)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1548)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
通过调试源代码,我深入研究了类YarnChild.java中出现的问题

childUGI.doAs(new PrivilegedExceptionAction<Object>() {
                @Override
                public Object run() throws Exception {
                    // use job-specified working directory
                    FileSystem.get(job).setWorkingDirectory(job.getWorkingDirectory());
                    taskFinal.run(job, umbilical); // run the task
                    return null;
                }
            });
但是,如果我以管理员权限启动NodeManager,则意味着不会发生上述异常。当我以正常权限启动NodeManager时,我不知道为什么job先生不工作


如果有人知道上述问题的原因和解决方法。请尽快帮我一个忙。

您是在Windows还是Linux环境下运行的?什么Hadoop版本?嗨@Lauri,谢谢回复。我已经在Windows8上安装了Hadoop 2.3.0。在运行Pig脚本的同时,我也遇到了这样的错误。您是从计算机上的源代码编译它的吗?对不起,我对Pig一无所知。是的,我是用maven从源代码处编译的。当我在cmd提示符下以admin权限启动Nodemanager时,意味着MR作业可以完美地工作。