Hadoop auxService:mapreduce_shuffle在配置单元上不存在
我正在使用Hive1.2.0和Hadoop2.6.0。每当我在我的机器上运行蜂巢时。。。select query工作正常,但在出现Hadoop auxService:mapreduce_shuffle在配置单元上不存在,hadoop,hive,hdfs,yarn,Hadoop,Hive,Hdfs,Yarn,我正在使用Hive1.2.0和Hadoop2.6.0。每当我在我的机器上运行蜂巢时。。。select query工作正常,但在出现count(*)的情况下,它会显示以下错误: 此任务的诊断消息:的容器启动失败 集装箱_14346464658807 _0001 _01 000005: org.apache.hadoop.warn.exceptions.InvalidAuxServiceException:auxService:mapreduce\u shuffle不存在 sun.reflect.N
count(*)
的情况下,它会显示以下错误:
此任务的诊断消息:的容器启动失败
集装箱_14346464658807 _0001 _01 000005:
org.apache.hadoop.warn.exceptions.InvalidAuxServiceException:auxService:mapreduce\u shuffle不存在
sun.reflect.NativeConstructorAccessorImpl.newInstance0(本机方法)
在
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
在
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
位于java.lang.reflect.Constructor.newInstance(Constructor.java:422)
在
org.apache.hadoop.warn.api.records.impl.pb.SerializedExceptionPBImpl.InstanceeException(SerializedExceptionPBImpl.java:168)
在
org.apache.hadoop.warn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
在
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
在
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
在
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
在
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
运行(Thread.java:745)
失败:执行错误,从返回代码2
org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce作业已启动:
阶段1:映射:1减少:1 HDFS读取:0 HDFS写入:0失败
花费的MapReduce CPU总时间:0毫秒
请在warn-site.xml中使用此选项;当您将框架设置为用作纱线时,它开始查找这些值
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
纱线.节点管理器.辅助服务
mapreduce_shuffle
warn.nodemanager.aux-services.mapreduce\u shuffle.class
org.apache.hadoop.mapred.ShuffleHandler
当您将框架设置为用作纱线时,它开始查找这些值。对于Java Map reduce,我也面临同样的错误。当我设置如下配置时:
Configuration conf = new Configuration();
conf.set("mapreduce.framework.name", "yarn");
Container launch failed for container_1448364218966_0001_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
我通过设置以下内容来解决此问题:
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
纱线.节点管理器.辅助服务
mapreduce_shuffle
warn.nodemanager.aux-services.mapreduce\u shuffle.class
org.apache.hadoop.mapred.ShuffleHandler
~应该是warn.nodemanager.aux-services.mapreduce\u shuffle.class?我使用了warn.nodemanager.aux-services.mapreduce\u shuffle.class,它成功了@AravindKrishnakumar你是在那之后重新启动蜂巢还是?@Anil你是在那之后重新启动蜂巢还是?你是在那之后重新启动蜂巢还是执行了哪些步骤?