Hadoop AWS EMR S3DistCp:auxService:mapreduce_shuffle不存在

我通过SSH连接到AWS EMR v5.4.0实例，我想调用s3distcp。演示如何设置emr步骤以调用该步骤，但运行该步骤时会出现以下错误：

Container launch failed for container_1492469375740_0001_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:390)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

我照做了，但它仍然不起作用。

在配置

mapreduce\u shuffle

之后，我发现我需要：

$ initctl list | grep yarn
hadoop-yarn-resourcemanager start/running, process 1256
hadoop-yarn-proxyserver start/running, process 702
hadoop-yarn-nodemanager start/running, process 896
$ sudo stop hadoop-yarn-nodemanager
$ sudo start hadoop-yarn-nodemanager

另外，如果它有帮助，

warn site.xml

文件位于：

/etc/hadoop/conf/warn site.xml

。它已经有一个用于

纱线.nodemanager.aux服务的条目

，但没有配置mapreduce\u shuffle：

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>spark_shuffle,</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
  <value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>

纱线.节点管理器.辅助服务
星火洗牌，
纱线.nodemanager.aux-services.spark_shuffle.class
org.apache.spark.network.warn.YarnShuffleService

所以我加了一句：

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>spark_shuffle,mapreduce_shuffle</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
  <value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>

纱线.节点管理器.辅助服务
spark_shuffle，mapreduce_shuffle
warn.nodemanager.aux-services.mapreduce\u shuffle.class
org.apache.hadoop.mapred.ShuffleHandler
纱线.nodemanager.aux-services.spark_shuffle.class
org.apache.spark.network.warn.YarnShuffleService

如果您正在使用AWS数据管道或Apache Airflow之类的工具动态旋转EMR集群，我想知道您是如何做到这一点的，因为在旋转、运行EMR活动，然后将其拆下时，您无法进入EMR集群并进行这样的修改。我不知道如何在我的场景中修复这个错误。。。