Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Apache spark 试图在纱线簇上运行spark提交作业,但我一直收到以下警告。我如何解决这个问题?_Apache Spark_Hdfs_Yarn_Spark Submit - Fatal编程技术网

Apache spark 试图在纱线簇上运行spark提交作业,但我一直收到以下警告。我如何解决这个问题?

Apache spark 试图在纱线簇上运行spark提交作业,但我一直收到以下警告。我如何解决这个问题?,apache-spark,hdfs,yarn,spark-submit,Apache Spark,Hdfs,Yarn,Spark Submit,我仔细研究了类似的问题,并尝试了提到的所有其他问题。当我查看hdfs上的Thread nodemanager日志时,我看到以下可能导致错误的警告。如何修复这些警告 WARN YarnClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources. war

我仔细研究了类似的问题,并尝试了提到的所有其他问题。当我查看hdfs上的Thread nodemanager日志时,我看到以下可能导致错误的警告。如何修复这些警告

WARN YarnClusterScheduler: Initial job has not accepted any resources; check 
your cluster UI to ensure that workers are registered and have sufficient 
resources.
warn-site.xml日志:

2017-09-13 14:29:52,640 WARN 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The 
Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class 
org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. 
Because these are not the same tools trying to send ServiceData and read 
Service Meta Data may have issues unless the refer to the name in the config.
<configuration>
    <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
    </property>

<property>
  <name>yarn.application.classpath</name>
  <value>/usr/local/hadoop/etc/hadoop, /usr/local/hadoop/share/hadoop/common
/*, /usr/local/hadoop/share/hadoop/common/lib/*, 
/usr/local/hadoop/share/hadoop/
hdfs/*, /usr/local/hadoop/share/hadoop/hdfs/lib/*, 
/usr/local/hadoop/share/hadoo
p/mapreduce/*, /usr/local/hadoop/share/hadoop/mapreduce/lib/*, 
/usr/local/hadoop
/share/hadoop/yarn/*, /usr/local/hadoop/share/hadoop/yarn/lib/*</value>
</property>
<property>
   <name>nodemanager.resource.cpu-vcores</name>
   <value>2</value>
</property>
<property>
<description>
  Number of seconds after an application finishes before the nodemanager's
  DeletionService will delete the application's localized file directory
  and log directory.
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://sandbox:9000</value>
</property>
<property>
  <name>dfs.client.use.legacy.blockreader</name>
  <value>true</value>
</property>

</configuration>

纱线.节点管理器.辅助服务
mapreduce_shuffle
.application.classpath
/usr/local/hadoop/etc/hadoop,/usr/local/hadoop/share/hadoop/common
/*,/usr/local/hadoop/share/hadoop/common/lib/*,
/usr/local/hadoop/share/hadoop/
hdfs/*,/usr/local/hadoop/share/hadoop/hdfs/lib/*,
/usr/local/hadoop/share/hadoo
p/mapreduce/*,/usr/local/hadoop/share/hadoop/mapreduce/lib/*,
/usr/local/hadoop
/share/hadoop/thread/*,/usr/local/hadoop/share/hadoop/thread/lib/*
nodemanager.resource.cpu-vcores
2.
应用程序完成后的秒数,节点管理器
DeletionService将删除应用程序的本地化文件目录
和日志目录。
core-site.xml日志:

2017-09-13 14:29:52,640 WARN 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The 
Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class 
org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. 
Because these are not the same tools trying to send ServiceData and read 
Service Meta Data may have issues unless the refer to the name in the config.
<configuration>
    <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
    </property>

<property>
  <name>yarn.application.classpath</name>
  <value>/usr/local/hadoop/etc/hadoop, /usr/local/hadoop/share/hadoop/common
/*, /usr/local/hadoop/share/hadoop/common/lib/*, 
/usr/local/hadoop/share/hadoop/
hdfs/*, /usr/local/hadoop/share/hadoop/hdfs/lib/*, 
/usr/local/hadoop/share/hadoo
p/mapreduce/*, /usr/local/hadoop/share/hadoop/mapreduce/lib/*, 
/usr/local/hadoop
/share/hadoop/yarn/*, /usr/local/hadoop/share/hadoop/yarn/lib/*</value>
</property>
<property>
   <name>nodemanager.resource.cpu-vcores</name>
   <value>2</value>
</property>
<property>
<description>
  Number of seconds after an application finishes before the nodemanager's
  DeletionService will delete the application's localized file directory
  and log directory.
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://sandbox:9000</value>
</property>
<property>
  <name>dfs.client.use.legacy.blockreader</name>
  <value>true</value>
</property>

</configuration>

fs.default.name
hdfs://sandbox:9000
dfs.client.use.legacy.blockreader
真的
hdfs-site.xml日志:

2017-09-13 14:29:52,640 WARN 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The 
Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class 
org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. 
Because these are not the same tools trying to send ServiceData and read 
Service Meta Data may have issues unless the refer to the name in the config.
<configuration>
    <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
    </property>

<property>
  <name>yarn.application.classpath</name>
  <value>/usr/local/hadoop/etc/hadoop, /usr/local/hadoop/share/hadoop/common
/*, /usr/local/hadoop/share/hadoop/common/lib/*, 
/usr/local/hadoop/share/hadoop/
hdfs/*, /usr/local/hadoop/share/hadoop/hdfs/lib/*, 
/usr/local/hadoop/share/hadoo
p/mapreduce/*, /usr/local/hadoop/share/hadoop/mapreduce/lib/*, 
/usr/local/hadoop
/share/hadoop/yarn/*, /usr/local/hadoop/share/hadoop/yarn/lib/*</value>
</property>
<property>
   <name>nodemanager.resource.cpu-vcores</name>
   <value>2</value>
</property>
<property>
<description>
  Number of seconds after an application finishes before the nodemanager's
  DeletionService will delete the application's localized file directory
  and log directory.
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://sandbox:9000</value>
</property>
<property>
  <name>dfs.client.use.legacy.blockreader</name>
  <value>true</value>
</property>

</configuration>

dfs.replication
1.

请让我知道,如果我试图找到一个解决方案,我的初始警告在错误的方向,因为应用程序一直在运行,但没有数据发送到hdfs。谢谢大家!

我通过向hdfs-site.xml添加以下属性修复了警告:dfs.namenode.rpc-bind-host 0.0.0.0但是,在没有数据传输到hdfs的情况下,我仍然收到相同的初始警告。您修复了它吗?我通过将以下属性添加到hdfs-site.xml来修复警告:dfs.namenode.rpc-bind-host 0.0.0。但是,在没有数据传输到hdfs的情况下,我仍然收到相同的初始警告。您修复了吗?