Hadoop 简单纱线应用的NoClassDefFoundError

我正在尝试从运行simple Thread应用程序。但我在应用程序错误日志中遇到以下异常

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/conf/YarnConfiguration
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
    at java.lang.Class.getMethod0(Class.java:2774)
    at java.lang.Class.getMethod(Class.java:1663)
    at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.conf.YarnConfiguration

但如果我在所有DataNode上运行“纱线类路径”命令，我会看到以下输出：

/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-yarn/lib/*

---

它具有应用程序所需的Thread客户端、Thread api、Thread common和hadoop common JAR的路径。有谁能告诉我我可能忘记设置正确的类路径的方向。

我发现Hadoop在迭代YarnConfiguration属性时没有解析$Hadoop\u HOME和$Thread\u HOME环境变量。在Thread客户端中运行以下命令将打印未解析的配置，如

$HADOOP\u HOME/，$HADOOP\u HOME/lib/

因此，如果为Thread.application.classpath属性提供完整路径，NoClassDefFoundError问题将得到解决

<property>
    <description>CLASSPATH for YARN applications. A comma-separated list of CLASSPATH entries</description>
    <name>yarn.application.classpath</name>
    <value>
        /etc/hadoop/conf,
        /usr/lib/hadoop/*,
        /usr/lib/hadoop/lib/*,
        /usr/lib/hadoop-hdfs/*,
        /usr/lib/hadoop-hdfs/lib/*,
        /usr/lib/hadoop-mapreduce/*,
        /usr/lib/hadoop-mapreduce/lib/*,
        /usr/lib/hadoop-yarn/*,
        /usr/lib/hadoop-yarn/lib/*
    </value>
  </property>

纱线应用程序的类路径。类路径项的逗号分隔列表
.application.classpath
/etc/hadoop/conf，
/usr/lib/hadoop/*，
/usr/lib/hadoop/lib/*，
/usr/lib/hadoop hdfs/*，
/usr/lib/hadoop hdfs/lib/*，
/usr/lib/hadoop mapreduce/*，
/usr/lib/hadoop mapreduce/lib/*，
/usr/lib/hadoop纱线/*，
/usr/lib/hadoop纱线/lib/*

当ResourceManager和/或NodeManager守护进程启动时，应用程序类路径不完整时，问题将发生在纱线集群上。即使是像附带的spark shell这样简单的东西也会失败：

user@linux$ spark-shell --master yarn-client

遗憾的是，您只有在启动应用程序时才能发现这一点；或者运行它足够长的时间来运行缺少的类。为了解决这个问题，我获取了以下CLASSPATH命令的输出

user@linux$ yarn classpath

并将其清理干净（因为它包含重复项和非规范项），将其附加到下面的纱线配置指令中，该指令位于 /etc/hadoop/conf/warn site.xml，最后重新启动了纱线集群守护进程：

user@linux$ sudo vi /etc/hadoop/conf/yarn-site.xml
[ ... ]
<property>
  <name>yarn.application.classpath</name>
    <value>
      $HADOOP_CONF_DIR,
      $HADOOP_COMMON_HOME/*,
      $HADOOP_COMMON_HOME/lib/*,
      $HADOOP_HDFS_HOME/*,
      $HADOOP_HDFS_HOME/lib/*,
      $HADOOP_MAPRED_HOME/*,
      $HADOOP_MAPRED_HOME/lib/*,
      $YARN_HOME/*,
      $YARN_HOME/lib/*,
      /etc/hadoop/conf,
      /usr/lib/hadoop/*,
      /usr/lib/hadoop/lib,
      /usr/lib/hadoop/lib/*,
      /usr/lib/hadoop-hdfs,
      /usr/lib/hadoop-hdfs/*,
      /usr/lib/hadoop-hdfs/lib/*,
      /usr/lib/hadoop-yarn/*,
      /usr/lib/hadoop-yarn/lib/*,
      /usr/lib/hadoop-mapreduce/*,
      /usr/lib/hadoop-mapreduce/lib/*
    </value>
</property>

user@linux$sudo vi/etc/hadoop/conf/warn-site.xml
[ ... ]
.application.classpath
$HADOOP\u CONF\u DIR，
$HADOOP\u COMMON\u HOME/*，
$HADOOP\u COMMON\u HOME/lib/*，
$HADOOP\u HDFS\u HOME/*，
$HADOOP\u HDFS\u HOME/lib/*，
$HADOOP\u MAPRED\u HOME/*，
$HADOOP\u MAPRED\u HOME/lib/*，
$THEAN_HOME/*，
$THEAN_HOME/lib/*，
/etc/hadoop/conf，
/usr/lib/hadoop/*，
/usr/lib/hadoop/lib，
/usr/lib/hadoop/lib/*，
/usr/lib/hadoop hdfs，
/usr/lib/hadoop hdfs/*，
/usr/lib/hadoop hdfs/lib/*，
/usr/lib/hadoop纱线/*，
/usr/lib/hadoop纱线/lib/*，
/usr/lib/hadoop mapreduce/*，
/usr/lib/hadoop mapreduce/lib/*

上面不包含环境变量引用的条目是我添加的。请记住，在重新启动ResourceManager和NameNode守护进程之前，将此修改后的文件复制到您集群上的所有节点

通常，您需要将所有未提供的依赖项（类和模块）打包到应用程序存档中。=：）

user@linux$ sudo vi /etc/hadoop/conf/yarn-site.xml
[ ... ]
<property>
  <name>yarn.application.classpath</name>
    <value>
      $HADOOP_CONF_DIR,
      $HADOOP_COMMON_HOME/*,
      $HADOOP_COMMON_HOME/lib/*,
      $HADOOP_HDFS_HOME/*,
      $HADOOP_HDFS_HOME/lib/*,
      $HADOOP_MAPRED_HOME/*,
      $HADOOP_MAPRED_HOME/lib/*,
      $YARN_HOME/*,
      $YARN_HOME/lib/*,
      /etc/hadoop/conf,
      /usr/lib/hadoop/*,
      /usr/lib/hadoop/lib,
      /usr/lib/hadoop/lib/*,
      /usr/lib/hadoop-hdfs,
      /usr/lib/hadoop-hdfs/*,
      /usr/lib/hadoop-hdfs/lib/*,
      /usr/lib/hadoop-yarn/*,
      /usr/lib/hadoop-yarn/lib/*,
      /usr/lib/hadoop-mapreduce/*,
      /usr/lib/hadoop-mapreduce/lib/*
    </value>
</property>