Apache spark Oozie shell操作引发spark作业的NullPointerException_Apache Spark_Yarn_Oozie

Apache spark Oozie shell操作引发spark作业的NullPointerException

apache-spark

Apache spark Oozie shell操作引发spark作业的NullPointerException,apache-spark,yarn,oozie,Apache Spark,Yarn,Oozie,我有一个带有spark submit命令的shell脚本，可以使用oozie shell操作运行 Oozie能够从shell脚本运行spark submit命令，但在Thread中部署时作业失败非常感谢您的帮助作业在NullPointerException下抛出： Exception in thread "main" java.lang.NullPointerException at scala.collection.mutable.ArrayOps$ofRef$.length$ex

我有一个带有spark submit命令的shell脚本，可以使用oozie shell操作运行

Oozie能够从shell脚本运行spark submit命令，但在Thread中部署时作业失败

非常感谢您的帮助

作业在NullPointerException下抛出：

Exception in thread "main" java.lang.NullPointerException
    at scala.collection.mutable.ArrayOps$ofRef$.length$extension(ArrayOps.scala:114)
    at scala.collection.mutable.ArrayOps$ofRef.length(ArrayOps.scala:114)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:32)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
    at org.apache.spark.deploy.yarn.Client$$anonfun$createConfArchive$2$$anonfun$apply$6.apply(Client.scala:540)
    at org.apache.spark.deploy.yarn.Client$$anonfun$createConfArchive$2$$anonfun$apply$6.apply(Client.scala:537)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.apache.spark.deploy.yarn.Client$$anonfun$createConfArchive$2.apply(Client.scala:537)
    at org.apache.spark.deploy.yarn.Client$$anonfun$createConfArchive$2.apply(Client.scala:536)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.apache.spark.deploy.yarn.Client.createConfArchive(Client.scala:536)
    at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:495)
    at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:727)
    at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:143)
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1018)
    at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1078)
    at org.apache.spark.deploy.yarn.Client.main(Client.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:730)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)

Workflow.xml：

<workflow-app name="test-job" xmlns="uri:oozie:workflow:0.5">
 <start to="shell-node"/>
<action name="shell-node">
    <shell xmlns="uri:oozie:shell-action:0.2">
        <job-tracker>${jobTracker}</job-tracker>
         <name-node>${nameNode}</name-node>
         <configuration>
            <property>
                <name>mapred.job.queue.name</name>
                <value>${queueName}</value>
            </property>
        </configuration>
        <exec>${myscript}</exec>
        <file>${myscriptPath}</file><file>hdfs://<HDFS_PATH>/application.properties#application.properties</file><file>hdfs://<HDFS_PATH>/test-job.jar#test-job.jar</file><file>hdfs://<HDFS_PATH>/myusr.keytab#myusr.keytab</file>
       <capture-output/>
       </shell>
       <ok to="end"/>
       <error to="fail"/>
       </action>
      <kill name="fail">
       <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
     </kill>
    <kill name="fail-output">
    <message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>
</kill>
<end name="end"/>

<workflow-app name="test-job" xmlns="uri:oozie:workflow:0.5">
<credentials>
<credential name="hcat" type="hcat">
    <property>
        <name>hcat.metastore.uri</name>
        <value>${hcatMetastoreUri}</value>
    </property>
    <property>
        <name>hcat.metastore.principal</name>
        <value>${hcatMetastorePrincipal}</value>
    </property>
</credential>
<credential name="hive2" type="hive2">
    <property>
        <name>hive2.jdbc.url</name>
        <value>${hive2JdbcUrl}</value>
    </property>
    <property>
        <name>hive2.server.principal</name>
        <value>${hive2ServerPrincipal}</value>
    </property>
</credential>
</credentials>
<start to="shell-node"/>
<action name="shell-node" cred="hcat, hive2">
<shell xmlns="uri:oozie:shell-action:0.2">
      <job-tracker>${jobTracker}</job-tracker>
      <name-node>${nameNode}</name-node>
     <configuration>
        <property>
            <name>mapred.job.queue.name</name>
            <value>${queueName}</value>
        </property>
    </configuration>
    <exec>${myscript}</exec>
    <env-var>HADOOP_CONF_DIR=/etc/hadoop/conf</env-var>
    <env-var>YARN_CONF_DIR=/etc/hadoop/conf</env-var>
    <file>${myscriptPath}</file><file>hdfs://<HDFS_PATH>/application.properties#application.properties</file><file>hdfs://<HDFS_PATH>/test-job.jar#test-job.jar</file><file>hdfs://<HDFS_PATH>/myusr.keytab#myusr.keytab</file>
   <capture-output/>
   </shell>
   <ok to="end"/>
   <error to="fail"/>
   </action>
  <kill name="fail">
   <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
 </kill>
<kill name="fail-output">
<message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>


${jobTracker}
${nameNode}
mapred.job.queue.name
${queueName}
${myscript}
${myscriptPath}hdfs:///application.properties#application.propertieshdfs:///test-job.jar#测试作业。jarhdfs:///myusr.keytab#myusr.keytab
Shell操作失败，错误消息[${wf:errorMessage（wf:lastErrorNode（））}]
输出不正确，应为[Hello Oozie]，但为[${wf:actionData（'shell-node'）['my_output']}]

通过添加两个系统参数来解决此问题。 HADOOP_CONF_DIR=/etc/HADOOP/CONF 纱线\u CONF\u DIR=/etc/hadoop/CONF

更新的workflow.xml：

<workflow-app name="test-job" xmlns="uri:oozie:workflow:0.5">
 <start to="shell-node"/>
<action name="shell-node">
    <shell xmlns="uri:oozie:shell-action:0.2">
        <job-tracker>${jobTracker}</job-tracker>
         <name-node>${nameNode}</name-node>
         <configuration>
            <property>
                <name>mapred.job.queue.name</name>
                <value>${queueName}</value>
            </property>
        </configuration>
        <exec>${myscript}</exec>
        <file>${myscriptPath}</file><file>hdfs://<HDFS_PATH>/application.properties#application.properties</file><file>hdfs://<HDFS_PATH>/test-job.jar#test-job.jar</file><file>hdfs://<HDFS_PATH>/myusr.keytab#myusr.keytab</file>
       <capture-output/>
       </shell>
       <ok to="end"/>
       <error to="fail"/>
       </action>
      <kill name="fail">
       <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
     </kill>
    <kill name="fail-output">
    <message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>
</kill>
<end name="end"/>

<workflow-app name="test-job" xmlns="uri:oozie:workflow:0.5">
<credentials>
<credential name="hcat" type="hcat">
    <property>
        <name>hcat.metastore.uri</name>
        <value>${hcatMetastoreUri}</value>
    </property>
    <property>
        <name>hcat.metastore.principal</name>
        <value>${hcatMetastorePrincipal}</value>
    </property>
</credential>
<credential name="hive2" type="hive2">
    <property>
        <name>hive2.jdbc.url</name>
        <value>${hive2JdbcUrl}</value>
    </property>
    <property>
        <name>hive2.server.principal</name>
        <value>${hive2ServerPrincipal}</value>
    </property>
</credential>
</credentials>
<start to="shell-node"/>
<action name="shell-node" cred="hcat, hive2">
<shell xmlns="uri:oozie:shell-action:0.2">
      <job-tracker>${jobTracker}</job-tracker>
      <name-node>${nameNode}</name-node>
     <configuration>
        <property>
            <name>mapred.job.queue.name</name>
            <value>${queueName}</value>
        </property>
    </configuration>
    <exec>${myscript}</exec>
    <env-var>HADOOP_CONF_DIR=/etc/hadoop/conf</env-var>
    <env-var>YARN_CONF_DIR=/etc/hadoop/conf</env-var>
    <file>${myscriptPath}</file><file>hdfs://<HDFS_PATH>/application.properties#application.properties</file><file>hdfs://<HDFS_PATH>/test-job.jar#test-job.jar</file><file>hdfs://<HDFS_PATH>/myusr.keytab#myusr.keytab</file>
   <capture-output/>
   </shell>
   <ok to="end"/>
   <error to="fail"/>
   </action>
  <kill name="fail">
   <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
 </kill>
<kill name="fail-output">
<message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>


hcat.metastore.uri
${hcatMetastoreUri}
hcat.metastore.principal
${hcatMetastorePrincipal}
hive2.jdbc.url
${hive2JdbcUrl}
hive2.server.principal
${hive2ServerPrincipal}
${jobTracker}
${nameNode}
mapred.job.queue.name
${queueName}
${myscript}
HADOOP_CONF_DIR=/etc/HADOOP/CONF
纱线\u CONF\u DIR=/etc/hadoop/CONF
${myscriptPath}hdfs:///application.properties#application.propertieshdfs:///test-job.jar#测试作业。jarhdfs:///myusr.keytab#myusr.keytab
Shell操作失败，错误消息[${wf:errorMessage（wf:lastErrorNode（））}]
输出不正确，应为[Hello Oozie]，但为[${wf:actionData（'shell-node'）['my_output']}]

spark1.6（spark-assembly-1.6.0-cdh5.15.1-hadoop2.6.0-cdh5.15.1.jar）中发现了此问题。如果未设置HADOOP_CONF_DIR或Thread_CONF_DIR，则从oozie shell操作将其设置为Thread_CONF_DIR=/run/cloudera scm agent/process/105134纱线节点管理器，HADOOP_CONF_DIR=/run/cloudera scm agent/process/105134纱线节点管理器。这些是不存在的目录。Client.scala代码迭代此目录“dir.listFiles（）.foreach”的文件，该目录为NPE