Warning: file_get_contents(/data/phpspider/zhask/data//catemap/0/hadoop/6.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Shell 库包不能与oozie一起使用_Shell_Hadoop_Oozie_Oozie Coordinator_Sparkr - Fatal编程技术网

Shell 库包不能与oozie一起使用

Shell 库包不能与oozie一起使用,shell,hadoop,oozie,oozie-coordinator,sparkr,Shell,Hadoop,Oozie,Oozie Coordinator,Sparkr,嗨,我正在用shell脚本运行oozie。在这个shell脚本中,我使用的是sparkR作业。每当运行oozie作业时,我都会发现库出错 这是我的错误 Stdoutput Running /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/spark/bin/spark-submit --class edu.berkeley.cs.amplab.sparkr.SparkRRunner --files pi.R --master yarn-c

嗨,我正在用shell脚本运行oozie。在这个shell脚本中,我使用的是sparkR作业。每当运行oozie作业时,我都会发现库出错

这是我的错误

  Stdoutput Running /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/spark/bin/spark-submit --class edu.berkeley.cs.amplab.sparkr.SparkRRunner --files pi.R --master yarn-client   /SparkR-pkg/lib/SparkR/sparkr-assembly-0.1.jar pi.R yarn-client 4
  Stdoutput Error in library(SparkR) : there is no package called ‘SparkR’
  Stdoutput Execution halted
  Exit code of the Shell command 1
  <<< Invocation of Shell command completed <<< 
oozieProjectRoot=shell\u示例 oozie.wf.application.path=${oozieProjectRoot}/apps/shell

my workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.1" name="Test">
<start to="shell-node"/>
<action name="shell-node">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
 <configuration>
            <property>
                <name>mapred.job.queue.name</name>
                <value>${queueName}</value>
            </property>


        </configuration>

    <exec>script.sh</exec>
    <file>oozie-oozi/script.sh#script.sh</file>
    <file>/user/karun/examples/pi.R</file>
        <capture-output/>

    </shell>
    <ok to="end"/>
     <error to="fail"/>
     </action>
       <kill name="fail">
        <message>Incorrect output</message>

</kill>
<end name="end"/>

</workflow-app>

我不知道如何解决这个问题。如果您能提供任何帮助,我们将不胜感激……

您的群集的每个节点和R会话中是否都安装了SparkR软件包?R在其会话期间找不到该包。是..所有节点上以及R会话中安装的sparkR包?(install.packages(…)s完成了所有工作。问题是,当我运行shell脚本时,它工作正常。但当我通过oozie调用shell脚本时,它不工作
<workflow-app xmlns="uri:oozie:workflow:0.1" name="Test">
<start to="shell-node"/>
<action name="shell-node">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
 <configuration>
            <property>
                <name>mapred.job.queue.name</name>
                <value>${queueName}</value>
            </property>


        </configuration>

    <exec>script.sh</exec>
    <file>oozie-oozi/script.sh#script.sh</file>
    <file>/user/karun/examples/pi.R</file>
        <capture-output/>

    </shell>
    <ok to="end"/>
     <error to="fail"/>
     </action>
       <kill name="fail">
        <message>Incorrect output</message>

</kill>
<end name="end"/>

</workflow-app>
export SPARK_HOME=/opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/spark
export YARN_CONF_DIR=/etc/hadoop/conf
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
export HADOOP_CMD=/usr/bin/hadoop

/SparkR-pkg/lib/SparkR/sparkR-submit --master yarn-client pi.R yarn-client 4