Hadoop 将oozie作业输出发送到邮件ID_Hadoop_Hdfs_Oozie_Oozie Coordinator_Bigdata

Hadoop 将oozie作业输出发送到邮件ID

hadoop

Hadoop 将oozie作业输出发送到邮件ID,hadoop,hdfs,oozie,oozie-coordinator,bigdata,Hadoop,Hdfs,Oozie,Oozie Coordinator,Bigdata,我几乎没有像hadoop中那样需要每天执行的命令，结果需要发送到我的邮件ID。我如何使用oozie JOB完成这项工作？在oozie Coordinator的帮助下，您可以根据时间和数据触发器提交工作流。您可以在下面找到发送电子邮件的日常协调员工作的示例： <?xml version="1.0" encoding="UTF-8"?> <coordinator-app name="cord1" frequency="${coord:days(1)}" s

我几乎没有像hadoop中那样需要每天执行的命令，结果需要发送到我的邮件ID。我如何使用oozie JOB完成这项工作？

在oozie Coordinator的帮助下，您可以根据时间和数据触发器提交工作流。您可以在下面找到发送电子邮件的日常协调员工作的示例：

<?xml version="1.0" encoding="UTF-8"?>
<coordinator-app name="cord1" frequency="${coord:days(1)}"
             start="${jobStart}" end="${jobEnd}"
             timezone="UTC"
             xmlns="uri:oozie:coordinator:0.4">
  <action name="an-email">
        <email xmlns="uri:oozie:email-action:0.1">
            <to>bob@initech.com,the.other.bob@initech.com</to>
            <cc>will@initech.com</cc>
            <subject>Email notifications for ${wf:id()}</subject>
            <body>The wf ${wf:id()} successfully completed.</body>
        </email>
        <ok to="myotherjob"/>
        <error to="errorcleanup"/>
    </action>
</coordinator-app>

要设置以下内容，以便执行映射操作并在此之后发送电子邮件。在oozie-site.xml中，添加以下设置并重新启动oozie。将值替换为特定于您的环境的相同值

<property>
    <name>oozie.email.smtp.host</name>
    <value></value>
</property>
<property>
    <name>oozie.email.smtp.port</name>
    <value>25</value>
</property>
<property>
<name>oozie.email.from.address</name>
<value></value>
</property>
<property>
    <name>oozie.email.smtp.auth</name>
<value>false</value>
</property>
<property>
    <name>oozie.email.smtp.username</name>
    <value></value>
</property>
<property>
    <name>oozie.email.smtp.password</name>
    <value></value>
</property>


oozie.email.smtp.host
oozie.email.smtp.port
25
oozie.email.from.address
oozie.email.smtp.auth
假的
oozie.email.smtp.username
oozie.email.smtp.password

在workflow.xml中，将以下说明与环境设置一起添加。这包括mapreduce完成后的电子邮件触发操作

<workflow-app name="WorkFlowJavaMapReduceAction" xmlns="uri:oozie:workflow:0.1">
<start to="mapReduceAction" />
<action name="mapReduceAction">
    <map-reduce>
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <prepare>
            <delete path="${outputDir}" />
        </prepare>
        <configuration>
            <property>
                <name>mapred.mapper.new-api</name>
                <value>true</value>
            </property>
            <property>
                <name>mapred.reducer.new-api</name>
                <value>true</value>
            </property>
            <property>
                <name>mapred.job.queue.name</name>
                <value>${queueName}</value>
            </property>
            <property>
                <name>mapreduce.map.class</name>
                <value></value>
            </property>
            <property>
                <name>mapreduce.reduce.class</name>
                <value></value>
            </property>
            <property>
                <name>mapred.mapoutput.key.class</name>
                <value>org.apache.hadoop.io.Text</value>
            </property>
            <property>
                <name>mapred.mapoutput.value.class</name>
                <value>org.apache.hadoop.io.IntWritable</value>
            </property>
            <property>
                <name>mapred.output.key.class</name>
                <value>org.apache.hadoop.io.Text</value>
            </property>
            <property>
                <name>mapred.output.value.class</name>
                <value>org.apache.hadoop.io.IntWritable</value>
            </property>
            <property>
                <name>mapred.input.dir</name>
                <value>${inputDir}</value>
            </property>
            <property>
                <name>mapred.output.dir</name>
                <value>${outputDir}</value>
            </property>
            <property>
                <name>mapreduce.job.acl-view-job</name>
                <value>*</value>
            </property>
            <property>
                <name>oozie.launcher.mapreduce.job.acl-view-job</name>
                <value>*</value>
            </property>
            <property>
                <name>oozie.use.system.libpath</name>
                <value>false</value>
            </property>
            <property>
                <name>oozie.libpath</name>
                <value>${appPath}/lib</value>
            </property>
        </configuration>
    </map-reduce>
    <ok to="emailCommands" />
    <error to="killJob" />
</action>

<action name="emailCommands">
    <fs>
        <mkdir path='${makeDirectoryAbsPath}' />
        <move source='${dataInputDirectoryAbsPath}' target='${dataDestinationDirectoryRelativePath}' />
    </fs>
    <ok to="sendEmailSuccess" />
    <error to="sendEmailKill" />
</action>
<action name="sendEmailSuccess">
    <email xmlns="uri:oozie:email-action:0.1">
        <to>${emailToAddress}</to>
        <subject>Status of workflow ${wf:id()}</subject>
        <body>The workflow ${wf:id()} completed successfully</body>
    </email>
    <ok to="end" />
    <error to="end" />
</action>
<action name="sendEmailKill">
    <email xmlns="uri:oozie:email-action:0.1">
        <to>${emailToAddress}</to>
        <subject>Status of workflow ${wf:id()}</subject>
        <body>The workflow ${wf:id()} had issues and was killed. The error
            message is: ${wf:errorMessage(wf:lastErrorNode())}</body>
    </email>
    <ok to="end" />
    <error to="killJob" />
</action>

<kill name="killJob">
    <message>"Killed job due to error:
        ${wf:errorMessage(wf:lastErrorNode())}"</message>
</kill>
<end name="end" />    
</workflow-app>


Mobin是否可以将命令的输出数据发送到mail-in-body>wf${wf:id（）}成功完成。我正在执行hadoop fs-ls/命令，希望将目录的输出列表输入到我的邮件中。@Naveen运行hdfs命令是另一个工作流。协调器中不能同时有两个工作流。但您可以使用Oozie Bundle来拥有每天发生的多个工作流。一个是HDFS命令工作流，两个是电子邮件工作流。请记住，作业之间的依赖关系需要由用户管理。
<workflow-app name="WorkFlowJavaMapReduceAction" xmlns="uri:oozie:workflow:0.1">
<start to="mapReduceAction" />
<action name="mapReduceAction">
    <map-reduce>
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <prepare>
            <delete path="${outputDir}" />
        </prepare>
        <configuration>
            <property>
                <name>mapred.mapper.new-api</name>
                <value>true</value>
            </property>
            <property>
                <name>mapred.reducer.new-api</name>
                <value>true</value>
            </property>
            <property>
                <name>mapred.job.queue.name</name>
                <value>${queueName}</value>
            </property>
            <property>
                <name>mapreduce.map.class</name>
                <value></value>
            </property>
            <property>
                <name>mapreduce.reduce.class</name>
                <value></value>
            </property>
            <property>
                <name>mapred.mapoutput.key.class</name>
                <value>org.apache.hadoop.io.Text</value>
            </property>
            <property>
                <name>mapred.mapoutput.value.class</name>
                <value>org.apache.hadoop.io.IntWritable</value>
            </property>
            <property>
                <name>mapred.output.key.class</name>
                <value>org.apache.hadoop.io.Text</value>
            </property>
            <property>
                <name>mapred.output.value.class</name>
                <value>org.apache.hadoop.io.IntWritable</value>
            </property>
            <property>
                <name>mapred.input.dir</name>
                <value>${inputDir}</value>
            </property>
            <property>
                <name>mapred.output.dir</name>
                <value>${outputDir}</value>
            </property>
            <property>
                <name>mapreduce.job.acl-view-job</name>
                <value>*</value>
            </property>
            <property>
                <name>oozie.launcher.mapreduce.job.acl-view-job</name>
                <value>*</value>
            </property>
            <property>
                <name>oozie.use.system.libpath</name>
                <value>false</value>
            </property>
            <property>
                <name>oozie.libpath</name>
                <value>${appPath}/lib</value>
            </property>
        </configuration>
    </map-reduce>
    <ok to="emailCommands" />
    <error to="killJob" />
</action>

<action name="emailCommands">
    <fs>
        <mkdir path='${makeDirectoryAbsPath}' />
        <move source='${dataInputDirectoryAbsPath}' target='${dataDestinationDirectoryRelativePath}' />
    </fs>
    <ok to="sendEmailSuccess" />
    <error to="sendEmailKill" />
</action>
<action name="sendEmailSuccess">
    <email xmlns="uri:oozie:email-action:0.1">
        <to>${emailToAddress}</to>
        <subject>Status of workflow ${wf:id()}</subject>
        <body>The workflow ${wf:id()} completed successfully</body>
    </email>
    <ok to="end" />
    <error to="end" />
</action>
<action name="sendEmailKill">
    <email xmlns="uri:oozie:email-action:0.1">
        <to>${emailToAddress}</to>
        <subject>Status of workflow ${wf:id()}</subject>
        <body>The workflow ${wf:id()} had issues and was killed. The error
            message is: ${wf:errorMessage(wf:lastErrorNode())}</body>
    </email>
    <ok to="end" />
    <error to="killJob" />
</action>

<kill name="killJob">
    <message>"Killed job due to error:
        ${wf:errorMessage(wf:lastErrorNode())}"</message>
</kill>
<end name="end" />    
</workflow-app>