来自Oozie Java Jobs的电子邮件
我们使用ClouderaCDH5.4.0,并一直试图从JavaAPI触发oozie作业来发送电子邮件。使用SMTP登录发送电子邮件依赖于两个第三方jar文件-activation.jar和mail.jar。当第三方文件放在文件系统的同一文件夹中时,JAVA程序可以从IDE/packages Jar文件发送电子邮件 但是,当我们将文件移动到HDFS并尝试配置oozie作业时,它无法完成 我们的oozie作业xml如下(email.xml): 这些文件以如下方式放置在HDFS文件夹中: /user/oozie/OozieWFConfigs/emailAppDef/EmailJavaProgram.jar /user/oozie/OozieWFConfigs/emailAppDef/email.xml /user/oozie/OozieWFConfigs/emailAppDef/job.properties /user/oozie/OozieWFConfigs/emailAppDef/lib/activation.jar /user/oozie/OozieWFConfigs/emailAppDef/lib/mail.jar 在论坛上读到放在lib文件夹中的jar文件将被自动拾取 使用Java API触发oozie作业,如下所示:来自Oozie Java Jobs的电子邮件,java,email,hadoop,jar,oozie,Java,Email,Hadoop,Jar,Oozie,我们使用ClouderaCDH5.4.0,并一直试图从JavaAPI触发oozie作业来发送电子邮件。使用SMTP登录发送电子邮件依赖于两个第三方jar文件-activation.jar和mail.jar。当第三方文件放在文件系统的同一文件夹中时,JAVA程序可以从IDE/packages Jar文件发送电子邮件 但是,当我们将文件移动到HDFS并尝试配置oozie作业时,它无法完成 我们的oozie作业xml如下(email.xml): 这些文件以如下方式放置在HDFS文件夹中: /user/
import java.util.Properties;
import org.apache.oozie.client.OozieClient;
import org.apache.oozie.client.WorkflowJob;
public class oozieclient {
public static void main(String[] args) {
OozieClient wc = new OozieClient("http://hdfs:hdfs@172.16.142.124:11000/oozie");
Properties conf = wc.createConfiguration();
conf.setProperty("nameNode", "hdfs://kwt-dev-hdpdn6.hadoop.local:8020");
conf.setProperty("jobTracker", "kwt-dev-hdpdn6.hadoop.local:8032");
conf.setProperty("queueName", "default");
conf.setProperty("oozie.libpath", "${nameNode}/user/oozie/OozieWFConfigs/emailAppDef/lib");
conf.setProperty("oozie.use.system.libpath", "true");
conf.setProperty("oozie.wf.rerun.failnodes", "true");
conf.setProperty("oozieProjectRoot",
"${nameNode}/user/oozie");
conf.setProperty("appPath",
"${oozieProjectRoot}/OozieWFConfigs/emailAppDef");
conf.setProperty(OozieClient.APP_PATH, "${appPath}/email.xml");
// conf.setProperty("inputDir", "${oozieProjectRoot}/data/*/*/*/*/*");
conf.setProperty("outputDir", "${appPath}/output");
try {
String jobId = wc.run(conf);
System.out.println("Workflow job, " + jobId + " submitted");
while (wc.getJobInfo(jobId).getStatus() == WorkflowJob.Status.RUNNING) {
System.out.println("Workflow job running ...");
Thread.sleep(10 * 1000);
}
System.out.println("Workflow job completed ...");
System.out.println(wc.getJobInfo(jobId));
} catch (Exception r) {
System.out.println("Errors " + r.getLocalizedMessage());
}
}
}
触发时,作业将一直运行到33%-50%,然后挂起。既不终止也不继续。谁能帮我一下吗?我不能在oozie中使用默认的电子邮件流程,因为我需要在该电子邮件工作后添加附件。我希望它能从一个使用activation.jar和email.jar的java程序中运行
触发作业时,配置如下:
应用路径hdfs://kwt-dev-hdpdn6.hadoop.local:8020/user/oozie/OozieWFConfigs/emailAppDef
jobTracker kwt-dev-hdpdn6.hadoop.本地:8032
mapreduce.job.user.name oozie
名称节点hdfs://kwt-dev-hdpdn6.hadoop.local:8020
oozie.use.system.libpath true
oozie.wf.application.pathhdfs://kwt-dev-hdpdn6.hadoop.local:8020/user/oozie/OozieWFConfigs/emailAppDef/email.xml
oozie.wf.rerun.failnodes true
木根hdfs://kwt-dev-hdpdn6.hadoop.local:8020/user/jinith.joseph
输出目录hdfs://kwt-dev-hdpdn6.hadoop.local:8020/user/oozie/OozieWFConfigs/emailAppDef/output
队列名称默认值
用户名oozie
该问题与HDFS连接有关。您的Oozie URI、Namenode和jobtracker不相等。我认为您应该用正确的IP地址替换本地主机。经过一周的试用,我们已经实现了从oozie jobs发送电子邮件。正如许多论坛和朋友所发现的,问题在于Guava版本不包含elapsedTime()函数 因此,如果我们有一个如下所示的工作流xml,它应该可以正常工作
<workflow-app name="Drill_HDFS_Email" xmlns="uri:oozie:workflow:0.5">
<start to="java-6abb"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="java-6abb">
<java>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.queue.name</name>
<value>default</value>
</property>
<property>
<name>oozie.launcher.mapreduce.job.classloader</name>
<value>true</value>
</property>
<property>
<name>oozie.launcher.mapreduce.job.ubertask.enable</name>
<value>false</value>
</property>
</configuration>
<main-class>com.drill.Emails</main-class>
<file>/user/oozie/OozieWFConfigs/drillEmailAppDef/lib/DrillJDBC.jar#DrillJDBC.jar</file>
<file>/user/oozie/OozieWFConfigs/drillEmailAppDef/lib/activation.jar#activation.jar</file>
<file>/user/oozie/OozieWFConfigs/drillEmailAppDef/lib/mail.jar#mail.jar</file>
<file>/user/oozie/OozieWFConfigs/drillEmailAppDef/lib/drill-jdbc-all-1.0.0.jar#drill-jdbc-all-1.0.0.jar</file>
</java>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>
操作失败,错误消息[${wf:errorMessage(wf:lastErrorNode())}]
${jobTracker}
${nameNode}
mapred.queue.name
违约
oozie.launcher.mapreduce.job.classloader
真的
oozie.launcher.mapreduce.job.ubertask.enable
假的
com.drill.email
/user/oozie/OozieWFConfigs/drillEmailAppDef/lib/DrillJDBC.jar#DrillJDBC.jar
/user/oozie/OozieWFConfigs/drillEmailAppDef/lib/activation.jar#activation.jar
/user/oozie/OozieWFConfigs/drillEmailAppDef/lib/mail.jar#mail.jar
/user/oozie/OozieWFConfigs/drillEmailAppDef/lib/drill-jdbc-all-1.0.0.jar#drill-jdbc-all-1.0.0.jar
正如您可能已经观察到的,有两个配置专门用于选择正确版本的番石榴
oozie.launcher.mapreduce.job.classloader=true
oozie.launcher.mapreduce.job.ubertask.enable=false
默认情况下,ubertask设置为true,这将尝试拾取
Cloudera/oozie的番石榴罐,正在开发较低版本和
不要包含elapsedTime()功能。如果我们将此属性设置为
false它将拾取钻头的罐子,其中包含正确的番石榴
版本
所有依赖的第三方jar和jar函数以及我们发送电子邮件的代码都作为文件包含在oozie工作流中。主类函数将在包含的jar中检查并拾取
在一些论坛中,我们读到“lib”文件夹中的jar文件是自动读取的。但如果不明确指定,我们就无法让它工作。可能我们仍然缺少一些配置
无论如何,希望这对将来的人有所帮助。嗨,莫宾,你是说在job.properties吗?@Jinith,是的。检查您的连接,同时检查日志可能会有所帮助。请尝试。虽然力工作得很好。在运行作业时,已使用拾取的配置更新问题。如果您的问题与“非问题”OOZIE-2066()>>>类似,请尝试将
mapreduce.job.user.classpath.first
属性设置为true
(对于Hadoop 2.x;OOZIE-2066讨论底部链接中提到的SO post中显示的备选方案)注意,在OOZIE启动器作业的上下文中,“用户类路径”指您的JAR和OOZIE共享库(如果有的话),以及用于引导启动器的OOZIE JAR。如果OOZIE发布的是旧版本,结果将是……随机的(例如,HDP 2.3.2使用Hadoop发布了一个旧的httpcore
,使用Oozie发布了一个更旧的版本,使用Hive ShareLib发布了一个更新的版本——总是“没有用户类路径优先就失败”,使用它失败的概率为0.5)。
import java.util.Properties;
import org.apache.oozie.client.OozieClient;
import org.apache.oozie.client.WorkflowJob;
public class oozieclient {
public static void main(String[] args) {
OozieClient wc = new OozieClient("http://hdfs:hdfs@172.16.142.124:11000/oozie");
Properties conf = wc.createConfiguration();
conf.setProperty("nameNode", "hdfs://kwt-dev-hdpdn6.hadoop.local:8020");
conf.setProperty("jobTracker", "kwt-dev-hdpdn6.hadoop.local:8032");
conf.setProperty("queueName", "default");
conf.setProperty("oozie.libpath", "${nameNode}/user/oozie/OozieWFConfigs/emailAppDef/lib");
conf.setProperty("oozie.use.system.libpath", "true");
conf.setProperty("oozie.wf.rerun.failnodes", "true");
conf.setProperty("oozieProjectRoot",
"${nameNode}/user/oozie");
conf.setProperty("appPath",
"${oozieProjectRoot}/OozieWFConfigs/emailAppDef");
conf.setProperty(OozieClient.APP_PATH, "${appPath}/email.xml");
// conf.setProperty("inputDir", "${oozieProjectRoot}/data/*/*/*/*/*");
conf.setProperty("outputDir", "${appPath}/output");
try {
String jobId = wc.run(conf);
System.out.println("Workflow job, " + jobId + " submitted");
while (wc.getJobInfo(jobId).getStatus() == WorkflowJob.Status.RUNNING) {
System.out.println("Workflow job running ...");
Thread.sleep(10 * 1000);
}
System.out.println("Workflow job completed ...");
System.out.println(wc.getJobInfo(jobId));
} catch (Exception r) {
System.out.println("Errors " + r.getLocalizedMessage());
}
}
}
<workflow-app name="Drill_HDFS_Email" xmlns="uri:oozie:workflow:0.5">
<start to="java-6abb"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="java-6abb">
<java>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.queue.name</name>
<value>default</value>
</property>
<property>
<name>oozie.launcher.mapreduce.job.classloader</name>
<value>true</value>
</property>
<property>
<name>oozie.launcher.mapreduce.job.ubertask.enable</name>
<value>false</value>
</property>
</configuration>
<main-class>com.drill.Emails</main-class>
<file>/user/oozie/OozieWFConfigs/drillEmailAppDef/lib/DrillJDBC.jar#DrillJDBC.jar</file>
<file>/user/oozie/OozieWFConfigs/drillEmailAppDef/lib/activation.jar#activation.jar</file>
<file>/user/oozie/OozieWFConfigs/drillEmailAppDef/lib/mail.jar#mail.jar</file>
<file>/user/oozie/OozieWFConfigs/drillEmailAppDef/lib/drill-jdbc-all-1.0.0.jar#drill-jdbc-all-1.0.0.jar</file>
</java>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>