Java Oozie一直在向类路径添加旧版本的httpcore jar
我一直收到一个异常,因为Oozie在类路径中添加了错误版本的httpcore jar。我尝试了不同的选择,比如Java Oozie一直在向类路径添加旧版本的httpcore jar,java,hadoop,oozie,Java,Hadoop,Oozie,我一直收到一个异常,因为Oozie在类路径中添加了错误版本的httpcore jar。我尝试了不同的选择,比如 oozie.launcher.mapreduce.task.classpath.user.precedence oozie.launcher.mapreduce.user.classpath.first oozie.launcher.mapreduce.task.classpath.user.priority根本不响应,当我使用oozie.launcher.mapreduce.use
oozie.launcher.mapreduce.task.classpath.user.precedence
oozie.launcher.mapreduce.user.classpath.first
oozie.launcher.mapreduce.task.classpath.user.priority根本不响应,当我使用oozie.launcher.mapreduce.user.classpath.first时,应用程序甚至不能加载一个类
在类路径中,我可以看到两个版本的http核心
httpcore-4.4.1.jar
httpcore-4.2.4.jar
当应用程序在独立模式下运行时,我不会得到那个异常
例外情况:
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, java.lang.NoSuchFieldError: INSTANCE
org.apache.oozie.action.hadoop.JavaMainException: java.lang.NoSuchFieldError: INSTANCE
at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NoSuchFieldError: INSTANCE
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.<clinit>(SSLConnectionSocketFactory.java:144)
at microsoft.exchange.webservices.data.core.ExchangeServiceBase.createConnectionSocketFactoryRegistry(ExchangeServiceBase.java:244)
at microsoft.exchange.webservices.data.core.ExchangeServiceBase.initializeHttpClient(ExchangeServiceBase.java:198)
at microsoft.exchange.webservices.data.core.ExchangeServiceBase.<init>(ExchangeServiceBase.java:174)
at microsoft.exchange.webservices.data.core.ExchangeServiceBase.<init>(ExchangeServiceBase.java:179)
at microsoft.exchange.webservices.data.core.ExchangeService.<init>(ExchangeService.java:3729)
at com.sonasoft.sonacloud.email.dispatcher.conn.EwsConnection.getConnection(EwsConnection.java:16)
at com.sonasoft.sonacloud.email.dispatcher.conn.EwsConnection.getConnection(EwsConnection.java:10)
at com.sonasoft.sonacloud.email.dispatcher.utils.EwsOperations.<init>(EwsOperations.java:47)
at com.sonasoft.sonacloud.email.dispatcher.utils.EwsOperations.getInstance(EwsOperations.java:53)
at com.sonasoft.sonacloud.email.dispatcher.main.MainClass.main(MainClass.java:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
... 15 more
失败的Oozie启动器,主类[org.apache.Oozie.action.hadoop.JavaMain],Main()引发异常,java.lang.NoSuchFieldError:INSTANCE
org.apache.oozie.action.hadoop.javamain异常:java.lang.NoSuchFieldError:实例
位于org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59)
位于org.apache.oozie.action.hadoop.launchemain.run(launchemain.java:47)
位于org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)处
在sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)中
在sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)中
位于java.lang.reflect.Method.invoke(Method.java:606)
在org.apache.oozie.action.hadoop.launchemapper.map(launchemapper.java:236)上
位于org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
位于org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
位于org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
位于org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
位于java.security.AccessController.doPrivileged(本机方法)
位于javax.security.auth.Subject.doAs(Subject.java:415)
位于org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
位于org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
原因:java.lang.NoSuchFieldError:实例
位于org.apache.http.conn.ssl.SSLConnectionSocketFactory(SSLConnectionSocketFactory.java:144)
位于microsoft.exchange.webservices.data.core.ExchangeServiceBase.createConnectionSocketFactoryRegistry(ExchangeServiceBase.java:244)
位于microsoft.exchange.webservices.data.core.ExchangeServiceBase.initializeHttpClient(ExchangeServiceBase.java:198)
位于microsoft.exchange.webservices.data.core.ExchangeServiceBase。(ExchangeServiceBase.java:174)
位于microsoft.exchange.webservices.data.core.ExchangeServiceBase。(ExchangeServiceBase.java:179)
位于microsoft.exchange.webservices.data.core.ExchangeService.(ExchangeService.java:3729)
位于com.sonasoft.sonacloud.email.dispatcher.conn.EwsConnection.getConnection(EwsConnection.java:16)
位于com.sonasoft.sonacloud.email.dispatcher.conn.EwsConnection.getConnection(EwsConnection.java:10)
位于com.sonasoft.sonacloud.email.dispatcher.utils.ewscoperations.(ewscoperations.java:47)
位于com.sonasoft.sonacloud.email.dispatcher.utils.ewscoperations.getInstance(ewscoperations.java:53)
位于com.sonasoft.sonacloud.email.dispatcher.main.MainClass.main(MainClass.java:41)
在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)处
在sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)中
在sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)中
位于java.lang.reflect.Method.invoke(Method.java:606)
位于org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
... 还有15个
Oozie客户端构建版本:4.2.0.2.3.2.0-2950
非常感谢您的帮助。您希望使用本地版本的
httpcore
JAR构建,但不希望它出现在您的类路径中,因为Hadoop将提供自己的版本。然后,您应该使用提供的范围,用于httpcore
JAR:
<project>
...
<dependencies>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<scope>provided</scope> <!-- this line is important -->
<version>4.4.1</version>
</dependency>
</dependencies>
</project>
...
org.apache.httpcomponents
对于提供的,
:
这很像compile,但表示您希望JDK或容器在运行时提供依赖关系
我们在HortonWorks发行版2.3.2中遇到了这样一个棘手的问题(真为他们感到羞耻):
- Oozie“launcher”作业总是在
类路径作为Hadoop客户端的一部分
- Oozie“launcher”作业始终获得
httpcore
和httpclient
捆绑在“Oozie”共享库中
- 配置单元/Hive2共享库包含
httpcore
和httpclient
,位于
最新版本
- 从Hadoop的角度来看,
user.classpath.first
适用于两者
ShareLibs,因此每种产品都有50/50的机会获得正确的订单
JAR(因此总的几率为25/75)
一句话:我们必须这样做
从“Oozie”共享库HDF中删除httpcore
和httpclient
主任(duh!)
为依赖配置单元JAR的所有操作(即配置单元操作、配置单元2操作、以某种方式调用JDBC驱动程序的Shell操作等)升起oozie.launcher.mapreduce.job.user.classpath.first
标志
Post scriptum——Oozie服务器在内存中保留每个共享库中的JAR列表,因此在服务器运行时删除JAR将触发新作业中的错误。如果您不想停止Oozie服务器,那么更新live ShareLib的“正确方法”是:(a)在新的带时间戳的目录中创建一个新版本[检查文档…]和(b)告诉服务器使用oozie admin-sharelibupdate
在较新的libs上重新同步,您能告诉我们两个httpcore
JAR文件的源代码吗?您好,httpcore-4.2.4.jar来自hdfs://ip:port//user/oozie/share/lib/lib_20151027124452/oozie/ 和httpcore-4.4.1 jar,我们假设您无法更改Hadoop的类加载器。那么你是如何构建你的应用程序的?你能用像Maven这样的东西吗