Java 多类加载器和Hadoop API

我们的应用程序有多个类加载器。这是为了避免软件访问的各种API（包括但不限于Hadoop）之间的依赖性冲突。如果不使用多类加载器策略（或Java 9模块，我们还不能支持），我们会在Hadoop JAR和其他所有东西之间遇到很多依赖冲突。因此，只需说我们为所有Hadoop JAR提供了一个URLClassLoader，并为其他库集提供了其他类加载器

问题在于：系统类加载器（JVM启动时使用的类加载器）没有Hadoop JAR。这意味着线程上下文类加载器（默认情况下是系统类加载器）没有Hadoop JAR。不幸的是，各种Hadoop API似乎都隐式地使用线程上下文类加载器，这导致了各种各样的ClassNotFound错误、ServiceLoader返回空列表、工厂找不到实现等。这让我们慢慢发疯，因为新的“找不到”当我们的客户发现新的安全选项和其他配置变体时，该领域偶尔会出现问题，这导致代码路径无法找到接口的类或实现

您可能会提出一个明显的建议：始终将线程上下文类加载器设置为Hadoop类加载器。不幸的是，这似乎不是一个解决方案。首先，生成的线程具有默认的系统类加载器作为线程上下文类加载器，我无法控制所有这些线程。其次，在每个可能的Hadoop入口点保存和恢复线程上下文类加载器很容易出错，而且成本可能太高

我找到的唯一答案是通过反复试验了解线程上下文类加载器（或ServiceLoader）加载了哪些类，并强制它们在我设置线程上下文类加载器的地方预加载。有更好的办法吗？是否有此类课程的列表

这是一个堆栈跟踪示例：

Error:ProcessNode(PSOutput16): org/apache/hadoop/fs/FileSystem.create:java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to load class: [org.apache.hadoop.yarn.api.records.impl.pb.ApplicationIdPBImpl]
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:468)
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:754)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:283)
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:528)
org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1484)
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1586)
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1571)
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:440)
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:433)
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:433)
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:374)
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:926)
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:907)
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to load class: [org.apache.hadoop.yarn.api.records.impl.pb.ApplicationIdPBImpl]
org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.newRecordInstance(RecordFactoryPBImpl.java:58)
org.apache.hadoop.yarn.util.Records.newRecord(Records.java:36)
org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(ApplicationId.java:49)
org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:83)
org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:145)
org.apache.hadoop.security.token.Token.identifierToString(Token.java:349)
org.apache.hadoop.security.token.Token.toString(Token.java:369)
java.lang.String.valueOf(String.java:2994)
java.lang.StringBuilder.append(StringBuilder.java:131)
org.apache.hadoop.security.UserGroupInformation.logAllUserInfo(UserGroupInformation.java:1959)
org.apache.hadoop.crypto.key.kms.KMSClientProvider.getActualUgi(KMSClientProvider.java:1056)
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:451)
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:754)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:283)
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:528)
org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1484)
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1586)
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1571)
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:440)
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:433)
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:433)
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:374)
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:926)
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:907)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.api.records.impl.pb.ApplicationIdPBImpl not found
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2255)
org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.newRecordInstance(RecordFactoryPBImpl.java:56)
org.apache.hadoop.yarn.util.Records.newRecord(Records.java:36)
org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(ApplicationId.java:49)
org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:83)