Java 如何在SpringBatch项目中使用JDBC和HikariCP连接到hive?
我正在尝试在spring批处理项目中使用hikaricp(kerberos和keytab)连接到HiveJDBC 下面是我的jdbcDataSource配置Java 如何在SpringBatch项目中使用JDBC和HikariCP连接到hive?,java,hive,spring-batch,spring-jdbc,jdbctemplate,Java,Hive,Spring Batch,Spring Jdbc,Jdbctemplate,我正在尝试在spring批处理项目中使用hikaricp(kerberos和keytab)连接到HiveJDBC 下面是我的jdbcDataSource配置 @Bean(name = "hiveJdbcBatchDataSource") @Qualifier(value = "hiveJdbcBatchDataSource") public DataSource hiveJdbcBatchDataSource() throws Exception { try {
@Bean(name = "hiveJdbcBatchDataSource")
@Qualifier(value = "hiveJdbcBatchDataSource")
public DataSource hiveJdbcBatchDataSource() throws Exception {
try {
HikariConfig config = new HikariConfig();
config.setDriverClassName(driverClassName);
config.setJdbcUrl(hiveUrl);
System.setProperty("java.security.krb5.conf", krb5ConfPath);
if (StringUtils.isNotBlank(keytabPath)) {
org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
conf.set("hadoop.security.authentication", "kerberos");
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab(principal, keytabPath);
} else {
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
config.setUsername(userName);
config.setPassword(password);
}
config.setConnectionTestQuery("show databases");
config.addDataSourceProperty("zeroDateTimeBehavior", zeroDateTimeBehavior);
config.addDataSourceProperty("cachePrepStmts", cachePrepStmts);
config.addDataSourceProperty("prepStmtCacheSize", prepStmtCacheSize);
config.addDataSourceProperty("prepStmtCacheSqlLimit", prepStmtCacheSqlLimit);
// connection pooling
config.setPoolName(poolName);
config.setMaximumPoolSize(maximumPoolSize);
config.setIdleTimeout(idleTimeoutMs);
config.setMaxLifetime(maxLifetimeMs);
return new HikariDataSource(config);
} catch (IOException e) {
throw new BeanInitializationException("IOException Failed to init data souce.", e);
} catch (Exception e) {
throw new Exception("Exception Failed to init data souce.", e);
}
}
我得到了以下例外
Caused by: org.springframework.batch.core.configuration.BatchConfigurationException: java.lang.IllegalArgumentException: DatabaseType not found for product name: [Apache Hive]
at org.springframework.batch.core.configuration.annotation.DefaultBatchConfigurer.initialize(DefaultBatchConfigurer.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
... 16 common frames omitted
Caused by: java.lang.IllegalArgumentException: DatabaseType not found for product name: [Apache Hive]
at org.springframework.batch.support.DatabaseType.fromProductName(DatabaseType.java:84)
at org.springframework.batch.support.DatabaseType.fromMetaData(DatabaseType.java:123)
at org.springframework.batch.core.repository.support.JobRepositoryFactoryBean.afterPropertiesSet(JobRepositoryFactoryBean.java:183)
at org.springframework.batch.core.configuration.annotation.DefaultBatchConfigurer.createJobRepository(DefaultBatchConfigurer.java:134)
at org.springframework.batch.core.configuration.annotation.DefaultBatchConfigurer.initialize(DefaultBatchConfigurer.java:113)
... 23 common frames omitted
我的pom包含以下依赖项
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.3</version>
</dependency>
org.apache.hive
一个重要的细节,您是想使用Apache Hive作为Spring Batch的作业存储库,还是只需要从Hive读取/写入数据(并使用另一个db作为Spring Batch元数据)?@MahmoudBenHassine谢谢,只需要从Hive读取/写入数据(并使用另一个db作为Spring Batch元数据)@MahmoudBenHassine实际上,我使用的是spring批处理版本4.1.2,spring引导版本2.1.7,并设置spring.datasource.initialization mode=never,spring.batch.initializeschema=never。我只想使用JDBCURL(hikaricp或SimpledataSource)和Kerberos密钥表连接到配置单元。在此配置单元上,我将在配置单元tasklet中执行查询。(注意:我的蜂巢没有用户名和密码)感谢您的更新。该错误意味着作业存储库工厂bean正试图将配置单元用作spring批处理元数据的数据源。为了使用另一个数据源,请查看@MahmoudBenHassineThanks,对于链接,实际上,我不想存储作业repo。元数据(状态管理表)也被转换到mysql中,它也充当我的ETL管道中的一个db