Google cloud dataflow 在apache beam中使用JdbcIO访问配置单元抛出java.lang.NoClassDefFoundError:org/apache/avro/reflect/AvroSchema
我可以使用JDBCIO和Avro编码器访问我的mysql表。现在,我正在尝试使用JdbcIO加载我的配置单元数据库。 从数据流连接到配置单元时引发以下异常。梁极客的任何帮助都会非常有用Google cloud dataflow 在apache beam中使用JdbcIO访问配置单元抛出java.lang.NoClassDefFoundError:org/apache/avro/reflect/AvroSchema,google-cloud-dataflow,apache-beam,Google Cloud Dataflow,Apache Beam,我可以使用JDBCIO和Avro编码器访问我的mysql表。现在,我正在尝试使用JdbcIO加载我的配置单元数据库。 从数据流连接到配置单元时引发以下异常。梁极客的任何帮助都会非常有用 Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/reflect/AvroSchema at org.apache.beam.sdk.coders.AvroCoder$AvroDeterminismChecker.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/reflect/AvroSchema
at org.apache.beam.sdk.coders.AvroCoder$AvroDeterminismChecker.recurse(AvroCoder.java:426)
at org.apache.beam.sdk.coders.AvroCoder$AvroDeterminismChecker.check(AvroCoder.java:419)
at org.apache.beam.sdk.coders.AvroCoder.<init>(AvroCoder.java:259)
at org.apache.beam.sdk.coders.AvroCoder.of(AvroCoder.java:120)
at com.google.cloud.bigquery.csv.loader.GoogleSQLPipeline.main(GoogleSQLPipeline.java:101)
Caused by: java.lang.ClassNotFoundException: org.apache.avro.reflect.AvroSchema
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 5 more
线程“main”java.lang.NoClassDefFoundError中的异常:org/apache/avro/reflect/AvroSchema
位于org.apache.beam.sdk.coders.AvroCoder$AvroDeterminismChecker.recurse(AvroCoder.java:426)
位于org.apache.beam.sdk.coders.AvroCoder$AvroDeterminismChecker.check(AvroCoder.java:419)
位于org.apache.beam.sdk.coders.AvroCoder(AvroCoder.java:259)
位于org.apache.beam.sdk.coders.AvroCoder.of(AvroCoder.java:120)
位于com.google.cloud.bigquery.csv.loader.GoogleSQLPipeline.main(GoogleSQLPipeline.java:101)
原因:java.lang.ClassNotFoundException:org.apache.avro.reflect.AvroSchema
位于java.net.URLClassLoader.findClass(URLClassLoader.java:381)
位于java.lang.ClassLoader.loadClass(ClassLoader.java:424)
位于sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
位于java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 还有5个
下面的代码段正在尝试访问配置单元:
dataflowPipeline
.apply(JdbcIO.<Customer>read()
.withDataSourceConfiguration(JdbcIO.DataSourceConfiguration
.create("org.apache.hive.jdbc.HiveDriver", "jdbc:hive2://<ip>/mydb")
.withUsername("username").withPassword("password"))
.withQuery(
"select c_customer_id,c_first_name,c_last_name,c_preferred_cust_flag,c_birth_day,c_birth_month,c_birth_year,c_birth_country,c_customer_sk,c_current_cdemo_sk,c_current_hdemo_sk from customer")
.withRowMapper(new JdbcIO.RowMapper<Customer>() {
@Override
public Customer mapRow(ResultSet resultSet) throws Exception
dataflowPipeline
.apply(JdbcIO.read()
.withDataSourceConfiguration(JdbcIO.DataSourceConfiguration
.create(“org.apache.hive.jdbc.HiveDriver”,“jdbc:hive2:///mydb”)
.withUsername(“用户名”).withPassword(“密码”))
.withQuery(
“选择c_客户id、c_名字、c_姓氏、c_首选客户标志、c_出生日、c_出生月、c_出生年、c_出生国、c_客户sk、c_当前_cdemo sk、c_当前_hdemo sk from customer”)
.withRowMapper(新的JdbcIO.RowMapper(){
@凌驾
公共客户映射行(ResultSet ResultSet)引发异常
POM依赖性:
com.google.cloud.dataflow
google云数据流java sdk全部
2.0.0
org.apache.beam
beam SDK java io jdbc
2.0.0
org.apache.hive
蜂窝jdbc
1.2.1
jdk.tools
jdk.tools
1.8.0_131
系统
${JAVA_HOME}/lib/tools.jar
org.apache.hadoop
hadoop通用
2.8.1
番石榴
番石榴
18
org.slf4j
slf4j-jdk14
1.7.14
为Avro 1.8.1添加了依赖项,并手动导入了import org.apache.Avro.reflect.AvroSchema;现在该问题已经解决。同样,与com.google.protobuf.GeneratedMessageV3相关的问题也通过手动导入此类得到解决。为Avro 1.8.1添加了依赖项,并手动导入了import org.apache.Avro.reflect.AvroSchema;现在t问题已经解决。同样,与com.google.protobuf.GeneratedMessageV3相关的问题也可以通过手动导入此类来解决。
<dependencies>
<dependency>
<groupId>com.google.cloud.dataflow</groupId>
<artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
<version>2.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-io-jdbc -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-jdbc</artifactId>
<version>2.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hive/hive-jdbc -->
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>jdk.tools</groupId>
<artifactId>jdk.tools</artifactId>
<version>1.8.0_131</version>
<scope>system</scope>
<systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.8.1</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>18.0</version>
</dependency>
<!-- slf4j API frontend binding with JUL backend -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<version>1.7.14</version>
</dependency>
</dependencies>