Java 拼花匠缺少与avro相关的拼花
我正在尝试使用ApacheFlink进行流处理,学习StreamingFileLink教程 书中提到,我可以指定我的AvroPOJO对象,并将我的流保存到带有avro模式的拼花文件中,保存到本地fs中 看起来是这样的:Java 拼花匠缺少与avro相关的拼花,java,apache-flink,avro,parquet,Java,Apache Flink,Avro,Parquet,我正在尝试使用ApacheFlink进行流处理,学习StreamingFileLink教程 书中提到,我可以指定我的AvroPOJO对象,并将我的流保存到带有avro模式的拼花文件中,保存到本地fs中 看起来是这样的: val-sink:StreamingFileSink[String]=StreamingFileSink .forBulkFormat( 新路径(“/base/Path”), 实木复合地板。用于特殊记录(类别[AvroPojo])) .build() 在我添加后(测试了1.9和
val-sink:StreamingFileSink[String]=StreamingFileSink
.forBulkFormat(
新路径(“/base/Path”),
实木复合地板。用于特殊记录(类别[AvroPojo]))
.build()
在我添加后(测试了1.9和1.8,在1.9中我使用了flink-parquet_2.11)
因此,我必须添加镶木地板avro依赖性manuelly
那就给我:
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
更新:
pom.xml
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_${scala.compat}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_${scala.compat}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-runtime-web_${scala.compat}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>6.7.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>6.7.0</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-avro</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-parquet_${scala.compat}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-avro</artifactId>
<version>1.10.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.6.5</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.pulsar/pulsar-common -->
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-common</artifactId>
<version>2.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.6.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.6.1</version>
</dependency>
org.apache.flink
flink-streaming-java_${scala.compat}
${flink.version}
org.apache.flink
flink-streaming-scala_${scala.compat}
${flink.version}
org.apache.flink
flink-runtime-web_${scala.compat}
${flink.version}
org.elasticsearch
弹性搜索
6.7.0
编译
org.elasticsearch.client
运输
6.7.0
乔达时间
乔达时间
2.7
org.apache.flink
弗林克·阿夫罗
${flink.version}
org.apache.flink
弗林克拼花地板${scala.compat}
${flink.version}
org.apache.parquet
镶木地板
1.10.0
org.apache.hadoop
hadoop客户端
2.6.5
org.apache.pulsar
公共脉冲星
2.4.1
org.apache.logging.log4j
log4japi
2.6.1
org.apache.logging.log4j
log4j型芯
2.6.1
问题:
1.没有配置hadoop,我如何编译这段代码?或者使用parquet和avro在我的本地fs中保存数据的任何其他方法?可以发布依赖项的文件吗。我想一定有什么事是真的missing@MIkCode更新了,它是在我添加hadoop客户端后编译的,不知道为什么。流式处理现在似乎可以工作了,但是reuslt没有在我的本地磁盘上持久化。你可以发布依赖项的文件吗。我想一定有什么事是真的missing@MIkCode更新了,它是在我添加hadoop客户端后编译的,不知道为什么。流式处理现在似乎正在工作,但在我的本地磁盘上并没有持久化ReusElt
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_${scala.compat}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_${scala.compat}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-runtime-web_${scala.compat}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>6.7.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>6.7.0</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-avro</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-parquet_${scala.compat}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-avro</artifactId>
<version>1.10.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.6.5</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.pulsar/pulsar-common -->
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-common</artifactId>
<version>2.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.6.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.6.1</version>
</dependency>