Google Dataflow作业在eclipse上运行时运行良好,但在我使用mvn编译它并使用java-cp命令运行文件时出现错误
我正在从Dataflow sdk 1.x迁移到使用apache beam的2.x。当我在eclipse上使用run as java application运行代码时,代码运行正常。但是当我使用mvn clean compile assembly:single编译代码时,然后使用命令java-cp运行类文件 我得到这些日志,然后是一个异常Google Dataflow作业在eclipse上运行时运行良好,但在我使用mvn编译它并使用java-cp命令运行文件时出现错误,java,google-cloud-dataflow,apache-beam,Java,Google Cloud Dataflow,Apache Beam,我正在从Dataflow sdk 1.x迁移到使用apache beam的2.x。当我在eclipse上使用run as java application运行代码时,代码运行正常。但是当我使用mvn clean compile assembly:single编译代码时,然后使用命令java-cp运行类文件 我得到这些日志,然后是一个异常 INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files
from the classpath: will stage 1 files. Enable logging at DEBUG level to see
which files will be staged.
Apr 25, 2018 11:17:01 AM com.example.LogParser parse
INFO: Currently processing file: File1.gz
Exception in thread "main" java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/
String;Ljava/lang/Object;Ljava/lang/Object;)V at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write.expand(BigQueryIO.java:1512)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write.expand(BigQueryIO.java:1041)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:472)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:286)
at com.example.LogParser.parse(LogParser.java:541)
at
com.example.LogParser.main(LogParser.java:167)
我试着更新番石榴的版本,但没用
我还注意到,当我使用eclipse运行类文件时,我得到的日志与使用jar文件运行时的日志不同
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files
from the classpath: will stage 147 files. Enable logging at DEBUG level to
see which files will be staged.
我能够通过添加这些maven依赖项而不是maven assembly插件,然后使用mvn包而不是mvn assembly来解决这个错误
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven-compiler-plugin.version}</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<parallel>all</parallel>
<threadCount>4</threadCount>
<redirectTestOutputToFile>true</redirectTestOutputToFile>
</configuration>
<dependencies>
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>${maven-surefire-plugin.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>${maven-jar-plugin.version}</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${maven-shade-plugin.version}</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>${project.artifactId}-bundled-${project.version}</finalName>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/LICENSE</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
</configuration>
</execution>
</executions>
</plugin>