Maven 火花“;三角洲;找不到源
在spark项目中使用kafka和delta_核心依赖项时,我收到了下一个警告:Maven 火花“;三角洲;找不到源,maven,apache-spark,maven-shade-plugin,delta-lake,Maven,Apache Spark,Maven Shade Plugin,Delta Lake,在spark项目中使用kafka和delta_核心依赖项时,我收到了下一个警告: [WARNING] delta-core_2.12-0.7.0.jar, spark-sql-kafka-0-10_2.12-3.1.1.jar define 1 overlapping resources: [WARNING] - META-INF/services/org.apache.spark.sql.sources.DataSourceRegister 这会导致找不到增量源。如何将delta和k
[WARNING] delta-core_2.12-0.7.0.jar, spark-sql-kafka-0-10_2.12-3.1.1.jar define 1 overlapping resources:
[WARNING] - META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
这会导致找不到增量源。如何将delta和kafka都包括在内?谢谢
这是我的maven配置:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql-kafka-0-10_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>io.delta</groupId>
<artifactId>delta-core_${scala.version}</artifactId>
<version>0.7.0</version>
</dependency>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>
META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
</resource>
</transformer>
</transformers>
<finalName>${project.artifactId}-${project.version}</finalName>
<artifactSet>
<includes>
<include>org.scalactic:*</include>
<include>io.delta:*</include>
<include>org.apache.spark:*</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
```
org.apache.spark
spark-sql-kafka-0-10{scala.version}
${spark.version}
三角洲
delta-core{scala.version}
0.7.0
...
org.apache.maven.plugins
maven阴影插件
3.2.2
包裹
阴凉处
META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
${project.artifactId}-${project.version}
org.scalatic:*
io.delta:*
org.apache.spark:*
*:*
```
我解决了它。我的问题是我同时使用了maven shade和maven assembly插件。删除maven组装插件成功了