spark submit正在抛出java.lang.ClassNotFoundException:SparkPIIDetector
我有一个简单的JavaSpark应用程序(SimpleApp.jar),我正试图在Windows环境中使用下面的命令运行它spark submit正在抛出java.lang.ClassNotFoundException:SparkPIIDetector,java,maven,apache-spark,window,Java,Maven,Apache Spark,Window,我有一个简单的JavaSpark应用程序(SimpleApp.jar),我正试图在Windows环境中使用下面的命令运行它 spark-submit.cmd --class SimpleJar --master local SimpleApp.jar 它一直在扔 java.lang.ClassNotFoundException: SimpleApp at java.net.URLClassLoader.findClass(Unknown Source) at j
spark-submit.cmd --class SimpleJar --master local SimpleApp.jar
它一直在扔
java.lang.ClassNotFoundException: SimpleApp
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Unknown Source)
at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:686)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
我的项目的目录结构如下
SimpleApp
- infrastructure (AN ANSIBLE MODULE)
- SimpleApp (A MAVEN MODULE)
- - src
- - - main
- - - - java
- - - - - SimpleApp.java
- - - - resources
- - pom.xml
在POM中,我使用shade插件来构建我的uber jar,就像这样
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<!-- Run shade goal on package phase -->
<execution>
<id>SimpleAppId</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<!-- add Main-Class to manifest file -->
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>SimpleApp</mainClass>
</transformer>
</transformers>
<shadedClassifierName>SimpleApp</shadedClassifierName>
</configuration>
</execution>
</executions>
</plugin>
org.apache.maven.plugins
maven阴影插件
2.3
单纯形
包裹
阴凉处
单纯形
单纯形
有人能告诉我这里有什么配置错误吗。
这是否与windows环境有关?
我能够在AWS EMR上部署它,没有任何问题。
此外,这也可能有用
unzip -p SimpleApp.jar META-INF/MANIFEST.MF
Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Built-By: <author>
Created-By: Apache Maven 3.3.9
Build-Jdk: 1.8.0_191
Main-Class: SimpleApp
解压-p SimpleApp.jar META-INF/MANIFEST.MF
清单版本:1.0
Archiver版本:丛式Archiver
建造人:
创建人:ApacheMaven 3.3.9
构建Jdk:1.8.0¿
主类:SimpleApp
在我的机器上运行的spark与我在应用程序中使用的依赖项之间存在版本不匹配。一旦我换上最新版本的spark,一切都很顺利