原因：java.lang.BootstrapMethodError:java.lang.NoClassDefFoundError:scala/runtime/LambdaDeserialize

我试图使用maven构建下面的spark scala项目。构建成功，但当我运行该jar时，它给出了以下错误。请帮助我修复

spark scala码

package com.spark.DataAnalysis

      import org.apache.log4j.Level
      import org.apache.spark.sql.{Dataset, SparkSession}
      import org.apache.spark.sql.functions._
      import org.apache.spark.SparkContext
      import org.apache.spark.SparkConf

object TwitterData {
  def main(args: Array[String]) {
    println("Start")
    System.setProperty("hadoop.home.dir","C://Sankha//Study//spark-2.3.4-bin-hadoop2.7//spark-2.3.4-bin-hadoop2//spark-2.3.4-bin-hadoop2.7")
    val conf = new SparkConf().setAppName("Spark Scala WordCount Example").setMaster("local[1]")
        val spark = SparkSession.builder().appName("CsvExample").master("local").getOrCreate()
        val sc = new SparkContext(conf)
        val csvData = sc.textFile("C:\\Sankha\\Study\\data\\twitter-airline-sentiment\\Tweets.csv",3)
        val map_data = csvData.map(x=> x.split(",")).filter(x=> (x.length  < 13)).filter(x=> x(5) == "Virgin America")
        println(map_data.count())

  }
}

从命令行运行spark代码，如下所示

spark-submit --class sparkWCExample.spWCExample.Twitter --master local[2] C:\Sankha\Study\spark_ws\spWCExample\target\spWCExample-0.0.1-SNAPSHOT.jar C:\Sankha\Study\spark_ws\spWCExample\target\out

例外情况：

20/03/04 02:45:58 INFO Executor: Adding file:/C:/Users/sankh/AppData/Local/Temp/spark-ae5c0e2c-76f7-42d9-bd2a-6b1f5b191bd8/userFiles-ef86ac49-debf-4d19-b2e9-4f0c1cb83325/spWCExample-0.0.1-SNAPSHOT.jar to class loader
20/03/04 02:45:58 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.io.IOException: unexpected exception type
        at java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1736)
        at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1266)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2078)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)

Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1260)
        ... 61 more
Caused by: java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
        at sparkWCExample.spWCExample.Twitter$.$deserializeLambda$(Twitter.scala)
        ... 71 more
Caused by: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
        ... 72 more
Caused by: java.lang.ClassNotFoundException: scala.runtime.LambdaDeserialize
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 72 more

请告知

POM xml如下所示：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>sparkWCExample</groupId>
  <artifactId>spWCExample</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>spWCExample</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
        <dependency>
          <groupId>junit</groupId>
          <artifactId>junit</artifactId>
          <version>3.8.1</version>
          <scope>test</scope>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
            <version>2.4.5</version>
        </dependency>

            <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.12</artifactId>
            <version>2.4.5</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library -->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.12.3</version>
        </dependency>
  </dependencies>
  <build>
 <plugins>
     <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-compiler-plugin</artifactId>
         <version>3.3</version>
     </plugin>
 </plugins>
 </build>
</project>

4.0.0
Sparkwce示例
SPWC示例
0.0.1-快照
罐子
SPWC示例
http://maven.apache.org
UTF-8
朱尼特
朱尼特
3.8.1
测试
org.apache.spark
spark-core_2.12
2.4.5
org.apache.spark
spark-sql_2.12
2.4.5
org.scala-lang
scala图书馆
2.12.3
org.apache.maven.plugins
maven编译器插件
3.3

---

请检查并让我知道您的代码和POM似乎存在一些问题

说到代码，您已经分别创建了sparksession和sparkcontext，尽管您可以只创建sparksession对象，并且它具有sparkcontext。 此外，您已经在代码和spark submit命令中设置了spark属性。我建议您创建一个单独的SparkProperty文件，并将其传递到spark submit命令（我也将共享该文件和命令）

因此，您可以按如下方式编写代码：

   package com.spark.DataAnalysis

      import org.apache.log4j.Level
      import org.apache.spark.sql.{Dataset, SparkSession}
      import org.apache.spark.sql.functions._
      import org.apache.spark.SparkContext
      import org.apache.spark.SparkConf

object TwitterData {
  def main(args: Array[String]) {
    println("Start")
    System.setProperty("hadoop.home.dir","C://Sankha//Study//spark-2.3.4-bin-hadoop2.7//spark-2.3.4-bin-hadoop2//spark-2.3.4-bin-hadoop2.7")
        val spark = SparkSession.builder().appName("CsvExample").master("local").getOrCreate()
        val csvData = .textFile("C:\\Sankha\\Study\\data\\twitter-airline-sentiment\\Tweets.csv",3)
        val map_data = csvData.map(x=> x.split(",")).filter(x=> (x.length  < 13)).filter(x=> x(5) == "Virgin America")
        println(map_data.count())
        spark.close

  }
}

spark.master    local[2]
spark.submit.deployMode client
spark.driver.memory     2G
spark.executor.memory   2G
spark.executor.instances        2
spark.executor.cores    2
spark.driver.maxResultSize      1g

现在，添加sparkProperties.properties文件后，您的spark submit命令将更易于编写：

spark-submit --class sparkWCExample.spWCExample.Twitter C:\Sankha\Study\spark_ws\spWCExample\target\spWCExample-0.0.1-SNAPSHOT.jar C:\Sankha\Study\spark_ws\spWCExample\target\out --properties-file sparkProperties.properties

---

我希望我已经详细地回答了你的问题。如果您还有任何疑问，请随时询问。

您可以在此处共享您的pom.xml吗？我已更新了上面的pom.xml。请立即告知错误关闭HookManager:删除Spark temp目录时出现异常：C:\Sankha\temp\Spark-0463536d-891f-420e-b555-F8E4243E4F9\userFiles-165a61af-5484-4580-bc8e-c009816b076c java.io.IOException:未能删除：C:\Sankha\temp\spark-0463536d-891f-420e-b555-f8e42243e4f9\userFiles-165a61af-5484-4580-bc8e-c009816b076c\spWCExample-0.0.1-SNAPSHOT-jar-with-dependencies.jar我尝试了您的步骤，它成功了。谢谢。我现在只收到下面的temp文件删除错误，这与之前收到的输出一样好。我是spark的新手，因此，如果我看到其他问题，我可能会再次与您联系。希望您能帮助meSure，您可以随时联系。对于上面列出的错误，您可以尝试在完成计算后关闭SparkSession。这将确保为其他应用程序释放集群资源。我已经对上面的代码做了适当的更改。请告诉我这是否有帮助。你能帮我解决以下我面临的问题吗

spark.master    local[2]
spark.submit.deployMode client
spark.driver.memory     2G
spark.executor.memory   2G
spark.executor.instances        2
spark.executor.cores    2
spark.driver.maxResultSize      1g

spark-submit --class sparkWCExample.spWCExample.Twitter C:\Sankha\Study\spark_ws\spWCExample\target\spWCExample-0.0.1-SNAPSHOT.jar C:\Sankha\Study\spark_ws\spWCExample\target\out --properties-file sparkProperties.properties