Scala 在Ubuntu上运行Intellij中的Spark时出错
当我运行下面的代码时,我得到一个错误,它指向注释指示的行。我为Spark带来了JAR文件。在我编写代码时,就如何编写Spark代码提出了一些建议,因此IDE似乎能够识别Spark库。这是用Scala编写的,我对Scala和Spark都是新手。否则,Scala在IDE中运行良好Scala 在Ubuntu上运行Intellij中的Spark时出错,scala,apache-spark,intellij-idea,Scala,Apache Spark,Intellij Idea,当我运行下面的代码时,我得到一个错误,它指向注释指示的行。我为Spark带来了JAR文件。在我编写代码时,就如何编写Spark代码提出了一些建议,因此IDE似乎能够识别Spark库。这是用Scala编写的,我对Scala和Spark都是新手。否则,Scala在IDE中运行良好 import org.apache.spark.sql.SparkSession object Spark { def main(args: Array[String]): Unit = { val logF
import org.apache.spark.sql.SparkSession
object Spark {
def main(args: Array[String]): Unit = {
val logFile = "/home/thomaswolff/ubuntu/spark-2.1.0-bin-hadoop2.7/LICENSE"
val spark = SparkSession.builder.appName("Simple Application").getOrCreate()
val logData = spark.read.textFile(logFile).cache()
val numAs = logData.filter(line => line.contains("a")).count()
val numBs = logData.filter(line => line.contains("b")).count()
println(s"Lines with a: $numAs, Lines with a b: $numBs")
spark.stop()
}
}
以下是在Intellij中运行代码时发生的错误:
[error] Total time: 1 s, completed Oct 8, 2019 5:51:42 PM
[IJ]sbt:ScalaSpark_1> all
{file:/home/thomaswolff/IdeaProjects/ScalaSpark_1/}scalaspark_1/products
{file:/home/thomaswolff/IdeaProjects/ScalaSpark_1/}scalaspark_1/test:products
[info] Compiling 4 Scala sources to
/home/thomaswolff/IdeaProjects/ScalaSpark_1/target/scala-2.13/classes ...
[error] /home/thomaswolff/IdeaProjects/ScalaSpark_1/src/main/scala/Spark.scala:1:12: object
apache is not a member of package org
[error] import org.apache.spark.sql.SparkSession
[error] ^
[error] /home/thomaswolff/IdeaProjects/ScalaSpark_1/src/main/scala/Spark.scala:6:17: not
found: value SparkSession
[error] val spark = SparkSession.builder.appName("Simple Application").getOrCreate()
更新:我安装了Intellij 2018.2的旧版本,并像以前一样加载了JAR,但现在我遇到了以下错误:
Error: Could not find or load main class Spark
Caused by: java.lang.ClassNotFoundException: Spark
我知道问题出在哪里了。我的build.sbt设置不正确。以下是:
name := "sbt_spark_scala"
version := "0.1"
scalaVersion := "2.11.8"
val sparkVersion = "2.2.1"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-graphx" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion,
"org.apache.spark" %% "spark-mllib" % sparkVersion,
"org.apache.spark" %% "spark-hive" % sparkVersion,
"org.apache.spark" %% "spark-streaming" % sparkVersion % "provided"
)
这段youtube视频帮了我很多忙:您使用的是哪个版本的Scala插件和Intellij?Intellij版本是2019.2.3。请稍等Scala版本Scala版本是:Scala代码运行程序版本2.11.12您可以快速尝试的是使用较旧的Intellij IDE。您可以安装JetBrain工具箱,并使用它快速测试其他版本。我在这方面取得了一些成功。显然,最新版本的Scala插件与spark并不完全兼容。你能分享你的
build.sbt
?