Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Scala 在Ubuntu上运行Intellij中的Spark时出错_Scala_Apache Spark_Intellij Idea - Fatal编程技术网

Scala 在Ubuntu上运行Intellij中的Spark时出错

Scala 在Ubuntu上运行Intellij中的Spark时出错,scala,apache-spark,intellij-idea,Scala,Apache Spark,Intellij Idea,当我运行下面的代码时,我得到一个错误,它指向注释指示的行。我为Spark带来了JAR文件。在我编写代码时,就如何编写Spark代码提出了一些建议,因此IDE似乎能够识别Spark库。这是用Scala编写的,我对Scala和Spark都是新手。否则,Scala在IDE中运行良好 import org.apache.spark.sql.SparkSession object Spark { def main(args: Array[String]): Unit = { val logF

当我运行下面的代码时,我得到一个错误,它指向注释指示的行。我为Spark带来了JAR文件。在我编写代码时,就如何编写Spark代码提出了一些建议,因此IDE似乎能够识别Spark库。这是用Scala编写的,我对Scala和Spark都是新手。否则,Scala在IDE中运行良好

import org.apache.spark.sql.SparkSession

object Spark {
  def main(args: Array[String]): Unit = {
    val logFile = "/home/thomaswolff/ubuntu/spark-2.1.0-bin-hadoop2.7/LICENSE"
    val spark = SparkSession.builder.appName("Simple Application").getOrCreate()
    val logData = spark.read.textFile(logFile).cache()
    val numAs = logData.filter(line => line.contains("a")).count()
    val numBs = logData.filter(line => line.contains("b")).count()
    println(s"Lines with a: $numAs, Lines with a b: $numBs")
    spark.stop()
  }

}
以下是在Intellij中运行代码时发生的错误:

[error] Total time: 1 s, completed Oct 8, 2019 5:51:42 PM
[IJ]sbt:ScalaSpark_1> all 
{file:/home/thomaswolff/IdeaProjects/ScalaSpark_1/}scalaspark_1/products 
{file:/home/thomaswolff/IdeaProjects/ScalaSpark_1/}scalaspark_1/test:products
[info] Compiling 4 Scala sources to 
/home/thomaswolff/IdeaProjects/ScalaSpark_1/target/scala-2.13/classes ...
[error] /home/thomaswolff/IdeaProjects/ScalaSpark_1/src/main/scala/Spark.scala:1:12: object 
apache is not a member of package org
[error] import org.apache.spark.sql.SparkSession
[error]            ^
[error] /home/thomaswolff/IdeaProjects/ScalaSpark_1/src/main/scala/Spark.scala:6:17: not 
found: value SparkSession
[error]     val spark = SparkSession.builder.appName("Simple Application").getOrCreate()
更新:我安装了Intellij 2018.2的旧版本,并像以前一样加载了JAR,但现在我遇到了以下错误:

Error: Could not find or load main class Spark
Caused by: java.lang.ClassNotFoundException: Spark

我知道问题出在哪里了。我的build.sbt设置不正确。以下是:

name := "sbt_spark_scala"

version := "0.1"

scalaVersion := "2.11.8"

val sparkVersion = "2.2.1"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-graphx" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-mllib" % sparkVersion,
  "org.apache.spark" %% "spark-hive" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided"
)

这段youtube视频帮了我很多忙:

您使用的是哪个版本的Scala插件和Intellij?Intellij版本是2019.2.3。请稍等Scala版本Scala版本是:Scala代码运行程序版本2.11.12您可以快速尝试的是使用较旧的Intellij IDE。您可以安装JetBrain工具箱,并使用它快速测试其他版本。我在这方面取得了一些成功。显然,最新版本的Scala插件与spark并不完全兼容。你能分享你的
build.sbt