Scala (Spark)对象{name}不是包org.apache.Spark.ml的成员
我尝试在apache spark上使用scala运行自包含应用程序,基于以下示例: 以下是我的完整代码:Scala (Spark)对象{name}不是包org.apache.Spark.ml的成员,scala,apache-spark,sbt,apache-spark-mllib,Scala,Apache Spark,Sbt,Apache Spark Mllib,我尝试在apache spark上使用scala运行自包含应用程序,基于以下示例: 以下是我的完整代码: import org.apache.spark.ml.classification.LogisticRegression import org.apache.spark.ml.linalg.{Vector, Vectors} import org.apache.spark.ml.param.ParamMap import org.apache.spark.sql.Row object m
import org.apache.spark.ml.classification.LogisticRegression
import org.apache.spark.ml.linalg.{Vector, Vectors}
import org.apache.spark.ml.param.ParamMap
import org.apache.spark.sql.Row
object mllibexample1 {
def main(args: Array[String]) {
val spark = SparkSession
.builder()
.master("local[*]")
.appName("logistic regression example 1")
.getOrCreate()
val training = spark.createDataFrame(Seq(
(1.0, Vectors.dense(0.0, 1.1, 0.1)),
(0.0, Vectors.dense(2.0, 1.0, -1.0)),
(0.0, Vectors.dense(2.0, 1.3, 1.0)),
(1.0, Vectors.dense(0.0, 1.2, -0.5))
)).toDF("label", "features")
val lr = new LogisticRegression()
println("LogisticRegression parameters:\n" + lr.explainParams() + "\n")
lr.setMaxIter(100)
.setRegParam(0.01)
val model1 = lr.fit(training)
println("Model 1 was fit using parameters: " + model1.parent.extractParamMap)
}
}
build.sbt中的依赖项:
name := "example"
version := "1.0.0"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.0.1",
"org.apache.spark" %% "spark-sql" % "2.0.1",
"org.apache.spark" %% "spark-mllib-local" % "2.0.1",
"com.github.fommil.netlib" % "all" % "1.1.2"
)
但是,在sbt shell中运行程序后,我得到以下错误:
[info] Compiling 1 Scala source to /dataplatform/example/target/scala-2.11/classes...
[error] /dataplatform/example/src/main/scala/mllibexample1.scala:1: object classification is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.classification.LogisticRegression
[error] ^
[error] /dataplatform/example/src/main/scala/mllibexample1.scala:3: object param is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.param.ParamMap
[error] ^
[error] /dataplatform/example/src/main/scala/mllibexample1.scala:8: not found: value SparkSession
[error] val spark = SparkSession
[error] ^
[error] /dataplatform/example/src/main/scala/mllibexample1.scala:22: not found: type LogisticRegression
[error] val lr = new LogisticRegression()
我可以在spark interactive shell中成功运行此代码。我在*.sbt文件中遗漏了什么吗
谢谢,
Bayu您错过了一个MLlib依赖项:
"org.apache.spark" %% "spark-mllib" % "2.0.1"
本地是不够的。我有同样的问题,我有一个Maven Scala项目 我使用了下面的Maven依赖项。添加此maven依赖项后,问题得到解决
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>2.0.2</version>
</dependency
org.apache.spark
spark-mllib_2.11
2.0.2
当我尝试导入它时,我可能会得到它的副本:[warn]::org.apache.spark#spark-mllib_2.12;2.0.1:未找到