Scala ClassCastException:java.lang.Double不能强制转换为org。使用LabeledPoint时使用apache.spark.mllib.linalg.Vector
我试图使用SVMWithSGD来训练我的模型,但在尝试访问我的训练时遇到了ClassCastException。 我的train_数据帧架构如下所示:Scala ClassCastException:java.lang.Double不能强制转换为org。使用LabeledPoint时使用apache.spark.mllib.linalg.Vector,scala,apache-spark,apache-spark-sql,rdd,apache-spark-mllib,Scala,Apache Spark,Apache Spark Sql,Rdd,Apache Spark Mllib,我试图使用SVMWithSGD来训练我的模型,但在尝试访问我的训练时遇到了ClassCastException。 我的train_数据帧架构如下所示: train_data.printSchema root |-- label: string (nullable = true) |-- features: vector (nullable = true) |-- label_index: double (nullable = false) 我创建了一个LabeledPointRDD来在S
train_data.printSchema
root
|-- label: string (nullable = true)
|-- features: vector (nullable = true)
|-- label_index: double (nullable = false)
我创建了一个LabeledPointRDD来在SVNWithSGD上使用它
val targetInd = train_data.columns.indexOf("label_index")`
val featInd = Array("features").map(train_data.columns.indexOf(_))
val train_lp = train_data.rdd.map(r => LabeledPoint( r.getDouble(targetInd),
Vectors.dense(featInd.map(r.getDouble(_)).toArray)))
但是当我打电话的时候
SVM与SGD列车(列车lp、数量)
它给了我:
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGSched
uler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGSche
duler.scala:1877)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGSche
duler.scala:1876)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:
59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.appl
y(DAGScheduler.scala:926)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.appl
y(DAGScheduler.scala:926)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.sc
ala:926)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGSche
duler.scala:2110)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGSchedu
ler.scala:2059)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGSchedu
ler.scala:2048)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1364)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:1
51)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:1
12)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.take(RDD.scala:1337)
at org.apache.spark.rdd.RDD$$anonfun$first$1.apply(RDD.scala:1378)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:1
51)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:1
12)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.first(RDD.scala:1377)
at org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm.generateInitia
lWeights(GeneralizedLinearAlgorithm.scala:204)
at org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm.run(Generalize
dLinearAlgorithm.scala:234)
at org.apache.spark.mllib.classification.SVMWithSGD$.train(SVM.scala:217)
at org.apache.spark.mllib.classification.SVMWithSGD$.train(SVM.scala:255)
... 55 elided
Caused by: java.lang.ClassCastException: java.lang.Double cannot be cast to org.
apache.spark.mllib.linalg.Vector
我的列车数据是基于标签(文件名)和功能(代表图像功能的json文件)创建的。尝试使用此选项-
模式
train_data.printSchema
根
|--标签:字符串(nullable=true)
|--特征:向量(可空=真)
|--标签索引:双精度(nullable=false)
将代码修改为-
val train\u lp=train\u data.rdd.map(r=>LabeledPoint(r.getAs(“label\u index”)、r.getAs(“features”))