Scala crossvalidator的sparkml设置并行性
因此,我试图使用SparkML设置交叉验证,但我得到了一个运行时错误Scala crossvalidator的sparkml设置并行性,scala,apache-spark,machine-learning,apache-spark-mllib,cross-validation,Scala,Apache Spark,Machine Learning,Apache Spark Mllib,Cross Validation,因此,我试图使用SparkML设置交叉验证,但我得到了一个运行时错误 "value setParallelism is not a member of org.apache.spark.ml.tuning.CrossValidator" 我目前正在学习spark页面教程。我是新手,所以非常感谢您的帮助。下面是我的代码片段: import org.apache.spark.ml.{Pipeline, PipelineModel} import org.apache.spark.ml.class
"value setParallelism is not a member of org.apache.spark.ml.tuning.CrossValidator"
我目前正在学习spark页面教程。我是新手,所以非常感谢您的帮助。下面是我的代码片段:
import org.apache.spark.ml.{Pipeline, PipelineModel}
import org.apache.spark.ml.classification.LogisticRegression
import org.apache.spark.ml.feature.{HashingTF, Tokenizer}
import org.apache.spark.ml.linalg.Vector
import org.apache.spark.sql.Row
import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
import org.apache.spark.ml.tuning.{CrossValidator, ParamGridBuilder}
// Tokenizer
val tokenizer = new Tokenizer().setInputCol("tweet").setOutputCol("words")
// HashingTF
val hash_tf = new HashingTF().setInputCol(tokenizer.getOutputCol).setOutputCol("features")
// ML models
val l_regression = new LogisticRegression().setMaxIter(100).setRegParam(0.15)
// Pipeline
val pipe = new Pipeline().setStages(Array(tokenizer, hash_tf, l_regression))
val paramGrid = new ParamGridBuilder()
.addGrid(hash_tf.numFeatures, Array(10,100,1000))
.addGrid(l_regression.regParam, Array(0.1,0.01,0.001))
.build()
val c_validator = new CrossValidator()
.setEstimator(pipe)
.setEvaluator(new BinaryClassificationEvaluator)
.setEstimatorParamMaps(paramGrid)
.setNumFolds(3)
.setParallelism(2)
setParallelism
。您必须使用早期版本:
(仅限专家)参数设置器
(……)
def setParallelism(值:Int):CrossValidator.this.type
设置最大并行级别以并行评估模型。串行评估的默认值为1
注释@自(“2.3.0”)
如果答案有帮助,请接受它(参见-答案会占用受访者宝贵的时间