Scala I'；我使用spark 1.5，得到了这个错误

请查看错误

scala> :load beginner_spark_ml.scala

Loading beginner_spark_ml.scala...

import scala.xml._
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql._
import org.apache.spark.sql.types._
import org.apache.spark.sql.functions._
import org.apache.spark.ml.feature.{HashingTF, Tokenizer}
import org.apache.spark.ml.classification.LogisticRegression
import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
import org.apache.spark.ml.Pipeline

fileName: String = Posts.small.xml

textFile: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[8] at textFile at <console>:55

postsXml: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[12] at filter at <console>:60

postsRDD: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] = MapPartitionsRDD[13] at map at <console>:59

schemaString: String = Id Tags Text
schema: org.apache.spark.sql.types.StructType = StructType(StructField(Id,StringType,true), StructField(Tags,Str
<console>:65: error: not found: value spark
       val postsDf =spark.sqlContext.createDataFrame(postsRDD, schema)
                    ^
targetTag: String = java
myudf: String => Double = <function1>

sqlfunc: org.apache.spark.sql.UserDefinedFunction = UserDefinedFunction(<function1>,DoubleType,List(StringType))

<console>:57: error: not found: value postsDf
       val postsLabeled = postsDf.withColumn("Label", sqlfunc(col("Tags")) )

                          ^
<console>:51: error: not found: value postsLabeled
       val positive = postsLabeled.filter('Label > 0.0)
                      ^
<console>:51: error: not found: value postsLabeled
       val negative = postsLabeled.filter('Label < 1.0)
                      ^
<console>:51: error: not found: value positive
       val positiveTrain = positive.sample(false, 0.9)
                           ^
<console>:51: error: not found: value negative
       val negativeTrain = negative.sample(false, 0.9)
                           ^
<console>:51: error: not found: value positiveTrain
       val training = positiveTrain.unionAll(negativeTrain)
                      ^
<console>:51: error: not found: value negativeTrain
       val negativeTrainTmp = negativeTrain.withColumnRenamed("Label", "Flag").select('Id, 'Flag)
                              ^
<console>:51: error: not found: value negative
       val negativeTest = negative.join( negativeTrainTmp, negative("Id") === negativeTrainTmp("Id"), "LeftOuter
                          ^
<console>:51: error: not found: value positiveTrain
       val positiveTrainTmp = positiveTrain.withColumnRenamed("Label", "Flag").select('Id, 'Flag)
                              ^
<console>:51: error: not found: value positive
       val positiveTest = positive.join( positiveTrainTmp, positive("Id") === positiveTrainTmp("Id"), "LeftOuter
                          ^
<console>:51: error: not found: value negativeTest
       val testing = negativeTest.unionAll(positiveTest)
                     ^
numFeatures: Int = 64000
numEpochs: Int = 30
regParam: Double = 0.02
tokenizer: org.apache.spark.ml.feature.Tokenizer = tok_9006f8c2defa
hashingTF: org.apache.spark.ml.feature.HashingTF = hashingTF_9b094ffdf5f6
lr: org.apache.spark.ml.classification.LogisticRegression = logreg_9a578b75908b
pipeline: org.apache.spark.ml.Pipeline = pipeline_8f437ded5dfe
<console>:65: error: not found: value training
       val model = pipeline.fit(training)
                                ^
testTitle: String = Easiest way to merge a release into one JAR file
testBody: String =
Is there a tool or script which easily merges a bunch of
    href=&quot;http://en.wikipedia.org/wiki/JAR_%28file_format%29&quot;
    &gt;JAR&lt;/a&gt; files into one JAR file? A bonus would be to easily set the main-file manifest
    and make it executable. I would like to run it with something like:
    &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;java -jar
    rst.jar&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;
    As far as I can tell, it has no dependencies which indicates that it shouldn't be an easy
    single-file tool, but the downloaded ZIP file contains a lot of libraries.
testText: String =
Easiest way to merge a release into one JAR fileIs there a tool or script which easily merges a bunch of
    href=&quot;http://en.wikipedia.org/wiki/JAR_%28file_format%29&quot;
    &gt;JAR&lt;/a&gt; files into one JAR file? A bonus would be to easily set the main-file manifest
    and make it executable. I would like to run it with something like:
    &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;java -jar
    rst.jar&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;
    As far as I can tell, it has no dependencies which indicates that it shouldn't be an easy
    single-file tool, but the downloaded ZIP file contains a lot of libraries.
<console>:57: error: not found: value sqlContext
       val testDF = sqlContext.createDataFrame(Seq( (99.0, testText))).toDF("Label", "Text")
                    ^
<console>:51: error: not found: value model
       val result = model.transform(testDF)
                    ^
<console>:51: error: not found: value result
       val prediction = result.collect()(0)(6).asInstanceOf[Double]
                        ^
<console>:52: error: not found: value prediction
              print("Prediction: "+ prediction)
                                    ^
<console>:51: error: not found: value model
       val testingResult = model.transform(testing)
                           ^
<console>:51: error: not found: value testingResult
       val testingResultScores = testingResult.select("Prediction", "Label").rdd.
                                 ^
<console>:51: error: not found: value testingResultScores
       val bc = new BinaryClassificationMetrics(testingResultScores)
                                                ^
<console>:51: error: not found: value bc
       val roc = bc.areaUnderROC
                 ^
<console>:52: error: not found: value roc
              print("Area under the ROC:" + roc)
                                            ^

scala>

scala> :load beginner_spark_ml.scala
Loading beginner_spark_ml.scala...
import scala.xml._
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql._
import org.apache.spark.sql.types._
import org.apache.spark.sql.functions._
import org.apache.spark.ml.feature.{HashingTF, Tokenizer}
import org.apache.spark.ml.classification.LogisticRegression
import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
import org.apache.spark.ml.Pipeline
fileName: String = Posts.small.xml
textFile: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[15] at textFile at <console>:74
postsXml: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[19] at filter at <console>:79
postsRDD: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] = MapPartitionsRDD[20] at map at <console>:78
schemaString: String = Id Tags Text
schema: org.apache.spark.sql.types.StructType = StructType(StructField(Id,StringType,true), StructField(Tags,Str
<console>:84: error: not found: value sqlContext
       val postsDf =sqlContext.createDataFrame(postsRDD, schema)
                    ^
targetTag: String = java
myudf: String => Double = <function1>
sqlfunc: org.apache.spark.sql.UserDefinedFunction = UserDefinedFunction(<function1>,DoubleType,List(StringType))
<console>:76: error: not found: value postsDf
       val postsLabeled = postsDf.withColumn("Label", sqlfunc(col("Tags")) )
                          ^
<console>:70: error: not found: value postsLabeled
       val positive = postsLabeled.filter('Label > 0.0)
                      ^
<console>:70: error: not found: value postsLabeled
       val negative = postsLabeled.filter('Label < 1.0)
                      ^
<console>:70: error: not found: value positive
       val positiveTrain = positive.sample(false, 0.9)
                           ^
<console>:70: error: not found: value negative
       val negativeTrain = negative.sample(false, 0.9)
                           ^
<console>:70: error: not found: value positiveTrain
       val training = positiveTrain.unionAll(negativeTrain)
                      ^
<console>:70: error: not found: value negativeTrain
       val negativeTrainTmp = negativeTrain.withColumnRenamed("Label", "Flag").select('Id, 'Flag)
                              ^
<console>:70: error: not found: value negative
       val negativeTest = negative.join( negativeTrainTmp, negative("Id") === negativeTrainTmp("Id"), "LeftOuter
                          ^
<console>:70: error: not found: value positiveTrain
       val positiveTrainTmp = positiveTrain.withColumnRenamed("Label", "Flag").select('Id, 'Flag)
                              ^
<console>:70: error: not found: value positive
       val positiveTest = positive.join( positiveTrainTmp, positive("Id") === positiveTrainTmp("Id"), "LeftOuter
                          ^
<console>:70: error: not found: value negativeTest
       val testing = negativeTest.unionAll(positiveTest)
                     ^
numFeatures: Int = 64000
numEpochs: Int = 30
regParam: Double = 0.02
tokenizer: org.apache.spark.ml.feature.Tokenizer = tok_d760dda17221
hashingTF: org.apache.spark.ml.feature.HashingTF = hashingTF_b8fff6458ec2
lr: org.apache.spark.ml.classification.LogisticRegression = logreg_28b7c8065eb6
pipeline: org.apache.spark.ml.Pipeline = pipeline_83ccdd93d410
<console>:84: error: not found: value training
       val model = pipeline.fit(training)
                                ^
testTitle: String = Easiest way to merge a release into one JAR file
testBody: String =
Is there a tool or script which easily merges a bunch of
    href=&quot;http://en.wikipedia.org/wiki/JAR_%28file_format%29&quot;
    &gt;JAR&lt;/a&gt; files into one JAR file? A bonus would be to easily set the main-file manifest
    and make it executable. I would like to run it with something like:
    &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;java -jar
    rst.jar&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;
    As far as I can tell, it has no dependencies which indicates that it shouldn't be an easy
    single-file tool, but the downloaded ZIP file contains a lot of libraries.
testText: String =
Easiest way to merge a release into one JAR fileIs there a tool or script which easily merges a bunch of
    href=&quot;http://en.wikipedia.org/wiki/JAR_%28file_format%29&quot;
    &gt;JAR&lt;/a&gt; files into one JAR file? A bonus would be to easily set the main-file manifest
    and make it executable. I would like to run it with something like:
    &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;java -jar
    rst.jar&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;
    As far as I can tell, it has no dependencies which indicates that it shouldn't be an easy
    single-file tool, but the downloaded ZIP file contains a lot of libraries.
<console>:76: error: not found: value sqlContext
       val testDF = sqlContext.createDataFrame(Seq( (99.0, testText))).toDF("Label", "Text")
                    ^
<console>:70: error: not found: value model
       val result = model.transform(testDF)
                    ^
<console>:70: error: not found: value result
       val prediction = result.collect()(0)(6).asInstanceOf[Double]
                        ^
<console>:71: error: not found: value prediction
              print("Prediction: "+ prediction)
                                    ^
<console>:70: error: not found: value model
       val testingResult = model.transform(testing)
                           ^
<console>:70: error: not found: value testingResult
       val testingResultScores = testingResult.select("Prediction", "Label").rdd.
                                 ^
<console>:70: error: not found: value testingResultScores
       val bc = new BinaryClassificationMetrics(testingResultScores)
                                                ^
<console>:70: error: not found: value bc
       val roc = bc.areaUnderROC
                 ^
<console>:71: error: not found: value roc
              print("Area under the ROC:" + roc)
                                            ^

scala>

scala>：加载初学者\u spark\u ml.scala
正在加载初学者\u spark\u ml.scala。。。
导入scala.xml_
导入org.apache.spark.sql.catalyst.plans_
导入org.apache.spark.sql_
导入org.apache.spark.sql.types_
导入org.apache.spark.sql.functions_
导入org.apache.spark.ml.feature.{HashingTF，Tokenizer}
导入org.apache.spark.ml.classification.logisticReturnal
导入org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
导入org.apache.spark.ml.Pipeline
文件名：String=Posts.small.xml
textFile:org.apache.spark.rdd.rdd[String]=MapPartitionsRDD[8]位于textFile:55处
postsXml:org.apache.spark.rdd.rdd[String]=MapPartitionsRDD[12]位于：60处的过滤器
postsRDD:org.apache.spark.rdd.rdd[org.apache.spark.sql.Row]=MapPartitionsRDD[13]位于map at:59
schematring:String=Id标记文本
模式：org.apache.spark.sql.types.StructType=StructType（StructField（Id，StringType，true），StructField（Tags，Str
：65:错误：未找到：值火花
val postsDf=spark.sqlContext.createDataFrame（postsRDD，schema）
^
targetTag:String=java
myudf:String=>Double=
sqlfunc:org.apache.spark.sql.UserDefinedFunction=UserDefinedFunction（，DoubleType，List（StringType））
：57:错误：未找到：值postsDf
val postsLabeled=postsDf.withColumn（“标签”，sqlfunc（col（“标签”））
^
：51:错误：找不到：后标记值
val正值=后标签。过滤器（'标签>0.0）
^
：51:错误：找不到：后标记值
val negative=后标签过滤器（'标签<1.0）
^
：51:错误：未找到：值为正值
val positiveTrain=阳性。样本（假，0.9）
^
：51:错误：未找到：值为负
val negativeTrain=阴性。样本（假，0.9）
^
：51:错误：未找到：值为正值train
val training=positiveTrain.unionAll（负Train）
^
：51:错误：未找到：值negativeTrain
val negativeTrainMP=negativeTrain.WithColumnRename（“标签”、“标志”）。选择（'Id'，标志）
^
：51:错误：未找到：值为负
val negativeTest=negative.join（negativeTrainTmp，negative（“Id”）==negativeTrainTmp（“Id”），“LeftOuter
^
：51:错误：未找到：值为正值train
val positiveTrainMP=positiveTrain.WithColumnRename（“标签”、“标志”）。选择（'Id'，标志）
^
：51:错误：未找到：值为正值
val positiveTest=positive.join（positiveTrantmp，positive（“Id”）==positiveTrantmp（“Id”），“LeftOuter
^
：51:错误：未找到：值negativeTest
val测试=阴性测试。unionAll（阳性测试）
^
numFeatures:Int=64000
numEpochs:Int=30
regParam:Double=0.02
标记器：org.apache.spark.ml.feature.tokenizer=tok_9006f8c2defa
hashingTF:org.apache.spark.ml.feature.hashingTF=hashingTF_9b094; ffdf5f6
lr:org.apache.spark.ml.classification.LogisticRegression=logreg_9a578b75908b
管道：org.apache.spark.ml.pipeline=pipeline_8f437ded5dfe
：65:错误：未找到：值培训
val模型=pipeline.fit（培训）
^
testTitle:String=将发布合并到一个JAR文件的最简单方法
testBody:String=
是否有一个工具或脚本可以轻松地合并一组
href=”http://en.wikipedia.org/wiki/JAR_%28file_format%29"
将JAR/a文件转换为一个JAR文件？一个额外的好处是可以轻松设置主文件清单
并使其可执行。我希望使用以下内容运行它：
/p
；
；blockquote
；pjava-jar
rst.jar/p
；/blockquote
；
；p
据我所知，它没有依赖性，这表明它不应该是一个容易的解决方案
单文件工具，但下载的ZIP文件包含很多库。
testText:String=
将发行版合并到一个JAR文件的最简单方法是否有一个工具或脚本可以轻松地合并一组JAR文件
href=”http://en.wikipedia.org/wiki/JAR_%28file_format%29"
将JAR/a文件转换为一个JAR文件？一个额外的好处是可以轻松设置主文件清单
并使其可执行。我希望使用以下内容运行它：
/p
；
；blockquote
；pjava-jar
rst.jar/p
；/blockquote
；
；p
据我所知，它没有依赖性，这表明它不应该是一个容易的解决方案
单文件工具，但下载的ZIP文件包含很多库。
：57:错误：未找到：值sqlContext
val testDF=sqlContext.createDataFrame（Seq（（99.0，testText））.toDF（“标签”，“文本”）
^
：51:错误：未找到：值模型
val结果=model.transform（testDF）
^
：51:错误：未找到：值结果
val prediction=result.collect（）
^
：52:错误：未找到：值预测
打印（“预测：+预测）
^
：51:错误：未找到：值模型
val testingResult=model.transform（测试）
^
：51:错误：未找到：值测试结果
val testingResultScores=testingResult.select（“预测”、“标签”）.rdd。
^
：51:错误：未找到：值TestingResultCores
val bc=新的二进制分类度量（TestingResultCores）
^
：51:错误：未找到：值bc
val roc=bc.Area UnderRoc
^
：52:错误：未找到：值roc
打印（“ROC下方区域：+ROC”）
^
斯卡拉>
scala>：加载初学者\u spark\u ml.scala
正在加载初学者\u spark\u ml.scala。。。
导入scala.xml_
导入org.apache.spark.sql.catalyst.plans_
导入org.apache.spark.sql_
导入org.apache.spark.sql.types_
导入org.apache.spark.sql.functions_
导入org.apache.spa
val conf = new SparkConf().setAppName(appName).setMaster(master)
val sc = new SparkContext(conf)