Scala Spark submit Sql上下文创建语句不起作用
我下面的代码不适用于Spark submitScala Spark submit Sql上下文创建语句不起作用,scala,apache-spark,spark-streaming,apache-spark-sql,Scala,Apache Spark,Spark Streaming,Apache Spark Sql,我下面的代码不适用于Spark submit sqlContext.sql(s""" create external table if not exists landing ( date string, referrer string) partitioned by (partnerid string,dt string) row format delimited fields terminated by '\t' lines terminated by '\n' STORED AS TEXTF
sqlContext.sql(s"""
create external table if not exists landing (
date string,
referrer string)
partitioned by (partnerid string,dt string)
row format delimited fields terminated by '\t' lines terminated by '\n'
STORED AS TEXTFILE LOCATION 's3n://....'
""")
它给出了错误:
线程“main”java.lang.RuntimeException中出现异常:[1.2]失败:`with''应为“”,但找到标识符create
此代码适用于Spark shell,但不适用于Spark submit。原因是什么?默认情况下,spark shell中的“sqlContext”是“HiveContext”。也许您需要在脚本中新建一个HiveContext而不是sqlContext
你可以这样做:
import SparkContext._
import org.apache.spark.sql.hive._
val sc = new SparkContext()
val sqlContext = new HiveContext(sc)
可能缺少导入?也
导入sqlContext.implicits.\u
?类分析扩展可序列化的{val sc=new SparkContext()val sqlContext=SQLContextSingleton.getInstance(sc)导入sqlContext.implicits.\u=>我这样运行,但没有办法:(您最终找到了解决方案吗?