Apache spark 如何在spark中连接数据帧中的字符串和列?
我把今天的日期作为一个字符串。我需要将它与一个时间值连接起来,该时间值在数据帧中作为列显示 当我尝试此操作时,我得到Apache spark 如何在spark中连接数据帧中的字符串和列?,apache-spark,dataframe,spark-dataframe,Apache Spark,Dataframe,Spark Dataframe,我把今天的日期作为一个字符串。我需要将它与一个时间值连接起来,该时间值在数据帧中作为列显示 当我尝试此操作时,我得到字符串索引超出范围异常 我的代码: val todaydate = LocalDate.now().toString() println(todaydate) // o/p: 2016-12-10 val todayrec_cutoff = todaydate + (" ") + df.col("colname") 预期产出: 2016-12-10 05:00 2016-1
字符串索引超出范围
异常
我的代码:
val todaydate = LocalDate.now().toString()
println(todaydate) // o/p: 2016-12-10
val todayrec_cutoff = todaydate + (" ") + df.col("colname")
预期产出:
2016-12-10 05:00
2016-12-10 22:30
你可以像下面这样做
import java.time.LocalDate
val df = Seq(("05:00"), ("22:30")).toDF("time")
df.show
val todaydate = LocalDate.now().toString()
val df2 = df.select(concat(lit(todaydate+ " "),df.col("time"))).toDF("datetime");
df2.show
这会给你
+----------------+
| datetime|
+----------------+
|2016-12-10 05:00|
|2016-12-10 22:30|
+----------------+
在Python 2.7中,lit是否不可用?
**Please refer to below Scala code for string concat in prefix and postfix way.**
import org.apache.spark.sql.functions._
val empDF = MongoSpark.load(spark, readConfig) //dataframe empDF is loaded from Mongo DB using MongoSpark
val prefixVal= "PrefixArkay " //variable string
val postfixVal= " PostfixArkay"
//Prefix
val finalPreDF = ipDF.withColumn("EMP", concat(lit(prefix),empDF.col("EMP")) )
println("finalPreDF.show-> " + finalPreDF.show())
//Output will be as below
+-------------------+
| EMP|
+-------------------+
|PrefixArkay DineshS|
|+------------------+
val finalPostDF = ipDF.withColumn("EMP", concat(empDF.col("EMP"),lit(postfixVal)) )
println("finalPostDF.show-> " + finalPostDF .show())
//Output will be as below
+--------------------+
| EMP|
+--------------------+
|DineshS PostfixArkay|
|+-------------------+