Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/scala/16.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181

Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/6.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Scala 如何从spark数据帧计算滚动协方差矩阵_Scala_Apache Spark_Spark Dataframe - Fatal编程技术网

Scala 如何从spark数据帧计算滚动协方差矩阵

Scala 如何从spark数据帧计算滚动协方差矩阵,scala,apache-spark,spark-dataframe,Scala,Apache Spark,Spark Dataframe,我有一个Spark 2.2.0的货币价格数据框架,我将收益添加到其中 import org.apache.spark.sql.SparkSession import org.apache.spark.sql.expressions.Window import org.apache.spark.sql.functions._ val spark = SparkSession.builder.getOrCreate() val prices = spark.read.json("prices.js

我有一个Spark 2.2.0的货币价格数据框架,我将收益添加到其中

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions._

val spark = SparkSession.builder.getOrCreate()
val prices = spark.read.json("prices.json")

// make a window function and convert prices to returns
val window = Window.partitionBy("currency").orderBy("time")
val lagPrice = lag(col("close"), 1).over(window)
val percentReturn = col("close") / col("lastClose") - 1d
val logReturn = log(col("close") / col("lastClose"))

val returns = prices.withColumn("lastClose", lagPrice)
                    .withColumn("return", percentReturn)
                    .withColumn("logReturn", logReturn)
现在我想用一个窗口函数计算所有货币的滚动协方差矩阵(如移动平均数)。但我找不到任何文件或例子