Apache spark Spark批内的顺序处理
我有一个关于Spark批处理中的顺序处理的问题。这是一个程式化版本的问题，我试图得到答案，以保持它的简单Apache spark Spark批内的顺序处理,apache-spark,apache-spark-sql,scheduled-tasks,Apache Spark,Apache Spark Sql,Scheduled Tasks,我有一个关于Spark批处理中的顺序处理的问题。这是一个程式化版本的问题，我试图得到答案，以保持它的简单 import org.apache.spark.sql.SparkSession val spark = SparkSession .builder() .appName("Simple Dataframe Processing") .config("spark.some.config.option", "some-value") .getOrCreate() // Fo
import org.apache.spark.sql.SparkSession
val spark = SparkSession
.builder()
.appName("Simple Dataframe Processing")
.config("spark.some.config.option", "some-value")
.getOrCreate()
// For implicit conversions like converting RDDs to DataFrames
import spark.implicits._
val df = spark.read.json("devices.json")
// Displays the content of the DataFrame to stdout
df.show()
// +-------------------------+
// | device-guid| Operation|
// +----+-------+-------------
// |1234 | Add 3 |
// |1234 | Sub 3 |
// |1234 | Add 2 |
// |1234 | Sub 2 |
// |1234 | Add 1 |
// |1234 | Sub 1 |
// +----+-------+------------+
//I have a Database with one table with following columns
// device-guid (primary key) result
//I would like to take df and for each row in the df do a update operation to a single DB row, Adding or removing number as described in Operation column
//So the result I am expecting at the end of this in the DB is a single row with
// device-guid result
// 1234 0
df.foreach { row =>
UpdateDB(row) //Update the DB with the row's Operation.
//Actual method not shown
}