Apache spark 如何用动态查询流控制Spark流？
我有一个来自卡夫卡的数据流，称之为SourceStream 我有另一个Spark SQL查询流，其单个值是Spark SQL查询以及一个窗口大小 我希望这些查询应用于SourceStream数据，并将查询结果传递给接收器 例如。 源流Apache spark 如何用动态查询流控制Spark流？,apache-spark,pyspark,apache-spark-sql,spark-streaming,Apache Spark,Pyspark,Apache Spark Sql,Spark Streaming,我有一个来自卡夫卡的数据流，称之为SourceStream 我有另一个Spark SQL查询流，其单个值是Spark SQL查询以及一个窗口大小 我希望这些查询应用于SourceStream数据，并将查询结果传递给接收器 例如。 源流 Id type timestamp user amount ------- ------ ---------- ---------- -------- uuid1 A 342342
Id type timestamp user amount
------- ------ ---------- ---------- --------
uuid1 A 342342 ME 10.0
uuid2 B 234231 YOU 120.10
uuid3 A 234234 SOMEBODY 23.12
uuid4 A 234233 WHO 243.1
uuid5 C 124555 IT 35.12
...
....
Id window query
------- ------ ------
uuid13 1 hour select 'uuid13' as u, max(amount) as output from df where type = 'A' group by ..
uuid21 5 minute select 'uuid121' as u, count(1) as output from df where amount > 100 group by ..
uuid321 1 day select 'uuid321' as u, sum(amount) as output from df where amount > 100 group by ..
...
....