Warning: file_get_contents(/data/phpspider/zhask/data//catemap/4/json/15.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
如何通过scala在spark的结构化流媒体中显示json密钥_Json_Scala_Apache Spark_Streaming - Fatal编程技术网

如何通过scala在spark的结构化流媒体中显示json密钥

如何通过scala在spark的结构化流媒体中显示json密钥,json,scala,apache-spark,streaming,Json,Scala,Apache Spark,Streaming,我是spark的新手,试图理解结构化流媒体,我想在spark控制台上显示kafka的json密钥。任何帮助都会有很大帮助 火花代码 import org.apache.spark.sql._ import org.apache.spark.sql.functions._ import org.apache.spark.sql.types.StructType import org.apache.spark.sql.types._ import org.apache.spark.sql.strea

我是spark的新手,试图理解结构化流媒体,我想在spark控制台上显示kafka的json密钥。任何帮助都会有很大帮助

火花代码

import org.apache.spark.sql._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.types._
import org.apache.spark.sql.streaming.ProcessingTime

object kafka_stream {
  def main(args: Array[String]): Unit = {

    val spark = SparkSession
      .builder()
      .appName("kafka-consumer")
      .master("local[*]")
      .getOrCreate()
    import spark.implicits._

    spark.sparkContext.setLogLevel("WARN")

//    val schema = StructType().add("a", IntegerType()).add("b", StringType())

    val schema = StructType(Seq(
      StructField("a", IntegerType, true),
      StructField("b", StringType, true)
    ))

    val df = spark
    .readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "172.21.0.187:9093")
    .option("subscribe", "test")
    .option("startingOffsets", "earliest")
    .load()


    values = df.select("a.*")

    values.writeStream
      .outputMode("complete")
      .format("console")
      .start()a
      .awaitTermination()
我想显示整个json并将其显示到spark控制台

我对卡夫卡的投入

{
    "a": {
       "b": 1,
       "c": 2
    }
}
spark控制台的输出应该如下所示

 {
   "a": {
          "b": 1,
          "c": 2
       }
  }

是一样的吗?或者我没看到什么:)我没法告诉你,你能详细说明一下吗@dumitruuse values.show在你的代码中values=df.select(“a*)之后