Fatal编程技术网
  • dataframe/
  • Dataframe 遍历数据集中具有键值对数组的列，并找到一个具有最大值的对

Dataframe 遍历数据集中具有键值对数组的列，并找到一个具有最大值的对

dataframe apache-spark

Dataframe 遍历数据集中具有键值对数组的列，并找到一个具有最大值的对,dataframe,apache-spark,apache-spark-sql,apache-spark-dataset,Dataframe,Apache Spark,Apache Spark Sql,Apache Spark Dataset,我有一个数据帧中的数据，它是从azure eventhub获得的。 然后我将此数据转换为json对象，并将所需数据存储到数据集中，如下所示 +-----------------+--------------------+--------------------+--------------------+--------------------+ | NUM| SIG1| SIG2|

我有一个数据帧中的数据，它是从azure eventhub获得的。 然后我将此数据转换为json对象，并将所需数据存储到数据集中，如下所示

+-----------------+--------------------+--------------------+--------------------+--------------------+
|              NUM|                SIG1|                SIG2|                SIG3|                SIG4|
+-----------------+--------------------+--------------------+--------------------+--------------------+
|XXXXX01|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX02|[{"TIME":15695604780...|[{"TIME":15695604780...|[{"TIME":15695604780...|[{"TIME":15695604780...|
|XXXXX03|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX04|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX05|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX06|[{"TIME":15695605340...|[{"TIME":15695605340...|[{"TIME":15695605340...|[{"TIME":15695605340...|
|XXXXX07|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX08|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|

+-----------------+-----------------------------+---------------------------------------+---------------------------------------+----------------------------------------+
|              NUM|                         SIG1|                                   SIG2|                                   SIG3|                                    SIG4|
+-----------------+-----------------------------+---------------------------------------+---------------------------------------+----------------------------------------+
|XXXXX01|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":4.7825}]|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":5.7825}]|
|XXXXX02|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":6.7825}]|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":7.7825}]|
|XXXXX03|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":9.7825}]|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":8.7825}]|
用于从eventhub获取数据并将其存储到数据帧中的代码

val connectionString=ConnectionStringBuilder（）
.setEventHubName（）.build
val currTime=Instant.now
val ehConf=EventHubsConf（connectionString）
.setConsumerGroup（“”）
.设置开始位置（事件位置
.fromEnqueuedTime（currTime.减号（持续时间分钟（30）））
.setEndingPosition（EventPosition.fromEnqueuedTime（currTime））
val reader=spark.read.format（“eventhubs”）.options（ehConf.toMap.load（））
var信号=读卡器
.select（get_json_object（（$“body”）.cast（“string”），“$.NUM”）.alias（“NUM”），
get_json_object（（$“body”）.cast（“string”），“$.SIG1”）.alias（“SIG1”），
get_json_object（（$“body”）.cast（“string”），“$.SIG2”）.alias（“SIG2”），
获取对象（（$“body”）.cast（“string”），“$.SIG3”）.alias（“SIG3”），
获取json对象（（$“body”）.cast（“string”），“$.SIG4”）.alias（“SIG4”）
)
val SIGNALSFiltered=SIGNALS.filter（列（“SIG1”）。不为空&&
列（“SIG2”）.isNotNull和列（“SIG3”）.isNotNull和列（“SIG4”）.isNotNull）
信号过滤时获得的数据如下所示

+-----------------+--------------------+--------------------+--------------------+--------------------+
|              NUM|                SIG1|                SIG2|                SIG3|                SIG4|
+-----------------+--------------------+--------------------+--------------------+--------------------+
|XXXXX01|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX02|[{"TIME":15695604780...|[{"TIME":15695604780...|[{"TIME":15695604780...|[{"TIME":15695604780...|
|XXXXX03|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX04|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX05|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX06|[{"TIME":15695605340...|[{"TIME":15695605340...|[{"TIME":15695605340...|[{"TIME":15695605340...|
|XXXXX07|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX08|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|

+-----------------+-----------------------------+---------------------------------------+---------------------------------------+----------------------------------------+
|              NUM|                         SIG1|                                   SIG2|                                   SIG3|                                    SIG4|
+-----------------+-----------------------------+---------------------------------------+---------------------------------------+----------------------------------------+
|XXXXX01|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":4.7825}]|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":5.7825}]|
|XXXXX02|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":6.7825}]|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":7.7825}]|
|XXXXX03|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":9.7825}]|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":8.7825}]|
如果我们检查单个行的整个数据，它将如下所示

|XXXXX01|[{"TIME":1569560531000,"VALUE":3.7825},{"TIME":1569560475000,"VALUE":3.7812},{"TIME":1569560483000,"VALUE":3.7812},{"TIME":1569560491000,"VALUE":34.7875}]|
    [{"TIME":1569560537000,"VALUE":3.7825},{"TIME":1569560481000,"VALUE":34.7825},{"TIME":1569560489000,"VALUE":34.7825},{"TIME":1569560497000,"VALUE":34.7825}]|
    [{"TIME":1569560505000,"VALUE":34.7825},{"TIME":1569560513000,"VALUE":34.7825},{"TIME":1569560521000,"VALUE":34.7825},{"TIME":1569560527000,"VALUE":34.7825}]|
    [{"TIME":1569560535000,"VALUE":34.7825},{"TIME":1569560479000,"VALUE":34.7825},{"TIME":1569560487000,"VALUE":34.7825}]
我只需要每个列中的最高时间对，而不是整个时间值对。输出应如下所示

+-----------------+--------------------+--------------------+--------------------+--------------------+
|              NUM|                SIG1|                SIG2|                SIG3|                SIG4|
+-----------------+--------------------+--------------------+--------------------+--------------------+
|XXXXX01|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX02|[{"TIME":15695604780...|[{"TIME":15695604780...|[{"TIME":15695604780...|[{"TIME":15695604780...|
|XXXXX03|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX04|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX05|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX06|[{"TIME":15695605340...|[{"TIME":15695605340...|[{"TIME":15695605340...|[{"TIME":15695605340...|
|XXXXX07|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|
|XXXXX08|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|[{"TIME":15695605310...|

+-----------------+-----------------------------+---------------------------------------+---------------------------------------+----------------------------------------+
|              NUM|                         SIG1|                                   SIG2|                                   SIG3|                                    SIG4|
+-----------------+-----------------------------+---------------------------------------+---------------------------------------+----------------------------------------+
|XXXXX01|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":4.7825}]|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":5.7825}]|
|XXXXX02|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":6.7825}]|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":7.7825}]|
|XXXXX03|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":9.7825}]|[{"TIME":1569560531000,"VALUE":3.7825}]|[{"TIME":1569560531000,"VALUE":8.7825}]|
  • 如何遍历每行中的每一列并获得最高的时间值对
  • 在获得每列中的最高值（SIG1，….SIG4）后，必须仅更新其中最高的所有列中的时间值

  • 是否有任何方法可以转换基本数据集，如下所示？。列中的每个元素都应转换为新行

    • 您必须编写一个用户定义的函数，如下所示。这将循环您的数据并获得最大时间值。 注意：自定义项仅供参考，您可以根据需要进行更改

  • 如何遍历每行中的每一列并获得最高的时间值对
  • 在获得每列中的最高值（SIG1，….SIG4）后，必须仅更新其中最高的所有列中的时间值
    • 像上面一样编写相同的UDF，并将完整的行作为参数传递。然后将每个列值解析为Map并在所有列中获得最大值。

    无法在标记中添加Spark Dataframe:（您可以提供SIGNALSFiltered Dataframe的架构吗？
    scala>SIGNALSFiltered.printSchema根目录|--NUM:string（nullable=true）|--SIG1:string（nullable=true）|--SIG2:string（nullable=true）|--SIG3:string（nullable=true）|--SIG4:string（nullable=true）
    @Nikk@Nikk-SIGNALSFiltered是数据集的类型
    SIGNALSFiltered:org.apache.spark.sql.Dataset[org.apache.spark.sql.Row]
    刚刚更新了一个问题，还有一个要求..请检查一下！@Nikk