Apache kafka Kafka中KSQL操作的主题数据格式
我刚开始使用ksql,当我从一开始就打印主题时,我得到了以下格式的数据Apache kafka Kafka中KSQL操作的主题数据格式,apache-kafka,confluent-platform,ksqldb,Apache Kafka,Confluent Platform,Ksqldb,我刚开始使用ksql,当我从一开始就打印主题时,我得到了以下格式的数据 rowtime: 4/12/20, 9:00:05 AM MDT, key: {"messageId":null}, value: {"WHS":[{"Character Set":"UTF-8","action":"finished","Update-Date-Time":"2020-04-11 09:00:02:25","Number":0,"Abbr":"","Name":"","Name2":"","Count
rowtime: 4/12/20, 9:00:05 AM MDT, key: {"messageId":null}, value: {"WHS":[{"Character Set":"UTF-8","action":"finished","Update-Date-Time":"2020-04-11 09:00:02:25","Number":0,"Abbr":"","Name":"","Name2":"","Country-Code":"","Addr-1":"","Addr-2":"","Addr-3":"","Addr-4":"","City":"","State":""}]}
但是KSQL中的所有示例都有以下格式的数据
{"ROWTIME":1537436551210,"ROWKEY":"3375","rating_id":3375,"user_id":2,"stars":3,"route_id":6972,"rating_time":1537436551210,"channel":"web","message":"airport refurb looks great, will fly outta here more!"}
所以我无法执行任何操作,格式显示为
Key format: JSON or SESSION(KAFKA_STRING) or HOPPING(KAFKA_STRING) or TUMBLING(KAFKA_STRING) or KAFKA_STRING
Value format: JSON or KAFKA_STRING
关于我的话题。如何将数据修改为特定格式
谢谢,ksqlDB还不支持JSON消息键(请参阅跟踪) 但是,您仍然可以访问键和值中的数据。JSON键毕竟只是一个字符串 重新格式化后的值如下所示:
{
"WHS":[
{
"Character Set":"UTF-8",
"action":"finished",
"Update-Date-Time":"2020-04-11 09:00:02:25",
"Number":0,
"Abbr":"",
"Name":"",
"Name2":"",
"Country-Code":"",
"Addr-1":"",
"Addr-2":"",
"Addr-3":"",
"Addr-4":"",
"City":"",
"State":""
}
]
}
假设所有行共享一个公共格式,ksqlDB可以轻松地处理
要导入流,您应该能够运行如下操作:
{
"WHS":[
{
"Character Set":"UTF-8",
"action":"finished",
"Update-Date-Time":"2020-04-11 09:00:02:25",
"Number":0,
"Abbr":"",
"Name":"",
"Name2":"",
"Country-Code":"",
"Addr-1":"",
"Addr-2":"",
"Addr-3":"",
"Addr-4":"",
"City":"",
"State":""
}
]
}
——假设卡夫卡的v0.9
创建流内容
(
行键字符串键,
WHS阵列<
结构<
`字符集`字符串,
动作串,
`更新日期时间字符串,
数字字符串,
等
>
>
)
使用(kafka_topic='?',value_format='JSON');
值列
WHS
是一个结构数组(其中只有一个元素),结构定义了需要访问的所有字段。请注意,有些字段名需要引用,因为它们包含无效字符,例如空格和破折号。我尝试以Avro格式创建Value_格式的流,但仍然不起作用。