Java ApacheKafka-实现KTable
我是Kafka Streams API的新手,我正在尝试创建一个KTable。我有一个输入主题:Java ApacheKafka-实现KTable,java,apache-kafka,apache-kafka-streams,ktable,Java,Apache Kafka,Apache Kafka Streams,Ktable,我是Kafka Streams API的新手,我正在尝试创建一个KTable。我有一个输入主题:s-order-topic,它是一个json格式的消息,如下所示 { "current_ts": "2019-12-24 13:16:40.316952", "primary_keys": ["ID"], "before": null, "tokens": {"txid":"3.17.2493", "csn":"64913009"}, "op_type":"
s-order-topic
,它是一个json格式的消息,如下所示
{ "current_ts": "2019-12-24 13:16:40.316952",
"primary_keys": ["ID"],
"before": null,
"tokens": {"txid":"3.17.2493",
"csn":"64913009"},
"op_type":"I",
"after": { "CODE":"AAAA41",
"STATUS":"COMPLETED",
"ID":24},
"op_ts":"2019-12-24 13:16:40.316941",
"table":"S_ORDER"}
我阅读了本主题中的消息,我想创建一个KTable,该表包含as键、字段“之后”:“ID”
,以及值字段(“ID”
)之后)中的所有字段(除了“ID”
)
只有在使用默认聚合函数(即count)时,我才成功创建了KTable。但是我很难创建自己的聚合函数。下面我展示了我试图创建KTable的代码部分
KTable<Long, String> s_table = builder.stream("s-order-topic", Consumed.with(Serdes.Long(),Serdes.String()))
.mapValues(value -> {
String time;
JSONObject json = new JSONObject(value);
if (json.getString("op_type").equals("I")) {
time = "after";
}else {
time = "before";
}
JSONObject json2 = new JSONObject(json.getJSONObject(time).toString());
return json2.toString();
})
.groupBy((key, value) -> {
JSONObject json = new JSONObject(value);
return json.getLong("ID");
}, Grouped.with(Serdes.Long(), Serdes.String()))
.aggregate( ... );
KTable s_table=builder.stream(“s-order-topic”,consumered.with(Serdes.Long(),Serdes.String())
.mapValues(值->{
串时间;
JSONObject json=新的JSONObject(值);
if(json.getString(“op_type”).equals(“I”)){
time=“after”;
}否则{
time=“before”;
}
JSONObject json2=新的JSONObject(json.getJSONObject(time.toString());
返回json2.toString();
})
.groupBy((键、值)->{
JSONObject json=新的JSONObject(值);
返回json.getLong(“ID”);
},Grouped.with(Serdes.Long(),Serdes.String())
.合计(……);
如何实现这个KTable
我处理问题的方法正确吗
(mapValues->只保留“before”/“after”字段。groupBy->将ID作为消息的键。Aggregate->?)我想出了一个解决方案。我实现了KTable,如下所示:
KTable<String, String> s_table = builder.stream("s-order-topic", Consumed.with(Serdes.String(),Serdes.String()))
.mapValues(value -> {
String time;
JSONObject json = new JSONObject(value);
if (json.getString("op_type").equals("I")) {
time = "after";
}else {
time = "before";
}
JSONObject json2 = new JSONObject(json.getJSONObject(time).toString());
return json2.toString();
})
.groupBy((key, value) -> {
JSONObject json = new JSONObject(value);
return String.valueOf(json.getLong("ID"));
}, Grouped.with(Serdes.String(), Serdes.String()))
.reduce((prev,newval)->newval);
15 {"CODE":"AAAA17","STATUS":"PENDING","ID":15}
18 {"CODE":"AAAA50","STATUS":"SUBMITTED","ID":18}
4 {"CODE":"AAAA80","STATUS":"SUBMITTED","ID":4}
19 {"CODE":"AAAA83","STATUS":"SUBMITTED","ID":19}
18 {"CODE":"AAAA33","STATUS":"COMPLETED","ID":18}
5 {"CODE":"AAAA38","STATUS":"PENDING","ID":5}
10 {"CODE":"AAAA1","STATUS":"COMPLETED","ID":10}
3 {"CODE":"AAAA68","STATUS":"NOT COMPLETED","ID":3}
9 {"CODE":"AAAA89","STATUS":"PENDING","ID":9}
只是一个旁注:聚合是reduce()
的泛化,允许您更改输出类型(对于reduce,输入值类型==输出值类型)。因此,使用聚合实际上也应该起作用。@MatthiasJ.Sax感谢您的更正!因此,我的问题应该是,我没有为我的案例正确地实现聚合。