Json Mongodb debezium连接器将Rowkey填充为0(零),并将Key填充为0零,即使数据包含有效值
我在mongo db中加载了一个示例Json文件,该文件将使用debezium连接器发送给Kafka connect。我的问题是,即使原始json输入文件中有有效数据,RowKey和id key列值如何转换为0(零)?以下是产生此错误的步骤 步骤1:创建示例json文件Json Mongodb debezium连接器将Rowkey填充为0(零),并将Key填充为0零,即使数据包含有效值,json,key,debezium,ksqldb,Json,Key,Debezium,Ksqldb,我在mongo db中加载了一个示例Json文件,该文件将使用debezium连接器发送给Kafka connect。我的问题是,即使原始json输入文件中有有效数据,RowKey和id key列值如何转换为0(零)?以下是产生此错误的步骤 步骤1:创建示例json文件 {"id":1,"first_name":"Devinne","last_name":"Stubbe","email":"dstubbe0@java.com","gender":"Female","club_status":"p
{"id":1,"first_name":"Devinne","last_name":"Stubbe","email":"dstubbe0@java.com","gender":"Female","club_status":"platinum","comments":"Extended interactive initiative"}
{"id":2,"first_name":"Loise","last_name":"Salt","email":"lsalt1@apple.com","gender":"Female","club_status":"silver","comments":"Optimized neutral standardization"}
{"id":3,"first_name":"Benjamen","last_name":"Spittle","email":"bspittle2@howstuffworks.com","gender":"Male","club_status":"platinum","comments":"De-engineered systemic customer loyalty"}
{"id":4,"first_name":"Helena","last_name":"Pogosian","email":"hpogosian3@pagesperso-orange.fr","gender":"Female","club_status":"gold","comments":"Phased hybrid definition"}
{"id":5,"first_name":"Miller","last_name":"Karolewski","email":"mkarolewski4@g.co","gender":"Male","club_status":"platinum","comments":"Monitored systematic software"}
在customerprofile集合的mongodb中加载此数据
rs0:PRIMARY> db.collection.find();
rs0:PRIMARY> db.customerprofile.find();
{ "_id" : ObjectId("5b6213ba78aae5dc09ef8b5b"), "id" : 1, "first_name" :
"Devinne", "last_name" : "Stubbe", "email" : "dstubbe0@java.com", "gender" :
"Female", "club_st
atus" : "platinum", "comments" : "Extended interactive initiative" }
{ "_id" : ObjectId("5b6213ba78aae5dc09ef8b5c"), "id" : 2, "first_name" :
"Loise", "last_name" : "Salt", "email" : "lsalt1@apple.com", "gender" :
"Female", "club_status"
: "silver", "comments" : "Optimized neutral standardization" }
{ "_id" : ObjectId("5b6213ba78aae5dc09ef8b5d"), "id" : 3, "first_name" :
"Benjamen", "last_name" : "Spittle", "email" : "bspittle2@howstuffworks.com",
"gender" : "Male"
, "club_status" : "platinum", "comments" : "De-engineered systemic customer
loyalty" }
{ "_id" : ObjectId("5b6213ba78aae5dc09ef8b5e"), "id" : 4, "first_name" :
"Helena", "last_name" : "Pogosian", "email" : "hpogosian3@pagesperso-
orange.fr", "gender" : "Fe
male", "club_status" : "gold", "comments" : "Phased hybrid definition" }
{ "_id" : ObjectId("5b6213ba78aae5dc09ef8b5f"), "id" : 5, "first_name" :
"Miller", "last_name" : "Karolewski", "email" : "mkarolewski4@g.co", "gender"
: "Male", "club_status" : "platinum", "comments" : "Monitored systematic
software" }
{ "_id" : ObjectId("5b6213ba78aae5dc09ef8b60"), "id" : 6, "first_name" :
"Cammy", "last_name" : "Suche", "email" : "csuche5@cbsnews.com", "gender" :
"Male", "club_status" : "gold", "comments" : "Inverse client-server alliance"
}
步骤2:创建mongodb debezium连接器配置并激活连接器
在卡夫卡连接
{
"name": "mongodb-jsondb-connector",
"config":{
"connector.class": "io.debezium.connector.mongodb.MongoDbConnector",
"mongodb.hosts": "rs0/169.254.57.118:27017",
"mongodb.name": "jsondb",
"mongodb.members.auto.discover": "false",
"database.whitelist": "jsondb",
"transforms": "unwrap",
"transforms.unwrap.type":
"io.debezium.connector.mongodb.transforms.UnwrapFromMongoDbEnvelope",
"database.history.kafka.topic": "schema-changes.jsondb",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable": "false",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false"
}
}
步骤3:检查主题队列
ksql> print 'jsondb.jsondb.customerprofile' from beginning;
Format:JSON
{"ROWTIME":1533154567091,"ROWKEY":"
CREATE STREAM customers_profile \
(id integer, first_name string, last_name string, \
email string, gender string, club_status string, \
comments string) WITH \
(KAFKA_TOPIC='jsondb.jsondb.customerprofile',VALUE_FORMAT='json');
正如您在下面的输出中所看到的,通过id列使用
客观的
\"id\":\"5b6213ba78aae5dc09ef8b5b\"}","id":"5b6213ba78aae5dc09ef8b5b".
****我的问题是,由于我们使用JSONConverter作为密钥转换器,我的id列如何用串行id密钥更新?***
{\"id\":\"5b6213ba78aae5dc09ef8b5b\"}","id":"5b6213ba78aae5dc09ef8b5b","first_name":"Devinne","last_name":"Stubbe","email":"dstubbe0@java.com","gender":"Female","club_status":"platinum","comments":"Extended interactive initiative"}
{"ROWTIME":1533154567096,"ROWKEY":"
{\"id\":\"5b6213ba78aae5dc09ef8b5c\"}","id":"5b6213ba78aae5dc09ef8b5c","first_name":"Loise","last_name":"Salt","email":"lsalt1@apple.com","gender":"Female","club_status":"silver","comments":"Optimized neutral standardization"}
{"ROWTIME":1533154567096,"ROWKEY":"
{\"id\":\"5b6213ba78aae5dc09ef8b5d\"}","id":"5b6213ba78aae5dc09ef8b5d","first_name":"Benjamen","last_name":"Spittle","email":"bspittle2@howstuffworks.com","gender":"Male","club_status":"platinum","comments":"De-engineered systemic customer loyalty"}
{"ROWTIME":1533154567097,"ROWKEY":"{\"id\":\"5b6213ba78aae5dc09ef8b5e\"}","id":"5b6213ba78aae5dc09ef8b5e","first_name":"Helena","last_name":"Pogosian","email":"hpogosian3@pagesperso-orange.fr","gender":"Female","club_status":"gold","comments":"Phased hybrid definition"}
{"ROWTIME":1533154567097,"ROWKEY":"{\"id\":\"5b6213ba78aae5dc09ef8b5f\"}","id":"5b6213ba78aae5dc09ef8b5f","first_name":"Miller","last_name":"Karolewski","email":"mkarolewski4@g.co","gender":"Male","club_status":"platinum","comments":"Monitored systematic software"}
{"ROWTIME":1533154567099,"ROWKEY":"{\"id\":\"5b6213ba78aae5dc09ef8b60\"}","id":"5b6213ba78aae5dc09ef8b60","first_name":"Cammy","last_name":"Suche","email":"csuche5@cbsnews.com","gender":"Male","club_status":"gold","comments":"Inverse client-server alliance"}
步骤4:从主题队列创建流
ksql> print 'jsondb.jsondb.customerprofile' from beginning;
Format:JSON
{"ROWTIME":1533154567091,"ROWKEY":"
CREATE STREAM customers_profile \
(id integer, first_name string, last_name string, \
email string, gender string, club_status string, \
comments string) WITH \
(KAFKA_TOPIC='jsondb.jsondb.customerprofile',VALUE_FORMAT='json');
步骤5:这是流从CUSTOMER_REPART接收零(0)的地方
ksql> CREATE STREAM customers_stream WITH
(KAFKA_TOPIC='CUSTOMERS_REPART',VALUE_FORMAT='json',PARTITIONS=1) as SELECT *
FROM customers_profile PARTITION BY id;
步骤6:请检查此打印命令的输出,其中显示零
在ROWKEY和id key列值中。他们从哪里来的?我需要什么
要执行此操作,请确保ROWKEY和id key列显示来自原始数据的正确值
最初在mongodb中加载的json文件值
ksql> SET 'auto.offset.reset' = 'earliest';
Successfully changed local property 'auto.offset.reset' from 'null' to
'earliest'
ksql> print 'CUSTOMERS_REPART' from beginning;
Format:JSON
{“ROWTIME”:1533154567091,“ROWKEY”:“0”,“俱乐部状态”:“白金”,“性别”:“女性”,“评论”:“扩展互动倡议”,“ID”:0,“姓氏”:“Stubbe”,“电子邮件”:“dstubbe0”@
{“ROWTIME”:1533154567096,“ROWKEY”:“0”,“俱乐部状态”:“银牌”,“性别”:“女性”,“评论”:“优化中性标准化”,“ID”:0,“姓氏”:“Salt”,“电子邮件”:lsalt1@appl
{“ROWTIME”:1533154567096,“ROWKEY”:“0”,“俱乐部状态”:“白金”,“性别”:“男性”,“评论”:“非工程化系统客户忠诚度”,“ID”:0,“姓氏”:“Spitle”,“EMAIL”:“bs”:“Benjamen”}
{“ROWTIME”:1533154567097,“ROWKEY”:“0”,“俱乐部状态”:“金牌”,“性别”:“女性”,“评论”:“分阶段混合定义”,“ID”:0,“姓氏”:“波戈森”,“电子邮件”:hpogosian3@pagespe
{“ROWTIME”:1533154567097,“ROWKEY”:“0”,“俱乐部状态”:“白金”,“性别”:“男性”,“评论”:“监控系统软件”,“ID”:0,“姓氏”:“Karolewski”,“电子邮件”:“mkarolews”
{“ROWTIME”:1533154567099,“ROWKEY”:“0”,“俱乐部状态”:“金牌”,“性别”:“男性”,“评论”:“反向客户端-服务器联盟”,“ID”:0,“姓氏”:“Suche”,“电子邮件”:csuche5@cbsnews.c
{“ROWTIME”:1533154567104,“ROWKEY”:“0”,“俱乐部状态”:“白金”,“性别”:“男性”,“评论”:“分阶段统一软件”,“ID”:0,“姓氏”:“Suttling”,“电子邮件”:msuttling7@baidu.
{“ROWTIME”:1533154567104,“ROWKEY”:“0”,“俱乐部状态”:“金牌”,“性别”:“女性”,“评论”:“注重质量的全球软件”,“ID”:0,“姓氏”:“Stepney”,“电子邮件”:estepney8@we关于你的第一个问题,我想你看到了 关于第二个问题,我认为问题是id integer,因为id现在包含字符串