avro模式中的时间戳在Kafka Connect JDBC中产生不兼容的值验证
JDBC接收器连接器产生错误:avro模式中的时间戳在Kafka Connect JDBC中产生不兼容的值验证,jdbc,apache-kafka,apache-kafka-connect,confluent-platform,Jdbc,Apache Kafka,Apache Kafka Connect,Confluent Platform,JDBC接收器连接器产生错误: org.apache.kafka.connect.errors.DataException: Invalid Java object for schema type INT64: class java.util.Date for field: "some_timestamp_field" at org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:242) at o
org.apache.kafka.connect.errors.DataException: Invalid Java object for schema type INT64: class java.util.Date for field: "some_timestamp_field"
at org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:242)
at org.apache.kafka.connect.data.Struct.put(Struct.java:216)
at org.apache.kafka.connect.transforms.Cast.applyWithSchema(Cast.java:151)
at org.apache.kafka.connect.transforms.Cast.apply(Cast.java:107)
at org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:38)
at org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:480)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:301)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:205)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:173)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
源JDBC连接器(MySQL)注册的avro模式:
看起来异常是由以下代码块引起的:
因此,在avro模式中,timestamp字段注册为具有正确(timestamp)逻辑类型的INT64。但是connect将模式类型读取为INT64
,并将其与值类型java.util.Date
进行比较
这是一个bug,还是有解决方法?可能是我遗漏了一些东西,因为这看起来像一个标准的连接模型
提前谢谢
更新
接收器连接器配置:
{
"name": "sink",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"topics": "topic",
"connection.url": "jdbc:postgresql://host:port/db",
"connection.user": "user",
"connection.password": "password",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://host:port",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://host:port",
"auto.create": "true",
"insert.mode": "upsert",
"pk.mode": "record_value",
"pk.fields": "id"
}
}
卡夫卡中的反序列化数据:
{
"id":678148,
"some_timestamp_field":1543806057000,
...
}
我们已经为这个问题制定了一个解决办法。我们的目标是将id从BIGINT转换为STRING(TEXT/VARCHAR),并将记录保存在下游数据库中 但由于一个问题(可能),强制转换id字段不起作用。Kafka试图验证转换链中的时间戳字段,但读取模式类型/名称时出错,导致类型不匹配(请参阅上面的记录正文和错误日志) 因此,我们做了如下工作:
仅将id字段提取为键
->对键执行强制转换
->它的工作原理是键不包含时间戳字段
以下是变通配置:
{
"name": "sink",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"topics": "topic",
"connection.url": "jdbc:postgresql://host:port/db",
"connection.user": "user",
"connection.password": "password",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://host:port",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://host:port",
"transforms": "createKey,castKeyToString",
"transforms.createKey.type": "org.apache.kafka.connect.transforms.ValueToKey",
"transforms.createKey.fields": "id",
"transforms.castKeyToString.type": "org.apache.kafka.connect.transforms.Cast$Key",
"transforms.castKeyToString.spec": "id:string",
"auto.create": "true",
"insert.mode": "upsert",
"pk.mode": "record_key",
"pk.fields": "id"
}
}
免责声明:这不是一个合适的解决方案,只是一种变通方法。转换转换中的错误应该被修复。在我看来,强制转换应该只关注指定用于强制转换的字段,而不是消息中的其他字段
祝您有愉快的一天。您能展示示例主题数据吗?您的连接器配置?添加了主题数据和连接器配置。您正在运行哪个Kafka connect或Confluent版本?您粘贴的链接是针对Confluent Platform的未升级版本(尽管,我认为最近代码部分没有更改),我们正在使用随
kafka 1.1.0
附带的kafka connect。对于schema registry,我们使用的是confluent-oss-4.0.0
附带的可执行文件。从kafka connect jdbc开始,将使用版本kafka-connect-jdbc-5.2.0-SNAPSHOT
。@cricket_007,这是更新,当我们删除强制转换时,错误消失了。我们很可能面临这个问题:。有什么想法吗?
{
"name": "sink",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"topics": "topic",
"connection.url": "jdbc:postgresql://host:port/db",
"connection.user": "user",
"connection.password": "password",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://host:port",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://host:port",
"transforms": "createKey,castKeyToString",
"transforms.createKey.type": "org.apache.kafka.connect.transforms.ValueToKey",
"transforms.createKey.fields": "id",
"transforms.castKeyToString.type": "org.apache.kafka.connect.transforms.Cast$Key",
"transforms.castKeyToString.spec": "id:string",
"auto.create": "true",
"insert.mode": "upsert",
"pk.mode": "record_key",
"pk.fields": "id"
}
}