每当在Postgresql中执行任何更改事件(如Insert Update Delete)时,DEBEZIUM KAFKA连接器就会抛出一个错误
我在用Docker。我是卡夫卡连线的新手。我的用例是这样的,我有一个Postgres数据库,从那里我需要捕获关于卡夫卡主题的任何更改事件(INSERT-UPDATE-DELETE),并进一步处理它。但我被困在捕捉变化事件中。我正在浏览以下链接: 创建具有以下配置的连接器之后:每当在Postgresql中执行任何更改事件(如Insert Update Delete)时,DEBEZIUM KAFKA连接器就会抛出一个错误,postgresql,apache-kafka-connect,debezium,Postgresql,Apache Kafka Connect,Debezium,我在用Docker。我是卡夫卡连线的新手。我的用例是这样的,我有一个Postgres数据库,从那里我需要捕获关于卡夫卡主题的任何更改事件(INSERT-UPDATE-DELETE),并进一步处理它。但我被困在捕捉变化事件中。我正在浏览以下链接: 创建具有以下配置的连接器之后: {"name": "postgres-source", "config": {"connector.class":"io.de
{"name": "postgres-source",
"config": {"connector.class":"io.debezium.connector.postgresql.PostgresConnector",
"tasks.max":"1",
"database.hostname": "postgres",
"database.port": "5432",
"database.user": "postgres",
"database.password": "postgres",
"database.dbname" : "students",
"database.server.name": "dbserver15",
"database.whitelist": "students",
"database.history.kafka.bootstrap.servers": "kafka:9092",
"database.history.kafka.topic": "schema-changes.students",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"key.converter.schemas.enable": "false",
"value.converter.schemas.enable": "true",
"value.converter.schema.registry.url": "http://schema-registry:8081",
"transforms": "unwrap",
"transforms.unwrap.type": "io.debezium.transforms.UnwrapFromEnvelope"
}
}
我正在使用以下命令捕获数据库中的快照/更改事件:
kafka控制台使用者--引导服务器localhost:9092--从一开始--主题dbserver15.public.admission
它以这种格式显示数据:Struct{student_id=1,gre=337,toefl=118}
但只要我对该表执行任何INSERT-UPDATE-DELETE操作,卡夫卡连接器就会抛出以下错误:
org.apache.kafka.connect.errors.ConnectException: An exception ocurred in the change event producer. This connector will be stopped.
at io.debezium.connector.base.ChangeEventQueue.throwProducerFailureIfPresent(ChangeEventQueue.java:170)\n\tat io.debezium.connector.base.ChangeEventQueue.poll(ChangeEventQueue.java:151)
at io.debezium.connector.postgresql.PostgresConnectorTask.poll(PostgresConnectorTask.java:156)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:244)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:220)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.IllegalArgumentException: Invalid identifier:
at io.debezium.relational.TableIdParser$TableIdTokenizer.tokenize(TableIdParser.java:68)
at io.debezium.text.TokenStream.start(TokenStream.java:445)
at io.debezium.relational.TableIdParser.parse(TableIdParser.java:28)
at io.debezium.relational.TableId.parse(TableId.java:39)\n\tat io.debezium.connector.postgresql.PostgresSchema.parse(PostgresSchema.java:218)
at io.debezium.connector.postgresql.RecordsStreamProducer.process(RecordsStreamProducer.java:238)
at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$streamChanges$1(RecordsStreamProducer.java:131)
at io.debezium.connector.postgresql.connection.pgproto.PgProtoMessageDecoder.processMessage(PgProtoMessageDecoder.java:48)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.deserializeMessages(PostgresReplicationConnection.java:265)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:250)
at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:131)
at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$0(RecordsStreamProducer.java:117)
... 5 more
以下是我研究过的解决方案:
也就是说,snapshot.mode设置为exported,这允许连接器执行无锁快照。
但是当我添加“snapshot.mode”=“exported”时,会抛出一个错误,说明,
“snapshot.mode”值“exported”无效:值必须是always、never、initial\u only、initial和custom中的一个
谁能再详细一点,解释一下我遗漏了什么。我想这与配置有关。您能分享您正在捕获的表的名称吗?@Jiri Pechanec table name是:admission好的,请查看并确保您使用的是最新的Debezium和postgresimages@ZeeshanKareem你找到事情的真相了吗?我正在使用“debezium/debezium connector postgresql:1.4.0”,在添加“snapshot.mode”:“exported”到我的连接器配置后,我开始看到这些错误。您可以共享您捕获的表的名称吗?@Jiri Pechanec表名为:admissionok,请查看并确保您使用的是最新的Debezium和postgresimages@ZeeshanKareem你找到事情的真相了吗?我正在使用“debezium/debezium connector postgresql:1.4.0”,并在将“snapshot.mode”:“exported”添加到我的连接器配置后开始看到这些错误