Spring Kafka Connect如何使命名空间与数据库名称无关
我的环境 MySQL(5.7):我们有多个模式,命名约定是{application\u name}\u envSpring Kafka Connect如何使命名空间与数据库名称无关,spring,apache-kafka,spring-kafka,apache-kafka-connect,debezium,Spring,Apache Kafka,Spring Kafka,Apache Kafka Connect,Debezium,我的环境 MySQL(5.7):我们有多个模式,命名约定是{application\u name}\u env 示例:考虑我们有两个应用程序:APP1和APP2 Dev环境:数据库名称为app1\u Dev、app2\u Dev QA环境:数据库名称为app1_QA、app2_QA 脱苄(0.8.3)。该插件用于CDC MySQL日志 连接器配置为: { "name": "connector-1", "config": { "connector.class": "io.debezium.
示例:考虑我们有两个应用程序:APP1和APP2
Dev环境:数据库名称为app1\u Dev、app2\u Dev QA环境:数据库名称为app1_QA、app2_QA 脱苄(0.8.3)。该插件用于CDC MySQL日志 连接器配置为:{
"name": "connector-1",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"tasks.max": "1",
"database.hostname": "mysql",
"database.port": "3306",
"database.user": "debezium",
"database.password": "dbz",
"database.server.id": "184054",
"database.server.name": "dbserver1",
"database.history.kafka.bootstrap.servers": "kafka:9092",
"database.history.kafka.topic": "schema-changes.inventory",
"decimal.handling.mode": "double",
"snapshot.mode": "when_needed",
"table.whitelist":"{database_name}.account",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://schema-registry:8081",
"value.converter.schema.registry.url": "http://schema-registry:8081",
"transforms" : "setSchema",
"transforms.setSchema.type" : "org.apache.kafka.connect.transforms.SetSchemaMetadata$Value",
"transforms.setSchema.schema.name" : "com.test.Account"
}
}
Spring Java应用程序我正在使用Kafka Consumer(@KafkaListener)读取Debezium事件中的更改
我提供了avsc文件,并使用gradle avro插件生成类
来自开发环境的架构
{
"type":"record",
"name":"Accounts",
"namespace":"com.test",
"fields":[
{
"name":"before",
"type":[
"null",
{
"type":"record",
"name":"Value",
"namespace":"dbserver1.app1_dev.account",
"fields":[
{
"name":"id",
"type":"long"
}
],
"connect.name":"dbserver1.app1_dev.account.Value"
}
],
"default":null
},
{
"name":"after",
"type":[
"null",
"dbserver1.app1_dev.account.Value"
],
"default":null
},
{
"name":"source",
"type":{
"type":"record",
"name":"Source",
"namespace":"io.debezium.connector.mysql",
"fields":[
{
"name":"version",
"type":[
"null",
"string"
],
"default":null
},
{
"name":"name",
"type":"string"
},
{
"name":"server_id",
"type":"long"
},
{
"name":"ts_sec",
"type":"long"
},
{
"name":"gtid",
"type":[
"null",
"string"
],
"default":null
},
{
"name":"file",
"type":"string"
},
{
"name":"pos",
"type":"long"
},
{
"name":"row",
"type":"int"
},
{
"name":"snapshot",
"type":[
{
"type":"boolean",
"connect.default":false
},
"null"
],
"default":false
},
{
"name":"thread",
"type":[
"null",
"long"
],
"default":null
},
{
"name":"db",
"type":[
"null",
"string"
],
"default":null
},
{
"name":"table",
"type":[
"null",
"string"
],
"default":null
},
{
"name":"query",
"type":[
"null",
"string"
],
"default":null
}
],
"connect.name":"io.debezium.connector.mysql.Source"
}
},
{
"name":"op",
"type":"string"
},
{
"name":"ts_ms",
"type":[
"null",
"long"
],
"default":null
}
],
"connect.name":"com.test.Account"
}
问题:
因为我的数据库模式是动态的,即它们以env后缀结尾
在每个环境中生成的架构具有不同的命名空间
Dev:Dev.app1u Dev.accounts
QA:dev.app1u QA.accounts
由于名称空间不同,我无法在QA中反序列化我的开发代码。因此,如果使用在Dev中生成的模式,代码将无法在QA中工作
我想确保名称空间在所有环境中都是一致的。请使用
org.apache.kafka.connect.transforms.SetSchemaMetadata
SMT-请参见请使用org.apache.kafka.connect.transforms.SetSchemaMetadata
SMT-请参见我已经在使用的:)问题是嵌套级别的,名称空间不是正在更改。我已经在使用:)问题是在嵌套级别,命名空间没有更改。您能解决这个问题吗?如果您有,我将非常感谢您的指导。谢谢你能解决这个问题吗?如果您有,我将非常感谢您的指导。谢谢