elasticsearch,apache-kafka,debezium,Postgresql,elasticsearch,Apache Kafka,Debezium" /> elasticsearch,apache-kafka,debezium,Postgresql,elasticsearch,Apache Kafka,Debezium" />

Postgresql Debezium Postgres和ElasticSearch-在ElasticSearch中存储复杂对象

Postgresql Debezium Postgres和ElasticSearch-在ElasticSearch中存储复杂对象,postgresql,elasticsearch,apache-kafka,debezium,Postgresql,elasticsearch,Apache Kafka,Debezium,我在Postgres中有一个数据库,其中有一个表“product”,该表与“sales\u Channel”连接1到n。因此,一个产品可以有多个销售渠道。现在我想将它转移到ES并保持最新,所以我使用debezium和kafka。将单个表转移到ES是没有问题的。我可以查询销售渠道和产品。但我需要所有销售渠道的产品。我怎样才能让debezium转移这个 产品映射 { "settings": { "number_of_shards": 1

我在Postgres中有一个数据库,其中有一个表“product”,该表与“sales\u Channel”连接1到n。因此,一个产品可以有多个销售渠道。现在我想将它转移到ES并保持最新,所以我使用debezium和kafka。将单个表转移到ES是没有问题的。我可以查询销售渠道和产品。但我需要所有销售渠道的产品。我怎样才能让debezium转移这个

产品映射

    {
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "_doc": {
      "properties": {
        "id": {
          "type": "integer"
        }
      }
    }
  }
}
   {
  "name": "es-sink-product",
  "config": {
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "1",
    "topics": "product",
    "connection.url": "http://elasticsearch:9200",
    "transforms": "unwrap,key",
    "transforms.unwrap.type": "io.debezium.transforms.UnwrapFromEnvelope",
    "transforms.unwrap.drop.tombstones": "false",
    "transforms.unwrap.drop.deletes": "false",
    "transforms.key.type": "org.apache.kafka.connect.transforms.ExtractField$Key",
    "transforms.key.field": "id",
    "key.ignore": "false",
    "type.name": "_doc",
    "behavior.on.null.values": "delete"
  }
}
产品水槽

    {
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "_doc": {
      "properties": {
        "id": {
          "type": "integer"
        }
      }
    }
  }
}
   {
  "name": "es-sink-product",
  "config": {
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "1",
    "topics": "product",
    "connection.url": "http://elasticsearch:9200",
    "transforms": "unwrap,key",
    "transforms.unwrap.type": "io.debezium.transforms.UnwrapFromEnvelope",
    "transforms.unwrap.drop.tombstones": "false",
    "transforms.unwrap.drop.deletes": "false",
    "transforms.key.type": "org.apache.kafka.connect.transforms.ExtractField$Key",
    "transforms.key.field": "id",
    "key.ignore": "false",
    "type.name": "_doc",
    "behavior.on.null.values": "delete"
  }
}

您需要使用发件箱模式,请参见

也可以使用聚合对象,请参见