Kafka将mongodb的流数据连接到elasticsearch

我正在尝试使用kafka connect将数据从mongodb流式传输到elasticsearch

mongodb连接器流入kafka的数据如下所示

{
   "updatedAt" : {
      "$date" : 1591596275939
   },
   "createdAt" : {
      "$date" : 1362162600000
   },
   "name" : "my name",
   "_id" : {
      "$oid" : "5ee0cc7e0c3273f3d4a3c20f"
   },
   "documentId" : "mydoc1",
   "age" : 20,
   "language" : "English",
   "validFrom" : {
      "$date" : 978307200000
   },
   "remarks" : [
      "remarks"
   ],
   "married" : false
}

我在将数据保存到elasticsearch时遇到以下两个问题

_id是一个对象，我想在elasticsearch中使用“documentId”键作为_id

日期是一个带有$date键的对象，我不知道如何将其转换为正常日期

关于以上两个问题，谁能给我指一下正确的方向

Mongodb源配置

{
   "tasks.max" : "5",
   "change.stream.full.document" : "updateLookup",
   "name" : "mongodb-source",
   "value.converter" : "org.apache.kafka.connect.storage.StringConverter",
   "collection" : "collection",
   "poll.max.batch.size" : "1000",
   "connector.class" : "com.mongodb.kafka.connect.MongoSourceConnector",
   "batch.size" : "1000",
   "key.converter" : "org.apache.kafka.connect.storage.StringConverter",
   "key.converter.schemas.enable":"false",
   "value.converter.schemas.enable":"false",
   "connection.uri" : "mongodb://connection",
   "publish.full.document.only" : "true",
   "database" : "databasename",
   "poll.await.time.ms" : "5000",
   "topic.prefix" : "mongodb"
}

{
   "write.method" : "upsert",
   "errors.deadletterqueue.context.headers.enable" : "true",
   "name" : "elasticsearch-sink",
   "connection.password" : "password",
   "topic.index.map" : "mongodb.databasename.collection:elasticindexname",
   "connection.url" : "http://localhost:9200",
   "errors.log.enable" : "true",
   "flush.timeout.ms" : "20000",
   "errors.log.include.messages" : "true",
   "key.ignore" : "false",
   "type.name" : "_doc",
   "key.converter" : "org.apache.kafka.connect.json.JsonConverter",
   "value.converter" : "org.apache.kafka.connect.json.JsonConverter",
   "key.converter.schemas.enable":"false",
   "value.converter.schemas.enable":"false",
   "tasks.max" : "1",
   "batch.size" : "100",
   "schema.ignore" : "true",
   "schema.enable" : "false",
   "connector.class" : "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
   "read.timeout.ms" : "6000",
   "connection.username" : "elastic",
   "topics" : "mongodb.databasename.collection",
   "proxy.host": "localhost",
   "proxy.port": "8080"
}

弹性水槽配置

{
   "tasks.max" : "5",
   "change.stream.full.document" : "updateLookup",
   "name" : "mongodb-source",
   "value.converter" : "org.apache.kafka.connect.storage.StringConverter",
   "collection" : "collection",
   "poll.max.batch.size" : "1000",
   "connector.class" : "com.mongodb.kafka.connect.MongoSourceConnector",
   "batch.size" : "1000",
   "key.converter" : "org.apache.kafka.connect.storage.StringConverter",
   "key.converter.schemas.enable":"false",
   "value.converter.schemas.enable":"false",
   "connection.uri" : "mongodb://connection",
   "publish.full.document.only" : "true",
   "database" : "databasename",
   "poll.await.time.ms" : "5000",
   "topic.prefix" : "mongodb"
}

{
   "write.method" : "upsert",
   "errors.deadletterqueue.context.headers.enable" : "true",
   "name" : "elasticsearch-sink",
   "connection.password" : "password",
   "topic.index.map" : "mongodb.databasename.collection:elasticindexname",
   "connection.url" : "http://localhost:9200",
   "errors.log.enable" : "true",
   "flush.timeout.ms" : "20000",
   "errors.log.include.messages" : "true",
   "key.ignore" : "false",
   "type.name" : "_doc",
   "key.converter" : "org.apache.kafka.connect.json.JsonConverter",
   "value.converter" : "org.apache.kafka.connect.json.JsonConverter",
   "key.converter.schemas.enable":"false",
   "value.converter.schemas.enable":"false",
   "tasks.max" : "1",
   "batch.size" : "100",
   "schema.ignore" : "true",
   "schema.enable" : "false",
   "connector.class" : "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
   "read.timeout.ms" : "6000",
   "connection.username" : "elastic",
   "topics" : "mongodb.databasename.collection",
   "proxy.host": "localhost",
   "proxy.port": "8080"
}

例外情况

Caused by: org.apache.kafka.connect.errors.DataException: MAP is not supported as the document id.
        at io.confluent.connect.elasticsearch.DataConverter.convertKey(DataConverter.java:107)
        at io.confluent.connect.elasticsearch.DataConverter.convertRecord(DataConverter.java:182)
        at io.confluent.connect.elasticsearch.ElasticsearchWriter.tryWriteRecord(ElasticsearchWriter.java:291)
        at io.confluent.connect.elasticsearch.ElasticsearchWriter.write(ElasticsearchWriter.java:276)
        at io.confluent.connect.elasticsearch.ElasticsearchSinkTask.put(ElasticsearchSinkTask.java:174)
        at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:538)
        ... 10 more

连接器链接：

https://docs.mongodb.com/kafka-connector/master/kafka-source/
https://docs.confluent.io/current/connect/kafka-connect-elasticsearch

---

@一位板球运动员，我已经在问题中更新了，谢谢。连接服务器或连接器是否有错误日志？您在主题中看到了哪些数据？您是否考虑过添加Kafka Streams或KSQL来处理数据，而不仅仅是从一个数据库到另一个数据库？我也添加了例外。关于kafka streams/KSQL，我希望避免任何复杂性。我希望它是一个简单的卡夫卡连接设置的生产者->卡夫卡->消费者。非常普遍，它从来没有那么容易，unfortunately@OneCricketeer我已经在问题中更新了，谢谢。连接服务器或连接器是否有错误日志？您在主题中看到了哪些数据？您是否考虑过添加Kafka Streams或KSQL来处理数据，而不仅仅是从一个数据库到另一个数据库？我也添加了例外。关于kafka streams/KSQL，我希望避免任何复杂性。我希望它是一个简单的卡夫卡连接设置的生产者->卡夫卡->消费者。非常普遍，它从来没有那么容易，不幸的是