Apache kafka 如何让RabbitMQ Kafka连接器生成的字节数组数据获得密钥？

我使用官方的Kafka RabbitMQ连接器从RabbitMQ实例获取JSOn消息

当前，此连接器的配置如下所示

name=RabbitMQConsentConsumer
connector.class=io.confluent.connect.rabbitmq.RabbitMQSourceConnector
rabbitmq.host=HOST
rabbitmq.port=PORT
rabbitmq.username=USER
rabbitmq.password=PASSWORD
rabbitmq.virtual.host=vhost
tasks.max=1
kafka.topic=TOPIC
rabbitmq.queue=QUEUE
errors.log.enable=true
errors.log.include.messages=true
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
key.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter.schemas.enable=false

使用该配置，我可以将JSON消息作为字节数组注入Kafka，并被读者识别为JSON内容。这“相当”好

但我想从Kafka Connect中的JSON内容生成密钥。 为此，我想使用变压器

所以我开始玩变形金刚。。。我添加了以下配置

transforms=MakeMap
transforms.MakeMap.type=org.apache.kafka.connect.transforms.HoistField$Value
transforms.MakeMap.fields=jsonEvent

不幸的是，它失败了，并显示以下消息

[2019-04-25 08:04:25,128] ERROR Error encountered in task RabbitMQConsentConsumer-0. Executing stage 'VALUE_CONVERTER' with class 'org.apache.kafka.connect.converters.ByteArrayConverter', where source record is = SourceRecord{sourcePartition={routingKey=TOPIC}, sourceOffset={deliveryTag=1}} ConnectRecord{topic='TOPIC', kafkaPartition=null, key=null, keySchema=Schema{STRING}, value=[B@20a106a8, valueSchema=Schema{BYTES}, timestamp=1556179465126, headers=ConnectHeaders(headers=[ConnectHeader(key=rabbitmq.consumer.tag, value=amq.ctag-7PHmHnIJF2WCnxJShauM3A, schema=Schema{STRING}), ConnectHeader(key=rabbitmq.content.type, value=null, schema=Schema{STRING}), ConnectHeader(key=rabbitmq.content.encoding, value=null, schema=Schema{STRING}), ConnectHeader(key=rabbitmq.delivery.mode, value=1, schema=Schema{INT32}), ConnectHeader(key=rabbitmq.priority, value=0, schema=Schema{INT32}), ConnectHeader(key=rabbitmq.correlation.id, value=null, schema=Schema{STRING}), ConnectHeader(key=rabbitmq.reply.to, value=null, schema=Schema{STRING}), ConnectHeader(key=rabbitmq.expiration, value=null, schema=Schema{STRING}), ConnectHeader(key=rabbitmq.message.id, value=null, schema=Schema{STRING}), ConnectHeader(key=rabbitmq.timestamp, value=null, schema=Schema{org.apache.kafka.connect.data.Timestamp:INT64}), ConnectHeader(key=rabbitmq.type, value=null, schema=Schema{STRING}), ConnectHeader(key=rabbitmq.user.id, value=null, schema=Schema{STRING}), ConnectHeader(key=rabbitmq.app.id, value=null, schema=Schema{STRING}), ConnectHeader(key=rabbitmq.delivery.tag, value=1, schema=Schema{INT64}), ConnectHeader(key=rabbitmq.redeliver, value=false, schema=Schema{BOOLEAN}), ConnectHeader(key=rabbitmq.exchange, value=, schema=Schema{STRING}), ConnectHeader(key=rabbitmq.routing.key, value=TOPIC, schema=Schema{STRING})])}. (org.apache.kafka.connect.runtime.errors.LogReporter)
org.apache.kafka.connect.errors.DataException: Invalid schema type for ByteArrayConverter: STRUCT
  at org.apache.kafka.connect.converters.ByteArrayConverter.fromConnectData(ByteArrayConverter.java:55)
  at org.apache.kafka.connect.runtime.WorkerSourceTask.lambda$convertTransformedRecord$2(WorkerSourceTask.java:269)
  at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)
  at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)
  at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)
  at org.apache.kafka.connect.runtime.WorkerSourceTask.convertTransformedRecord(WorkerSourceTask.java:269)
  at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:293)
  at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:228)
  at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
  at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
[2019-04-25 08:04:25,128] INFO WorkerSourceTask{id=RabbitMQConsentConsumer-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)
[2019-04-25 08:04:25,128] INFO WorkerSourceTask{id=RabbitMQConsentConsumer-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)
[2019-04-25 08:04:25,128] ERROR WorkerSourceTask{id=RabbitMQConsentConsumer-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler
  at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)
  at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)
  at org.apache.kafka.connect.runtime.WorkerSourceTask.convertTransformedRecord(WorkerSourceTask.java:269)
  at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:293)
  at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:228)
  at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
  at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.connect.errors.DataException: Invalid schema type for ByteArrayConverter: STRUCT
  at org.apache.kafka.connect.converters.ByteArrayConverter.fromConnectData(ByteArrayConverter.java:55)
  at org.apache.kafka.connect.runtime.WorkerSourceTask.lambda$convertTransformedRecord$2(WorkerSourceTask.java:269)
  at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)
  at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)
  ... 11 more
[2019-04-25 08:04:25,129] ERROR WorkerSourceTask{id=RabbitMQConsentConsumer-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)

---

据我所知，似乎我的

org.apache.kafka.connect.transforms.highfield$Value

转换失败，因为我的数据是字节数组，而不是结构/映射。考虑到这一点，我可以将我的消息从字节数组“升级”到json吗？

连接器的数据管道是，首先从源检索数据，对其应用转换，并在转换后的数据顶部应用提供的转换器，然后将其推送到提供的主题中

现在在您的例子中，数据可能是通过对该数据应用转换后进入bytearray的，它正在转换为bytearray格式不支持的结构类型。所以，你得到了这个例外

HostField用于在为源数据提供架构时将数据包装为结构格式。

---

正如@Robin Moffatt所说，请尝试使用JSON转换器，然后尝试，它应该可以工作。

为什么不直接使用

org.apache.kafka.connect.JSON.JsonConverter

？请确认，如果在连接器配置中不使用transform，此连接器是否符合预期？@RobinMoffatt似乎RabbitMQ连接器值模式将值强制为字节数组（请参见

com.github.jcustenborder.kafka.connect.RabbitMQ.MessageConverter.schema_value

）：据我所知，行

字段（FIELD_MESSAGE_BODY，SchemaBuilder.bytes（）.doc（“值体（不透明的，特定于客户端的字节数组）”））.build（）

似乎强制消息值为字节数组，据我所知）似乎无法使用Kafka connect进行处理transformers@GokulPotluri是的，如果我按原样使用第一个属性块中的配置，消息会正确发送到Kafka broker。我不熟悉代码。但是，您是否尝试过

org.apache.Kafka.connect.json.JsonConverter

，但出现了错误？