Java 春云溪卡夫卡>;使用来自合流REST代理的Avro消息
我有以下情况:Java 春云溪卡夫卡>;使用来自合流REST代理的Avro消息,java,spring,apache-kafka,spring-cloud-stream,confluent-platform,Java,Spring,Apache Kafka,Spring Cloud Stream,Confluent Platform,我有以下情况: Producer通过Confluent的REST代理(在Confluent的模式注册表上注册模式)向Kafka主题发送Avro编码的消息,如中所述 启用Spring Cloud Stream的消息侦听主题以获取新消息 我的应用程序如下所示: @SpringBootApplication @EnableBinding(Sink.class) public class MyApplication { private static Logger log = LoggerFact
- Producer通过Confluent的REST代理(在Confluent的模式注册表上注册模式)向Kafka主题发送Avro编码的消息,如中所述
- 启用Spring Cloud Stream的消息侦听主题以获取新消息
@SpringBootApplication
@EnableBinding(Sink.class)
public class MyApplication {
private static Logger log = LoggerFactory.getLogger(MyApplication.class);
public static void main(String[] args) {
SpringApplication.run(MyApplication.class, args);
}
@StreamListener(Sink.INPUT)
public void myMessageSink(MyMessage message) {
log.info("Received new message: {}", message);
}
}
spring.cloud.stream.bindings.input.destination=myTopic
spring.cloud.stream.bindings.input.group=${spring.application.name}
spring.cloud.stream.bindings.input.contentType=application/*+avro
而MyMessage是由Avro根据Avro模式创建的类
我的application.properties如下所示:
@SpringBootApplication
@EnableBinding(Sink.class)
public class MyApplication {
private static Logger log = LoggerFactory.getLogger(MyApplication.class);
public static void main(String[] args) {
SpringApplication.run(MyApplication.class, args);
}
@StreamListener(Sink.INPUT)
public void myMessageSink(MyMessage message) {
log.info("Received new message: {}", message);
}
}
spring.cloud.stream.bindings.input.destination=myTopic
spring.cloud.stream.bindings.input.group=${spring.application.name}
spring.cloud.stream.bindings.input.contentType=application/*+avro
现在我的问题是,每次收到新消息时,都会引发以下异常:
org.springframework.messaging.MessagingException: Exception thrown while invoking MyApplication#myMessageSink[1 args]; nested exception is org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -27
at org.springframework.cloud.stream.binding.StreamListenerAnnotationBeanPostProcessor$StreamListenerMessageHandler.handleRequestMessage(StreamListenerAnnotationBeanPostProcessor.java:316) ~[spring-cloud-stream-1.1.0.RELEASE.jar:1.1.0.RELEASE]
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109) ~[spring-integration-core-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127) ~[spring-integration-core-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116) ~[spring-integration-core-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148) ~[spring-integration-core-4.3.2.RELEASE.jar:4.3.2.RELEASE]
...
Caused by: org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -27
at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:336) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:263) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:430) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:422) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.readMapKey(GenericDatumReader.java:335) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:321) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:177) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:116) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:230) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:174) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152) ~[avro-1.8.1.jar:1.8.1]
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144) ~[avro-1.8.1.jar:1.8.1]
at org.springframework.cloud.stream.schema.avro.AbstractAvroMessageConverter.convertFromInternal(AbstractAvroMessageConverter.java:91) ~[spring-cloud-stream-schema-1.1.0.RELEASE.jar:1.1.0.RELEASE]
at org.springframework.messaging.converter.AbstractMessageConverter.fromMessage(AbstractMessageConverter.java:175) ~[spring-messaging-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.messaging.converter.CompositeMessageConverter.fromMessage(CompositeMessageConverter.java:67) ~[spring-messaging-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.messaging.handler.annotation.support.PayloadArgumentResolver.resolveArgument(PayloadArgumentResolver.java:117) ~[spring-messaging-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:112) ~[spring-messaging-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:138) ~[spring-messaging-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:107) ~[spring-messaging-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.cloud.stream.binding.StreamListenerAnnotationBeanPostProcessor$StreamListenerMessageHandler.handleRequestMessage(StreamListenerAnnotationBeanPostProcessor.java:307) ~[spring-cloud-stream-1.1.0.RELEASE.jar:1.1.0.RELEASE]
... 35 common frames omitted
据我所知,问题在于合流堆栈将消息架构的ID作为消息有效负载的一部分包含在内,客户机应该在架构ID之后开始读取实际的Avro消息。
似乎我需要配置Kafka绑定以使用Confluent的KafkaAvroDeserializer,但我不知道如何实现这一点
(我可以使用Confluent的avro控制台消费者很好地检索消息,因此avro编码似乎没有问题)
我还尝试使用@EnableSchemaRegistry注释并配置ConfluentSchemaRegistryClient bean,但在我看来,这只控制模式的存储/检索位置,而不是实际的反序列化
这是否应该在某种程度上起作用?当设置每个绑定的
属性spring.cloud.stream.kafka.bindings.input.consumer.configuration.value.deserializer
以拥有Confluent的Kafkaavroderializer类名时,它是否起作用?回答了我自己的问题。我现在所做的是实现一个MessageConverter,它只删除任何消息的前4个字节,然后再将它们传递给Avro解码器。代码主要取自spring cloud stream的AbstractAvroMessageConverter:
public class ConfluentAvroSchemaMessageConverter extends AvroSchemaMessageConverter {
public ConfluentAvroSchemaMessageConverter() {
super(new MimeType("application", "avro+confluent"));
}
@Override
protected Object convertFromInternal(Message<?> message, Class<?> targetClass, Object conversionHint) {
Object result = null;
try {
byte[] payload = (byte[]) message.getPayload();
// byte array to contain the message without the confluent header (first 4 bytes)
byte[] payloadWithoutConfluentHeader = new byte[payload.length - 4];
ByteBuffer buf = ByteBuffer.wrap(payload);
MimeType mimeType = getContentTypeResolver().resolve(message.getHeaders());
if (mimeType == null) {
if (conversionHint instanceof MimeType) {
mimeType = (MimeType) conversionHint;
}
else {
return null;
}
}
// read first 4 bytes and copy the rest to the new byte array
// see https://groups.google.com/forum/#!topic/confluent-platform/rI1WNPp8DJU
buf.getInt();
buf.get(payloadWithoutConfluentHeader);
Schema writerSchema = resolveWriterSchemaForDeserialization(mimeType);
Schema readerSchema = resolveReaderSchemaForDeserialization(targetClass);
DatumReader<Object> reader = getDatumReader((Class<Object>) targetClass, readerSchema, writerSchema);
Decoder decoder = DecoderFactory.get().binaryDecoder(payloadWithoutConfluentHeader, null);
result = reader.read(null, decoder);
}
catch (IOException e) {
throw new MessageConversionException(message, "Failed to read payload", e);
}
return result;
}
公共类Confluent AvroschemMessageConverter扩展了AvroschemMessageConverter{
公共汇流AVROSCHEMA消息转换器(){
super(新的MimeType(“应用程序”、“avro+合流”);
}
@凌驾
受保护的对象convertFromInternal(消息消息、类targetClass、对象转换提示){
对象结果=空;
试一试{
字节[]有效载荷=(字节[])消息。getPayload();
//字节数组,用于包含不包含合流标头的消息(前4个字节)
byte[]payloadWithoutConfluentHeader=新字节[payload.length-4];
ByteBuffer buf=ByteBuffer.wrap(有效载荷);
MimeType-MimeType=getContentTypeResolver().resolve(message.getHeaders());
if(mimeType==null){
if(MimeType的转换提示实例){
mimeType=(mimeType)转换提示;
}
否则{
返回null;
}
}
//读取前4个字节,并将其余字节复制到新的字节数组中
//看https://groups.google.com/forum/#!主题/汇合平台/rI1WNPp8DJU
buf.getInt();
buf.get(无汇合头的有效载荷);
Schema writerSchema=resolverewriterschemaforderserialization(mimeType);
Schema readerSchema=ResolveReaderSchemaForderSerialization(targetClass);
DatumReader=getDatumReader((类)targetClass,readerSchema,writerSchema);
Decoder Decoder=DecoderFactory.get().binaryDecoder(payloadWithoutConfluentHeader,null);
结果=reader.read(空,解码器);
}
捕获(IOE异常){
抛出新的MessageConversionException(消息“读取有效负载失败”,e);
}
返回结果;
}
然后,我通过application.properties将传入卡夫卡主题的内容类型设置为application/avro+confluent
这至少可以让我检索使用合流堆栈编码的消息,但它当然不会以任何方式与模式注册表交互。看起来KafkaMessageChannelBinder
的使用者端点总是使用ByteArraydSerializer
作为键/值序列化程序。这可能是KafkaMessageCh的结果默认情况下,annelBinder
的生产者使用的是ByteArraySerializer
。对于非基于Spring云流的生产者,消费者应该能够使用上述属性覆盖所需的反序列化器。不幸的是,我没有这样做,我已经尝试过了。从我在中看到的情况来看,反序列化器被硬编码为默认的ByteArrayDeserializer.yeah,请在此处创建问题:。我们可以从那里跟踪它。谢谢!已创建问题: