Apache kafka 无法使用kafka avro控制台使用者读取avro消息。SerializationException:未知的魔法字节
我正在编写一个REST代理,就像合流REST代理一样。它接受JSON负载、模式主题和id,然后将JSON负载作为Avro对象写入流中。当我使用kafka avro console consumer阅读消息时,我收到了“未知魔法字节”错误 这是我的卡夫卡制作人配置:Apache kafka 无法使用kafka avro控制台使用者读取avro消息。SerializationException:未知的魔法字节,apache-kafka,kafka-consumer-api,avro,kafka-producer-api,Apache Kafka,Kafka Consumer Api,Avro,Kafka Producer Api,我正在编写一个REST代理,就像合流REST代理一样。它接受JSON负载、模式主题和id,然后将JSON负载作为Avro对象写入流中。当我使用kafka avro console consumer阅读消息时,我收到了“未知魔法字节”错误 这是我的卡夫卡制作人配置: properties.put("client.id", LocalHostUtils.getLocalHostName(null)); properties.put(ProducerConfig.KE
properties.put("client.id", LocalHostUtils.getLocalHostName(null));
properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
properties.put(AbstractKafkaAvroSerDeConfig.AUTO_REGISTER_SCHEMAS, false);
properties.put(KafkaAvroSerializerConfig.VALUE_SUBJECT_NAME_STRATEGY, RecordNameStrategy.class);
properties.put("schema.registry.url", configValuesManager.getString("dsp_kafka.schema_registry"));
if (KafkaUtils.isKafkaEnabled()) {
this.kafkaProducer = new KafkaProducer<String, Object>(properties);
}
这是toAvro
方法的实现:
Object toAvro(Schema schema, String jsonBody) throws IOException
{
DatumReader<Object> reader = new GenericDatumReader<Object>(schema);
Object object = reader.read(
null, decoderFactory.jsonDecoder(schema, jsonBody));
return object;
}
这是kafkaSchemaValidatingProducer上的publish
方法
public void publish(String topic, Integer partition, String key, Object data)
{
log.debug("publish topic={} key={} value={}", topic, key, data);
if (!KafkaUtils.isKafkaEnabled()) {
log.warn("Kafka is not enabled....");
return;
}
ProducerRecord<String, Object> record = new ProducerRecord<String, Object>(topic, key, data);
Future<RecordMetadata> metadataFuture = kafkaProducer.send(record, new Callback()
{
@Override
public void onCompletion(RecordMetadata metadata, Exception exception)
{
if (exception == null) {
log.info(metadata.toString());
return;
}
log.error("exception", exception);
}
});
kafkaProducer.flush();
}
这导致
[2019-08-26 16:30:36,351] ERROR Error processing message, skipping this message: (kafka.tools.ConsoleConsumer$:76)
org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
知道我为什么会出现“坏幻数错误”吗 我解决了这个问题。这是因为我没有在命令中指定密钥反序列化器
这是有效的命令
./bin/kafka-avro-console-consumer \
--bootstrap-server <bootstrap-server> \
--consumer.config client-ssl.properties \
--property schema.registry.url=<schema-registry-url> \
--topic <name-of-topic> \
--property print.key=true \
--property print.value=true \
--value-deserializer io.confluent.kafka.serializers.KafkaAvroDeserializer \
--key-deserializer org.apache.kafka.common.serialization.StringDeserializer
/bin/kafka avro控制台消费者\
--引导服务器\
--consumer.config client-ssl.properties\
--属性架构。注册表。url=\
--话题\
--属性print.key=true\
--属性print.value=true\
--值反序列化器io.confluent.kafka.serializers.KafkaAvroDeserializer\
--密钥反序列化程序org.apache.kafka.common.serialization.StringDeserializer
我解决了这个问题。这是因为我没有在命令中指定密钥反序列化器
这是有效的命令
./bin/kafka-avro-console-consumer \
--bootstrap-server <bootstrap-server> \
--consumer.config client-ssl.properties \
--property schema.registry.url=<schema-registry-url> \
--topic <name-of-topic> \
--property print.key=true \
--property print.value=true \
--value-deserializer io.confluent.kafka.serializers.KafkaAvroDeserializer \
--key-deserializer org.apache.kafka.common.serialization.StringDeserializer
/bin/kafka avro控制台消费者\
--引导服务器\
--consumer.config client-ssl.properties\
--属性架构。注册表。url=\
--话题\
--属性print.key=true\
--属性print.value=true\
--值反序列化器io.confluent.kafka.serializers.KafkaAvroDeserializer\
--密钥反序列化程序org.apache.kafka.common.serialization.StringDeserializer
[2019-08-26 16:30:36,351] ERROR Error processing message, skipping this message: (kafka.tools.ConsoleConsumer$:76)
org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
./bin/kafka-avro-console-consumer \
--bootstrap-server <bootstrap-server> \
--consumer.config client-ssl.properties \
--property schema.registry.url=<schema-registry-url> \
--topic <name-of-topic> \
--property print.key=true \
--property print.value=true \
--value-deserializer io.confluent.kafka.serializers.KafkaAvroDeserializer \
--key-deserializer org.apache.kafka.common.serialization.StringDeserializer