Apache kafka 如何从卡夫卡主题流到elasticsearch和confluent?
我从机器中读取数据,并将其作为JSON流式传输到卡夫卡主题。我想阅读本主题并使用confluent将streamdata存储到elasticsearch中 我的步骤: 1.创建要从JSON转换为AVRO的KSQL流 json流:Apache kafka 如何从卡夫卡主题流到elasticsearch和confluent?,apache-kafka,avro,apache-kafka-connect,confluent-platform,ksqldb,Apache Kafka,Avro,Apache Kafka Connect,Confluent Platform,Ksqldb,我从机器中读取数据,并将其作为JSON流式传输到卡夫卡主题。我想阅读本主题并使用confluent将streamdata存储到elasticsearch中 我的步骤: 1.创建要从JSON转换为AVRO的KSQL流 json流: CREATE STREAM source_json_pressure ( timestamp BIGINT, opcuaObject VARCHAR, value DOUBLE ) WITH (KAFKA_TOPIC='7d12h100mb
CREATE STREAM source_json_pressure
(
timestamp BIGINT,
opcuaObject VARCHAR,
value DOUBLE
)
WITH (KAFKA_TOPIC='7d12h100mbpressure',
VALUE_FORMAT='JSON');
avro流:
CREATE STREAM target_avro_pressure
WITH (
KAFKA_TOPIC='7d12h100mbpressure_avro',
VALUE_FORMAT='AVRO'
) AS
SELECT * FROM source_json_pressure;
ksql> print "7d12h100mbpressure_avro";
Format:AVRO
23.04.19 19:29:58 MESZ, jK?C, {"TIMESTAMP": 1556040449728, "OPCUAOBJECT": "DatLuDrUeb.EinDru", "VALUE": 7.42}
在此之后,我得到这个avro流:
CREATE STREAM target_avro_pressure
WITH (
KAFKA_TOPIC='7d12h100mbpressure_avro',
VALUE_FORMAT='AVRO'
) AS
SELECT * FROM source_json_pressure;
ksql> print "7d12h100mbpressure_avro";
Format:AVRO
23.04.19 19:29:58 MESZ, jK?C, {"TIMESTAMP": 1556040449728, "OPCUAOBJECT": "DatLuDrUeb.EinDru", "VALUE": 7.42}
My elasticsearch.properties:
15 name=elasticsearch-sink
16 connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
17 tasks.max=1
18 topics=7d12h100mbpressure_avro
19 key.ignore=true
20 connection.url=http://localhost:9200
21 type.name=kafka-connect
# Bootstrap Kafka servers. If multiple servers are specified, they should be comma-separated.
bootstrap.servers=localhost:9092
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://localhost:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://localhost:8081
config.storage.topic=connect-configs
offset.storage.topic=connect-offsets
status.storage.topic=connect-statuses
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
在此之后,我希望流是ES,但我得到的索引没有流数据
我哪里出错了
合流日志连接中的错误:
[2019-04-24 11:01:29,316] INFO [Consumer clientId=consumer-4, groupId=connect-elasticsearch-sink] Setting newly assigned partitions: 7d12h100mbpressure_avro-3, 7d12h100mbpressure_avro-2, 7d12h100mbpressure_avro-1, 7d12h100mbpressure_avro-0 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:290)
[2019-04-24 11:01:29,327] INFO [Consumer clientId=consumer-4, groupId=connect-elasticsearch-sink] Resetting offset for partition 7d12h100mbpressure_avro-3 to offset 0. (org.apache.kafka.clients.consumer.internals.Fetcher:584)
[2019-04-24 11:01:29,327] INFO [Consumer clientId=consumer-4, groupId=connect-elasticsearch-sink] Resetting offset for partition 7d12h100mbpressure_avro-2 to offset 0. (org.apache.kafka.clients.consumer.internals.Fetcher:584)
[2019-04-24 11:01:29,327] INFO [Consumer clientId=consumer-4, groupId=connect-elasticsearch-sink] Resetting offset for partition 7d12h100mbpressure_avro-1 to offset 0. (org.apache.kafka.clients.consumer.internals.Fetcher:584)
[2019-04-24 11:01:29,328] INFO [Consumer clientId=consumer-4, groupId=connect-elasticsearch-sink] Resetting offset for partition 7d12h100mbpressure_avro-0 to offset 0. (org.apache.kafka.clients.consumer.internals.Fetcher:584)
[2019-04-24 11:01:29,667] ERROR WorkerSinkTask{id=elasticsearch-sink-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:177)
org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)
at org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:484)
at org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:464)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:320)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.connect.errors.DataException: Failed to deserialize data for topic 7d12h100mbpressure_avro to Avro:
at io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:107)
at org.apache.kafka.connect.runtime.WorkerSinkTask.lambda$convertAndTransformRecord$0(WorkerSinkTask.java:484)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)
... 13 more
Caused by: org.apache.kafka.common.errors.SerializationException: Error retrieving Avro schema for id 92747
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema not found; error code: 40403
at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:226)
at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:252)
at io.confluent.kafka.schemaregistry.client.rest.RestService.getId(RestService.java:482)
at io.confluent.kafka.schemaregistry.client.rest.RestService.getId(RestService.java:475)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getSchemaByIdFromRegistry(CachedSchemaRegistryClient.java:151)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getBySubjectAndId(CachedSchemaRegistryClient.java:230)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getById(CachedSchemaRegistryClient.java:209)
at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserialize(AbstractKafkaAvroDeserializer.java:116)
at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserializeWithSchemaAndVersion(AbstractKafkaAvroDeserializer.java:215)
at io.confluent.connect.avro.AvroConverter$Deserializer.deserialize(AvroConverter.java:145)
at io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:90)
at org.apache.kafka.connect.runtime.WorkerSinkTask.lambda$convertAndTransformRecord$0(WorkerSinkTask.java:484)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)
at org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:484)
at org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:464)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:320)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-04-24 11:01:29,668] ERROR WorkerSinkTask{id=elasticsearch-sink-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:178)
我的connect-avro-distributed.properties:
15 name=elasticsearch-sink
16 connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
17 tasks.max=1
18 topics=7d12h100mbpressure_avro
19 key.ignore=true
20 connection.url=http://localhost:9200
21 type.name=kafka-connect
# Bootstrap Kafka servers. If multiple servers are specified, they should be comma-separated.
bootstrap.servers=localhost:9092
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://localhost:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://localhost:8081
config.storage.topic=connect-configs
offset.storage.topic=connect-offsets
status.storage.topic=connect-statuses
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
您可以从Elasticsink设置
key.ignore=true
,但是,这不会阻止Connect尝试反序列化记录
当您只需执行汇合启动
时,它将始终对键和值转换器使用AvroConverter
值得一提的是,KSQL中的VALUE\u FORMAT='AVRO'
只将值作为AVRO,我相信,不是键
其中一个原因可以解释为什么你会看到
- 找不到主题
- 找不到架构
- 检索id的Avro架构时出错
elasticsearch.properties
中,您可以覆盖key.converter
,使其成为类似于org.apache.kafka.connect.storage.StringConverter
此外,我建议使用
kafka avro console consumer
并包括--property print.key=true
选项,而不是使用Connect+KSQL进行调试,以查看是否出现类似错误 不清楚您运行了什么命令来启动Kafka connect如果执行融合日志连接,您是否看到任何错误?@ofitz能否共享您的Kafka connect worker配置?很可能您的转换器设置不正确。最好是你可以编辑你的问题,以包括卡夫卡连接工作日志中的任何错误(confluent log Connect
),如@cricket_007所说。这是一个重要的例外,但我如何解决它:原因:org.apache.kafka.connect.errors.DataException:未能将主题7D12H100MB压力的数据反序列化为avro:
检索id 92747的avro架构时出错
<代码>RestClientException:未找到架构。。。我认为您需要删除作为AvroConverter的key.converter
,因为KSQL不使用/kafka Avro控制台使用者创建Avro密钥——主题7d12h100mbpressure_avro1——引导服务器localhost:9092
是我的输出:{“TIMESTAMP”:{“long”:1556171473227},“opcuaoobject”:{“string”:“DatLuDrUeb.EinDru”},“VALUE”:{“double”:8.06}
如果我设置--property print.key=true
我会得到与启动confluent时哪个配置文件使用confluent相同的错误/confluent start
?我在confluent-5.2.1/etc/schema registry/connect avro distributed.properties
和confluent-5.2.1/etc/kafka/connect distributed.properties
中有配置,它使用connect avro distributed.properties
。但就像我说的,你可以覆盖Elasticsearch水槽属性中的转换器非常感谢你。在我覆盖键后,转换器就可以工作了!我怎样才能用elastic编写多个主题?我可以为多个主题配置elasticsearch.properties
,如下所示:topics=topic1,topic2
FYI:KSQL即将支持Avro格式的密钥(通常是“结构化密钥”)。