Apache kafka 合流BigQuerySinkConnector架构注册表错误_Apache Kafka_Apache Kafka Connect_Confluent Platform

Apache kafka 合流BigQuerySinkConnector架构注册表错误

apache-kafka

Apache kafka 合流BigQuerySinkConnector架构注册表错误,apache-kafka,apache-kafka-connect,confluent-platform,Apache Kafka,Apache Kafka Connect,Confluent Platform,我尝试使用合流BigQuerySinkConnector创建Kafka到BigQuery的数据管道。测试环境是一个docker容器。我在上面添加了（它不作为默认值存在）。我在Google BigQuery端做的所有定义（我希望…）。但它只是给出了模式注册表错误，我无法理解它为什么会发生。我在BigQuery数据集上创建了一个名为rest_avro的表。主题架构： { "fields": [ { "name": "name&

我尝试使用合流BigQuerySinkConnector创建Kafka到BigQuery的数据管道。测试环境是一个docker容器。我在上面添加了（它不作为默认值存在）。我在Google BigQuery端做的所有定义（我希望…）。但它只是给出了模式注册表错误，我无法理解它为什么会发生。我在BigQuery数据集上创建了一个名为rest_avro的表。主题架构：

{
  "fields": [
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "age",
      "type": [
        "null",
        "int"
      ]
    }
  ],
  "name": "User",
  "type": "record"
}

我在BigQuery表上手动定义了这个模式

连接器运行时没有错误

我的连接器配置已成功加载

[2020-08-19 13:21:46,803] INFO SinkConnectorConfig values:
    config.action.reload = restart
    connector.class = com.wepay.kafka.connect.bigquery.BigQuerySinkConnector
    errors.deadletterqueue.context.headers.enable = false
    errors.deadletterqueue.topic.name =
    errors.deadletterqueue.topic.replication.factor = 3
    errors.log.enable = false
    errors.log.include.messages = false
    errors.retry.delay.max.ms = 60000
    errors.retry.timeout = 0
    errors.tolerance = none
    header.converter = null
    key.converter = null
    name = kcbq-connect1
    tasks.max = 1
    topics = [rest-avro]
    topics.regex =
    transforms = []
    value.converter = null
 (org.apache.kafka.connect.runtime.SinkConnectorConfig)
[2020-08-19 13:21:46,803] INFO EnrichedConnectorConfig values:
    config.action.reload = restart
    connector.class = com.wepay.kafka.connect.bigquery.BigQuerySinkConnector
    errors.deadletterqueue.context.headers.enable = false
    errors.deadletterqueue.topic.name =
    errors.deadletterqueue.topic.replication.factor = 3
    errors.log.enable = false
    errors.log.include.messages = false
    errors.retry.delay.max.ms = 60000
    errors.retry.timeout = 0
    errors.tolerance = none
    header.converter = null
    key.converter = null
    name = kcbq-connect1
    tasks.max = 1
    topics = [rest-avro]
    topics.regex =
    transforms = []
    value.converter = null
 (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig)
[2020-08-19 13:21:46,804] INFO [Worker clientId=connect-1, groupId=compose-connect-group] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[2020-08-19 13:21:55,940] INFO [Consumer clientId=connector-consumer-kcbq-connect1-0, groupId=connect-kcbq-connect1] Seeking to offset 8 for partition rest-avro-0 (org.apache.kafka.clients.consumer.KafkaConsumer)

但控制台会出现如下错误。你知道吗

[2020-08-19 13:09:45,663] ERROR Task failed with org.apache.kafka.connect.errors.ConnectException error: Exception encountered while trying to fetch latest schema metadata from Schema Registry (com.wepay.kafka.connect.bigquery.write.batch.KCBQThreadPoolExecutor)
Exception in thread "pool-3-thread-79" org.apache.kafka.connect.errors.ConnectException: Exception encountered while trying to fetch latest schema metadata from Schema Registry
    at com.wepay.kafka.connect.bigquery.schemaregistry.schemaretriever.SchemaRegistrySchemaRetriever.retrieveSchema(SchemaRegistrySchemaRetriever.java:67)
    at com.wepay.kafka.connect.bigquery.SchemaManager.updateSchema(SchemaManager.java:58)
    at com.wepay.kafka.connect.bigquery.write.row.AdaptiveBigQueryWriter.attemptSchemaUpdate(AdaptiveBigQueryWriter.java:129)
    at com.wepay.kafka.connect.bigquery.write.row.AdaptiveBigQueryWriter.performWriteRequest(AdaptiveBigQueryWriter.java:96)
    at com.wepay.kafka.connect.bigquery.write.row.BigQueryWriter.writeRows(BigQueryWriter.java:117)
    at com.wepay.kafka.connect.bigquery.write.batch.TableWriter.run(TableWriter.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
    at sun.net.www.http.HttpClient.New(HttpClient.java:339)
    at sun.net.www.http.HttpClient.New(HttpClient.java:357)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:153)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:188)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.getLatestVersion(RestService.java:359)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.getLatestVersion(RestService.java:351)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getLatestSchemaMetadata(CachedSchemaRegistryClient.java:136)
    at com.wepay.kafka.connect.bigquery.schemaregistry.schemaretriever.SchemaRegistrySchemaRetriever.retrieveSchema(SchemaRegistrySchemaRetriever.java:63)
    ... 8 more

[2020-08-19 13:09:45663]错误任务失败，org.apache.kafka.connect.errors.ConnectException错误：尝试从架构注册表（com.wepay.kafka.connect.bigquery.write.batch.KCBQThreadPoolExecutor）获取最新架构元数据时遇到异常
线程“pool-3-thread-79”org.apache.kafka.connect.errors.ConnectException中出现异常：尝试从架构注册表获取最新架构元数据时遇到异常
在com.wepay.kafka.connect.bigquery.schemaregistry.schemaretriever.SchemaRegistrySchemaRetriever.retrieveSchema（SchemaRegistrySchemaRetriever.java:67）
在com.wepay.kafka.connect.bigquery.SchemaManager.updateSchema（SchemaManager.java:58）
位于com.wepay.kafka.connect.bigquery.write.row.AdaptiveBigQueryWriter.attemptSchemaUpdate（AdaptiveBigQueryWriter.java:129）
位于com.wepay.kafka.connect.bigquery.write.row.AdaptiveBigQueryWriter.performWriteRequest（AdaptiveBigQueryWriter.java:96）
在com.wepay.kafka.connect.bigquery.write.row.BigQueryWriter.writeRows（BigQueryWriter.java:117）上
在com.wepay.kafka.connect.bigquery.write.batch.TableWriter.run（TableWriter.java:77）上
位于java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1149）
位于java.util.concurrent.ThreadPoolExecutor$Worker.run（ThreadPoolExecutor.java:624）
运行（Thread.java:748）
原因：java.net.ConnectException:连接被拒绝（连接被拒绝）
位于java.net.PlainSocketImpl.socketConnect（本机方法）
位于java.net.AbstractPlainSocketImpl.doConnect（AbstractPlainSocketImpl.java:350）
位于java.net.AbstractPlainSocketImpl.connectToAddress（AbstractPlainSocketImpl.java:206）
位于java.net.AbstractPlainSocketImpl.connect（AbstractPlainSocketImpl.java:188）
位于java.net.socksocketimpl.connect（socksocketimpl.java:392）
位于java.net.Socket.connect（Socket.java:589）
位于java.net.Socket.connect（Socket.java:538）
位于sun.net.NetworkClient.doConnect（NetworkClient.java:180）
位于sun.net.www.http.HttpClient.openServer（HttpClient.java:463）
位于sun.net.www.http.HttpClient.openServer（HttpClient.java:558）
http.HttpClient.（HttpClient.java:242）
http.HttpClient.New（HttpClient.java:339）
http.HttpClient.New（HttpClient.java:357）
位于sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient（HttpURLConnection.java:1220）
位于sun.net.www.protocol.http.HttpURLConnection.plainConnect0（HttpURLConnection.java:1156）
位于sun.net.www.protocol.http.HttpURLConnection.plainConnect（HttpURLConnection.java:1050）
位于sun.net.www.protocol.http.HttpURLConnection.connect（HttpURLConnection.java:984）
位于sun.net.www.protocol.http.HttpURLConnection.getInputStream0（HttpURLConnection.java:1564）
位于sun.net.www.protocol.http.HttpURLConnection.getInputStream（HttpURLConnection.java:1492）
位于java.net.HttpURLConnection.getResponseCode（HttpURLConnection.java:480）
位于io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest（RestService.java:153）
位于io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest（RestService.java:188）
位于io.confluent.kafka.schemaregistry.client.rest.RestService.getLatestVersion（RestService.java:359）
位于io.confluent.kafka.schemaregistry.client.rest.RestService.getLatestVersion（RestService.java:351）
在io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getLatestSchemaMetadata（CachedSchemaRegistryClient.java:136）
在com.wepay.kafka.connect.bigquery.schemaregistry.schemaretriever.SchemaRegistrySchemaRetriever.retrieveSchema（SchemaRegistrySchemaRetriever.java:63）
... 8个以上

问题表明BigQuery接收器连接器无法从架构注册表容器检索当前架构

似乎在BigQuery接收器连接器属性上错误配置了SchemaRegistrLocation字段。很可能它已经被设定为http://localhost:8081.

正如随附的docker-compose.yml所说，该端点需要定义为http://schema-registry:8081 在停靠的环境中

通过以下属性，我成功地创建了BigQuery接收器连接器

{
  "name": "customer-connect1",
  "config": {
    "connector.class": "com.wepay.kafka.connect.bigquery.BigQuerySinkConnector",
    "tasks.max": "1",
    "topics": "dbserver1.venue_organization.customers",
    "topicsToTables": "dbserver1.venue_organization.customers=customers",
    "sanitizeTopics": "true",
    "autoCreateTables": "true",
    "autoUpdateSchemas": "true",
    "schemaRetriever": "com.wepay.kafka.connect.bigquery.schemaregistry.schemaretriever.SchemaRegistrySchemaRetriever",
    "schemaRegistryLocation": "http://schema-registry:8081",
    "bufferSize": "100000",
    "maxWriteSize": "10000",
    "tableWriteWait": "1000",
    "project": "venue-organization",
    "datasets": ".*=venue_organization",
    "keyfile": " /data/venue-organization-service-account.json"
  }
}

这里有更多信息：