如何使用topic regex选项创建具有多个主题的JDBC接收器连接器

创建了一个JDBC源连接器

catalog.pattern = test_01

源连接器配置

    {
  "name": "jdbcsource",
  "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
  "connection.url": "jdbc:mysql://192.168.1.8/test_01?nullCatalogMeansCurrent=true",
  "connection.user": "root",
  "connection.password": "********",
  "catalog.pattern": "test_01",
  "mode": "timestamp",
  "timestamp.column.name": "UpdateDate",
  "topic.prefix": "jdbcsource-"
}

{
  "name": "jdbcsink",
  "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
  "topics": [
    "jdbcsource-example01",
    "jdbcsource-examplerole"
  ],
  "connection.url": "jdbc:mysql://192.168.1.8/test_01?nullCatalogMeansCurrent=true",
  "connection.user": "root",
  "connection.password": "********",
  "auto.create": "true",
  "auto.evolve": "true"
}

在test_01下，数据库example01（父表）examplerole（子表）以外部引用作为用户id（参考下面的模式）

使用表中的topics.regex选项和键引用创建接收器连接器时

curl -XPOST -H 'Accept: application/json' -H "Content-type: application/json" -d '{
    "name": "MySQL-JDBC-sink-connector",
    "config": {
        "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
        "tasks.max": "1",
    "topics.regex": "jdbcsource*",
    "connection.url": "jdbc:mysql://192.168.1.8/test_01?nullCatalogMeansCurrent=true",
     "connection.user": "root",
    "connection.password": "********",
    "insert.mode": "insert",
    "pk.mode": "record_value",
    "pk.fields": "user_id",
    "auto.create": "true",
     "auto.evolve": "true"

}
}' 'localhost:8083/connectors'
echo "\n"

上述接收器配置出错，但配置中提供了topics regex选项

[2019-06-10 04:54:56,678] ERROR Uncaught exception in REST call to /connector-plugins/io.confluent.connect.jdbc.JdbcSinkConnector/config/validate (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper)
org.apache.kafka.common.config.ConfigException: Must configure one of topics or topics.regex
    at org.apache.kafka.connect.runtime.SinkConnectorConfig.validate(SinkConnectorConfig.java:96)
    at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:269)
    at org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource.validateConfigs(ConnectorPluginsResource.java:81)
    at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:243)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
    at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:502)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.

尝试使用选项创建多个jdbc接收器连接器

主题：

请参阅下面的配置

    {
  "name": "jdbcsource",
  "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
  "connection.url": "jdbc:mysql://192.168.1.8/test_01?nullCatalogMeansCurrent=true",
  "connection.user": "root",
  "connection.password": "********",
  "catalog.pattern": "test_01",
  "mode": "timestamp",
  "timestamp.column.name": "UpdateDate",
  "topic.prefix": "jdbcsource-"
}

{
  "name": "jdbcsink",
  "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
  "topics": [
    "jdbcsource-example01",
    "jdbcsource-examplerole"
  ],
  "connection.url": "jdbc:mysql://192.168.1.8/test_01?nullCatalogMeansCurrent=true",
  "connection.user": "root",
  "connection.password": "********",
  "auto.create": "true",
  "auto.evolve": "true"
}

低于误差

[2019-06-10 04:53:19,738] WARN [Consumer clientId=consumer-6, groupId=connect-jdbcsink] Error while fetching metadata with correlation id 2 : { jdbcsource-examplerole=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient)
[2019-06-10 04:53:19,739] ERROR WorkerSinkTask{id=jdbcsink-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
org.apache.kafka.common.errors.InvalidTopicException: Invalid topics: [ jdbcsource-examplerole]
[2019-06-10 04:53:19,739] ERROR WorkerSinkTask{id=jdbcsink-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
[2019-06-10 04:53:19,740] INFO Stopping task (io.confluent.connect.jdbc.sink.JdbcSinkTask)
[2019-06-10 04:53:19,749] INFO After filtering the tables are: `test_01`.`example01`,`test_01`.`examplerole` (io.confluent.connect.jdbc.source.TableMonitorThread)

尝试使用单个主题创建接收器连接器，我可以创建接收器连接器。在单个配置中创建多个接收器连接器时面临上述问题

---

请建议使用

topics.regex和auto.create

选项创建JDBC多接收器连接器的配置选项，如果指定

topics.regex

，则不要指定

主题。而不是

   "topics": [],
   "topics.regex": "example*",

你只需要

   "topics.regex": "example*",

如果要指定
topics.regex
，则不要指定
topics
。而不是

   "topics": [],
   "topics.regex": "example*",

你只需要

   "topics.regex": "example*",

所以你想把每一个话题都放在一边？或者您希望基于键关系在表之间联接？使用此处共享的配置时是否出现错误？错误WorkerSinkTask{id=sink-0}任务引发了未捕获且不可恢复的异常（org.apache.kafka.connect.runtime.WorkerTask）org.apache.kafka.connect.errors.ConnectException:org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError（RetryWithToleranceOperator.java:178）中的错误处理程序超出了容差，位于org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute（RetryWith Tolekeep进入日志；这只是任务已结束的一个指示器。请参阅故障诊断部分Hanks以获取信息@RobinMoffatt我看到了与此相关的问题，因此您希望按原样接收每个主题？或者您希望基于键关系在表之间联接？您是否使用已设置的配置时出错是否在此处重试？错误WorkerSinkTask{id=sink-0}任务引发了未捕获且不可恢复的异常（org.apache.kafka.connect.runtime.WorkerTask）org.apache.kafka.connect.errors.ConnectException:在org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError上的错误处理程序中超出了容差（RetryWithToleranceOperator.java:178）位于org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute（请继续查看日志；这只是任务已结束的一个指示器。请参阅故障排除章节了解@RobinMoffatt的信息。我看到了与此相关的问题，我提到了上述格式的配置。请检查上述配置EP，并请检查我写的内容：）在第一个示例中，您指定了
主题
和
主题.regex
，我的观点是您根本不需要
主题
。请检查编辑的配置，创建了接收器连接器，但没有创建表，但我看不到任何错误日志[2019-06-10 10:12:59796]INFO已完成创建连接器MySQL JDBC接收器连接器（org.apache.kafka.connect.runtime.Worker）[2019-06-10 10:12:59796]INFO SinkConnector配置值：config.action.reload=restart当我尝试通过合流UI创建接收器连接器时（9021），无法跳过主题的选项，因此我使curl请求仍然面临相同的问题我提到了上述格式中的配置。请检查上述配置EP，并请检查我写的内容：）在第一个示例中，您指定了
主题
和
主题.regex
，我的观点是您根本不需要
主题
。请检查编辑的配置，创建了接收器连接器，但没有创建表，但我看不到任何错误日志[2019-06-10 10:12:59796]INFO已完成创建连接器MySQL JDBC sink connector（org.apache.kafka.connect.runtime.Worker）[2019-06-10 10:12:59796]INFO sinkconnector配置值：config.action.reload=restart当我尝试通过合流UI（9021）创建sink connector时，无法跳过主题选项，因此我使curl请求仍然面临相同的问题