Apache kafka 如何在分布式模式下运行Kafka connect replicator?
我想使用confluent的replicator将数据从一个系统复制到另一个系统。我使用两个Ubuntu18.04系统,其中一个作为源,另一个作为目标 我尝试在分布式模式下运行kafka connect replicator,更改了以下配置:Apache kafka 如何在分布式模式下运行Kafka connect replicator?,apache-kafka,apache-kafka-connect,Apache Kafka,Apache Kafka Connect,我想使用confluent的replicator将数据从一个系统复制到另一个系统。我使用两个Ubuntu18.04系统,其中一个作为源,另一个作为目标 我尝试在分布式模式下运行kafka connect replicator,更改了以下配置: 在confluent/etc/kafka/server.properties中,我做了以下更改 在confluent/etc/kafka connect replicator/replicator.connect.distributed.propertie
name=replicator-source
connector.class=io.confluent.connect.replicator.ReplicatorSourceConnector
# source cluster connection info
src.kafka.bootstrap.servers=source.ip:9092
# Set to use direct connection to Zookeeper by Replicator on the source
src.zookeeper.connect=localhost:2181
# destination cluster connection info
dest.kafka.bootstrap.servers=destination.ip:9092
# Set to use direct connection to Zookeeper by Replicator on the destination
dest.zookeeper.connect=destination.ip:2181
# configure topics to replicate
topic.whitelist= test-topic
topic.rename.format=${topic}.replica
目的地
name=replicator-source
connector.class=io.confluent.connect.replicator.ReplicatorSourceConnector
# source cluster connection info
src.kafka.bootstrap.servers=source.ip:9092
# Set to use direct connection to Zookeeper by Replicator on the source
src.zookeeper.connect=source.ip:2181
# destination cluster connection info
dest.kafka.bootstrap.servers=destination.ip:9092
# Set to use direct connection to Zookeeper by Replicator on the destination
dest.zookeeper.connect=destination.ip:2181
# configure topics to replicate
topic.whitelist= test-topic
topic.rename.format=${topic}.replica
然后我在源系统中创建了topic,并使用下面的命令启动了连接器
PATH\u TO\u CONFLUENT>sudo./bin/connect distributed./etc/kafka connect replicator/replicator-connect-distributed.properties./etc/kafka connect replicator/quickstart replicator.properties
在此之后,我从源系统生成了主题中的数据,并尝试使用主题名{topic}.replica在目标系统中使用,但没有可用于使用的主题。不清楚您有哪些错误,但有一些注意事项
connectdistributed
只获取一个属性文件,而不是两个。您可以将属性作为JSON以HTTP方式发布到Connect集群,而不是在集群启动期间加载属性文件。快速启动文件用于connectstandalone
{"name": "your-replicator-name", "config": {
"src.kafka.bootstrap.servers": "...",
...
}
/etc/kafka/connect distributed.properties
应该是以分布式模式运行任何connect或Replicator群集的起点,尽管在Replicator connect distributed.properties中可能有类似的配置
bootstrap.server
应该只指向一个集群。源和目标将在src.kafka.bootstrap.servers
和dest.kafka.bootstrap.servers
是的,我知道了。我终于能够将数据从源系统复制到目标系统。我在源系统中创建主题,并在目标系统中运行confluent replicator。
name=replicator-source
connector.class=io.confluent.connect.replicator.ReplicatorSourceConnector
# source cluster connection info
src.kafka.bootstrap.servers=source.ip:9092
# Set to use direct connection to Zookeeper by Replicator on the source
src.zookeeper.connect=source.ip:2181
# destination cluster connection info
dest.kafka.bootstrap.servers=destination.ip:9092
# Set to use direct connection to Zookeeper by Replicator on the destination
dest.zookeeper.connect=destination.ip:2181
# configure topics to replicate
topic.whitelist= test-topic
topic.rename.format=${topic}.replica
{"name": "your-replicator-name", "config": {
"src.kafka.bootstrap.servers": "...",
...
}