Apache kafka 必须指定flume和Kafka主题

我试图从我的卡夫卡主题中提取数据并将其写入HDFS，我的flume配置似乎与我在几个示例中看到的相同，但我似乎无法回避下面的错误。我可以通过python从主题中使用，所以我知道我在这方面还可以。我使用的是flume版本1.6.0和java 9.0.1。我做错了什么让它不接受卡夫卡的主题

09 Jul 2018 17:17:26,973 INFO  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:145) -Creating channels
09 Jul 2018 17:17:26,984 INFO  [conf-file-poller-0] (org.apache.flume.channel.DefaultChannelFactory.create:42)  - Creating instance of channel kafka_hdfs_channel type memory
09 Jul 2018 17:17:26,989 INFO  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:200)  - Created channel kafka_hdfs_channel

09 Jul 2018 17:17:26,989 INFO  [conf-file-poller-0] (org.apache.flume.source.DefaultSourceFactory.create:41)  - Creating instance of source kafka_source, type org.apache.flume.source.kafka.KafkaSource
09 Jul 2018 17:17:26,993 ERROR [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadSources:361)  - Source kafka_source has been removed due to an error during configuration
org.apache.flume.conf.ConfigurationException: Kafka topic must be specified.
    at org.apache.flume.source.kafka.KafkaSource.configure(KafkaSource.java:180)
    at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:326)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:300)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.base/java.lang.Thread.run(Thread.java:844)}

这是我的水槽配置：

agentCDIS.sources = kafka_source
agentCDIS.channels = kafka_hdfs_channel
agentCDIS.sinks = hdfs_sink

agentCDIS.sources.kafka_source.type = org.apache.flume.source.kafka.KafkaSource
agentCDIS.sources.kafka_source.kafka.bootstrap.servers = 10.4.3.61:9092, 10.4.3.62:9092, 10.4.3.63:9092
agentCDIS.sources.kafka_source.kafka.topic = test
agentCDIS.sources.kafka_source.kafka.consumer.group.id = cn_flume_group
agentCDIS.sources.kafka_source.channels = kafka_hdfs_channel
agentCDIS.sources.kafka_source.interceptors = i1
agentCDIS.sources.kafka_source.interceptors.i1.type = timestamp
agentCDIS.sources.kafka_source.kafka.consumer.timeout.ms = 1000

agentCDIS.channels.kafka_hdfs_channel.type = memory
agentCDIS.channels.kafka_hdfs_channel.capacity = 10000
agentCDIS.channels.kafka_hdfs_channel.transactionCapacity = 1000

agentCDIS.sinks.hdfs_sink.type = hdfs
agentCDIS.sinks.hdfs_sink.hdfs.path = hdfs://10.4.16.16:8020/user/cnelson/kafka/%{topic}/%y-%m-%d
agentCDIS.sinks.hdfs_sink.hdfs.rollInterval = 5
agentCDIS.sinks.hdfs_sink.hdfs.rollSize = 0
agentCDIS.sinks.hdfs_sink.fileType = DataStream
agentCDIS.sinks.hdfs_sink.channel = kafka_hdfs_channel

agentCDIS.sinks.loggerSink.type = logger
agentCDIS.sinks.loggerSink.kafka_hdfs_channel = memoryChannel

agentCDIS.channels.memoryChannel.type = memory
agentCDIS.channels.memoryChannel.capacity = 100

---

我浏览了几次文章和配置，注意到-您提到您正在使用Flume的1.6版，根据，属性略有不同。请您尝试一下：

代替

agentCDIS.sources.kafka\u source.kafka.bootstrap.servers

=>尝试

agentCDIS.sources.kafka\u source.zookeeperConnect

-此属性的值将是kafka群集使用的zookeeper URI

而不是

agentCDIS.sources.kafka\u source.kafka.topic=test

=>尝试

agentCDIS.sources.kafka\u source.topic=test

不要使用

agentCDIS.sources.kafka\u source.kafka.consumer.group.id=cn\u flume\u group

=>尝试

agentCDIS.sources.kafka\u source.groupId=cn\u flume\u group

您在配置文件中使用的上述3个属性是从中引入的

---

我希望这有帮助

打字错误？主题而不是主题我尝试了每个版本的主题。。。我只是回去用卡夫卡再试一次，只是为了确定。同样的错误。根据源代码，它认为你的主题是空的，应该是

topic

No-argument。我已经更新了我的初始帖子，以反映正确的语法（我发布的是我尝试东西的第75次迭代）。无论如何，使用正确的语法会产生完全相同的错误。