Hadoop 使用Flume将数据从kafka摄取到HDFS::ConfigurationException:必须指定引导服务器
我正在尝试使用flume将数据从kafka源摄取到hdfs。下面是我的flume配置文件Hadoop 使用Flume将数据从kafka摄取到HDFS::ConfigurationException:必须指定引导服务器,hadoop,apache-kafka,flume,flume-ng,Hadoop,Apache Kafka,Flume,Flume Ng,我正在尝试使用flume将数据从kafka源摄取到hdfs。下面是我的flume配置文件 flume1.sources = kafka-source-1 flume1.channels = hdfs-channel-1 flume1.sinks = hdfs-sink-1 flume1.sources.kafka-source-1.type = org.apache.flume.source.kafka.KafkaSource flume1.sources.kafka-source-1.boot
flume1.sources = kafka-source-1
flume1.channels = hdfs-channel-1
flume1.sinks = hdfs-sink-1
flume1.sources.kafka-source-1.type = org.apache.flume.source.kafka.KafkaSource
flume1.sources.kafka-source-1.bootstrap.servers = localhost:9092
flume1.sources.kafka-source-1.zookeeperConnect = localhost:2181
flume1.sources.kafka-source-1.topic = MYNEWSFEEDS
flume1.sources.kafka-source-1.batchSize = 100
flume1.sources.kafka-source-1.channels = hdfs-channel-1
flume1.channels.hdfs-channel-1.type = memory
flume1.sinks.hdfs-sink-1.channel = hdfs-channel-1
flume1.sinks.hdfs-sink-1.type = hdfs
flume1.sinks.hdfs-sink-1.hdfs.writeFormat = Text
flume1.sinks.hdfs-sink-1.hdfs.fileType = DataStream
flume1.sinks.hdfs-sink-1.hdfs.filePrefix = test-events
flume1.sinks.hdfs-sink-1.hdfs.useLocalTimeStamp = true
flume1.sinks.hdfs-sink-1.hdfs.path = hdfs://quickstart.cloudera:8020/tmp
flume1.sinks.hdfs-sink-1.hdfs.rollCount=100
flume1.sinks.hdfs-sink-1.hdfs.rollSize=0
flume1.channels.hdfs-channel-1.capacity = 10000
flume1.channels.hdfs-channel-1.transactionCapacity = 1000
我正在使用以下命令运行flume agent:
sudo flume-ng agent --name flume1 --conf-file '/etc/flume-ng/conf/flafka.conf' Dflume.root.logger=TRACE,console
但我得到了以下错误:
18/03/12 16:49:18错误节点。AbstractConfigurationProvider:源
由于配置过程中出现错误,kafka-source-1已被删除
org.apache.flume.conf.ConfigurationException:引导服务器必须
指定于
org.apache.flume.source.kafka.KafkaSource.doConfigure(KafkaSource.java:330)
在
org.apache.flume.source.BasicSourceSemantics.configure(BasicSourceSemantics.java:65)
在
org.apache.flume.source.AbstractPollableSource.configure(AbstractPollableSource.java:63)
在
org.apache.flume.conf.Configurables.configure(Configurables.java:41)
在
org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:326)
在
org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97)
在
org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
在
Executors$RunnableAdapter.call(Executors.java:471)
位于java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
在
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
在
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
在
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
在
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
运行(Thread.java:745)
虽然我已经在conf文件中指定了引导服务器,但它仍然会给出相同的错误。尝试了许多排列和组合,但没有成功。根据,您应该更换
flume1.sources.kafka-source-1.bootstrap.servers = localhost:9092
与
除了卡夫卡连接,你还用水槽吗?
flume1.sources.kafka-source-1.kafka.bootstrap.servers = localhost:9092