Hadoop:使用flume假脱机目录将csv文件复制到hdfs,错误:INFO source.SpoolDirectorySource:假脱机目录源运行程序已关闭
我正在尝试使用flume spool dir将csv文件复制到hdfs。因为我是Hadoop概念的初学者。请帮助我解决以下问题 hdfs目录:/home/hdfsHadoop:使用flume假脱机目录将csv文件复制到hdfs,错误:INFO source.SpoolDirectorySource:假脱机目录源运行程序已关闭,hadoop,hadoop-streaming,flume-ng,Hadoop,Hadoop Streaming,Flume Ng,我正在尝试使用flume spool dir将csv文件复制到hdfs。因为我是Hadoop概念的初学者。请帮助我解决以下问题 hdfs目录:/home/hdfs 水槽方向:/etc/flume/ 请查找flume-hwdgteam01.conf文件,如下所示 # Define a source, a channel, and a sink hwdgteam01.sources = src1 hwdgteam01.channels = chan1 hwdgteam01.sinks = sink1
水槽方向:/etc/flume/ 请查找flume-hwdgteam01.conf文件,如下所示
# Define a source, a channel, and a sink
hwdgteam01.sources = src1
hwdgteam01.channels = chan1
hwdgteam01.sinks = sink1
# Set the source type to Spooling Directory and set the directory
# location to /home/flume/ingestion/
hwdgteam01.sources.src1.type = spooldir
hwdgteam01.sources.src1.spoolDir = /home/hwdgteam01/nandan/input-data
hwdgteam01.sources.src1.basenameHeader = true
# Configure the channel as simple in-memory queue
hwdgteam01.channels.chan1.type = memory
# Define the HDFS sink and set its path to your target HDFS directory
hwdgteam01.sinks.sink1.type = hdfs
hwdgteam01.sinks.sink1.hdfs.path = /home/datalanding
hwdgteam01.sinks.sink1.hdfs.fileType = DataStream
# Disable rollover functionallity as we want to keep the original files
hwdgteam01.sinks.sink1.rollCount = 0
hwdgteam01.sinks.sink1.rollInterval = 0
hwdgteam01.sinks.sink1.rollSize = 0
hwdgteam01.sinks.sink1.idleTimeout = 0
# Set the files to their original name
hwdgteam01.sinks.sink1.hdfs.filePrefix = %{basename}
# Connect source and sink
hwdgteam01.sources.src1.channels = chan1
hwdgteam01.sinks.sink1.channel = chan1
我执行命令的方式如下:
/usr/bin/flume ng agent--conf conf--conf file/home/hwdgteam01/nandan/config/flume-hwdgteam01.conf-Dflume.root.logger=DEBUG,console--name hwdgteam01
或
/usr/bin/flume ng agent-n hwdgteam01-f/home/hwdgteam01/nandan/config/flume-hwdgteam01.conf
/usr/bin/flume ng agent-n hwdgteam01-f/home/hwdgteam01/nandan/config/flume-hwdgteam01.conf
或
/home/hwdgteam01/nandan/config/flume ng代理-n hwdgteam01-f
或
/home/hwdgteam01/nandan/config/flume-hwdgteam01.conf
但是什么都没有解决,我得到了下面的错误
请告诉我哪里出了问题。
谢谢你的帮助