Hadoop 使用ec2实例IAM凭据将Flume接收器连接到s3

我有一个水槽水槽，可以将数据写入aws s3存储桶。 水槽配置如下所示

   aggregator.sinks.s3LogSink.type=hdfs
   aggregator.sinks.s3LogSink.channel=flumeLogAgentFileChannel
   aggregator.sinks.s3LogSink.hdfs.path=s3n://<aws bucket name>
   aggregator.sinks.s3LogSink.hdfs.fileType=DataStream
   aggregator.sinks.s3LogSink.hdfs.writeFormat=Text
   aggregator.sinks.s3LogSink.hdfs.rollCount = 0
   aggregator.sinks.s3LogSink.hdfs.rollSize = 67108864
   aggregator.sinks.s3LogSink.hdfs.batchSize = 1000
   aggregator.sinks.s3LogSink.hdfs.rollInterval = 0

当我开始水槽流程时。。我得到下面的错误

2016-04-18 18:03:06,326 (hdfs-s3LogSink-call-runner-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-04-18 18:03:06,359 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed
java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem could not be instantiated
        at java.util.ServiceLoader.fail(ServiceLoader.java:224)
        at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
        at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
        at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2623)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235)
        at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
        at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
        at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: com/amazonaws/event/ProgressListener
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)
        at java.lang.Class.getConstructor0(Class.java:2895)
        at java.lang.Class.newInstance(Class.java:354)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
        ... 18 more
Caused by: java.lang.ClassNotFoundException: com.amazonaws.event.ProgressListener
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 23 more

2016-04-18 18:03:06326（hdfs-s3LogSink-call-runner-0）[WARN-org.apache.hadoop.util.NativeCodeLoader.（NativeCodeLoader.java:62）]无法为您的平台加载本机hadoop库。。。在适用的情况下使用内置java类
2016-04-18 18:03:06359（SinkRunner PollingRunner DefaultSinkProcessor）[错误-org.apache.flume.sink.hdfs.HDFSEventSink.process（HDFSEventSink.java:459）]进程失败
java.util.ServiceConfigurationError:org.apache.hadoop.fs.FileSystem:Provider org.apache.hadoop.fs.s3a.S3AFileSystem无法实例化
在java.util.ServiceLoader.fail处（ServiceLoader.java:224）
在java.util.ServiceLoader.access$100（ServiceLoader.java:181）
位于java.util.ServiceLoader$LazyIterator.next（ServiceLoader.java:377）
位于java.util.ServiceLoader$1.next（ServiceLoader.java:445）
位于org.apache.hadoop.fs.FileSystem.loadFileSystems（FileSystem.java:2623）
位于org.apache.hadoop.fs.FileSystem.getFileSystemClass（FileSystem.java:2634）
位于org.apache.hadoop.fs.FileSystem.createFileSystem（FileSystem.java:2651）
位于org.apache.hadoop.fs.FileSystem.access$200（FileSystem.java:92）
位于org.apache.hadoop.fs.FileSystem$Cache.getInternal（FileSystem.java:2687）
位于org.apache.hadoop.fs.FileSystem$Cache.get（FileSystem.java:2669）
位于org.apache.hadoop.fs.FileSystem.get（FileSystem.java:371）
位于org.apache.hadoop.fs.Path.getFileSystem（Path.java:295）
在org.apache.flume.sink.hdfs.BucketWriter$1.call（BucketWriter.java:243）上
在org.apache.flume.sink.hdfs.BucketWriter$1.call（BucketWriter.java:235）上
位于org.apache.flume.sink.hdfs.BucketWriter$9$1.run（BucketWriter.java:679）
在org.apache.flume.auth.SimpleAuthenticator.execute（SimpleAuthenticator.java:50）
在org.apache.flume.sink.hdfs.BucketWriter$9.call（BucketWriter.java:676）上
在java.util.concurrent.FutureTask.run（FutureTask.java:262）处
位于java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1145）
位于java.util.concurrent.ThreadPoolExecutor$Worker.run（ThreadPoolExecutor.java:615）
运行（Thread.java:745）
原因：java.lang.NoClassDefFoundError:com/amazonaws/event/ProgressListener
位于java.lang.Class.getDeclaredConstructors0（本机方法）
位于java.lang.Class.privateGetDeclaredConstructors（Class.java:2595）
位于java.lang.Class.getConstructor0（Class.java:2895）
位于java.lang.Class.newInstance（Class.java:354）
位于java.util.ServiceLoader$LazyIterator.next（ServiceLoader.java:373）
... 还有18个
原因：java.lang.ClassNotFoundException:com.amazonaws.event.ProgressListener
在java.net.URLClassLoader$1.run（URLClassLoader.java:366）
在java.net.URLClassLoader$1.run（URLClassLoader.java:355）
位于java.security.AccessController.doPrivileged（本机方法）
位于java.net.URLClassLoader.findClass（URLClassLoader.java:354）
位于java.lang.ClassLoader.loadClass（ClassLoader.java:425）
位于sun.misc.Launcher$AppClassLoader.loadClass（Launcher.java:308）
位于java.lang.ClassLoader.loadClass（ClassLoader.java:358）
... 23多

---

有人能帮忙吗？

到bucket的正确路径是

s3n://:@

到bucket的正确路径是

s3n://:@

2016-04-18 18:03:06,326 (hdfs-s3LogSink-call-runner-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-04-18 18:03:06,359 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed
java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem could not be instantiated
        at java.util.ServiceLoader.fail(ServiceLoader.java:224)
        at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
        at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
        at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2623)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235)
        at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
        at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
        at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: com/amazonaws/event/ProgressListener
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)
        at java.lang.Class.getConstructor0(Class.java:2895)
        at java.lang.Class.newInstance(Class.java:354)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
        ... 18 more
Caused by: java.lang.ClassNotFoundException: com.amazonaws.event.ProgressListener
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 23 more