Apache flink Flink使用Presto写入S3

我想使用presto接口和BucketingSink从Flink 1.4.2写入S3。我按照说明添加了flink-conf.yaml s3.access-key和s3.secret-key，并将flink-s3-fs-presto-1.4.2.jar放在lib文件夹中。下面是产生的错误

若作业是在AWS环境中执行的，我希望根本不需要设置密钥。我相信这个假设是正确的

java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).
at org.apache.hadoop.fs.s3.S3Credentials.initialize(S3Credentials.java:70)
at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.initialize(Jets3tFileSystemStore.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy17.initialize(Unknown Source)
at org.apache.hadoop.fs.s3.S3FileSystem.initialize(S3FileSystem.java:91)
at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.createHadoopFileSystem(BucketingSink.java:1206)
at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initFileSystem(BucketingSink.java:411)
at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:355)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
at java.lang.Thread.run(Thread.java:748)

---

该应用程序似乎根本没有使用

flink-s3-fs-presto

，而是使用Hadoop不推荐使用的旧s3文件系统。粘贴的堆栈跟踪表明未为文件系统方案“s3://”拾取

flink-s3-fs-presto

请确保

flink-s3-fs-presto

JAR文件确实位于执行作业的TaskManager的

lib

文件夹中，而不仅仅是在客户端上

• 当您使用Thread或Mesos部署Flink作业时，这应该会自动发生
• 通过容器部署Flink时，请确保JAR文件位于容器映像的lib文件夹中
• 单独或手动运行Flink TaskManager时，请确保集群中的所有TaskManager在启动之前都将JAR文件保存在lob文件夹中

---

你的flink-conf.yaml看起来怎么样？请不要发布真实的凭证！Hadoop实现是否已被弃用？你能提供一个信息来源吗？