Amazon s3 使用403写入S3时，EMR上运行的Spark偶尔会失败

我有一份在AWS EMR上运行的spark工作。该作业执行一些数据处理，并将数据帧作为csv写入s3

作业在写入s3时偶尔会失败，并抛出403。我几乎可以肯定，这不是权限的问题，因为作业几乎70%的时间都成功完成，并且将输出毫无问题地写入s3，但我时不时会收到这样的消息

Traceback (most recent call last):
  File "/home/sparkProcessor/controller.py", line 58, in <module>
    processor.process()
  File "/home/sparkProcessor/processor.py", line 32, in process
    self.write_outputs(df1, df2, df3)
  File "/home/sparkProcessor/processor.py", line 168, in write_outputs
    df1.coalesce(1).write.csv(self.configurations['OUTPUT_DIRECTORY'] + "/" + dir_name+ "/output", mode="overwrite", header="true")
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 931, in csv
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o199.csv.
: org.apache.spark.SparkException: Job aborted.
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:198)
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:156)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
    at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
    at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
    at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
    at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
    at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
    at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
    at org.apache.spark.sql.DataFrameWriter.csv(DataFrameWriter.scala:664)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.AccessDeniedException: s3a://sparkOutputBucket/output/_temporary/0/task_20200402173032_0032_m_000000/part-00000-47cc7efd-8dfe-4d5c-8e20-c5b3ab08278b-c000.csv: getFileStatus on s3a://sparkOutputBucket/output/_temporary/0/task_20200402173032_0032_m_000000/part-00000-47cc7efd-8dfe-4d5c-8e20-c5b3ab08278b-c000.csv: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: <request ID>; S3 Extended Request ID: <request ID>), S3 Extended Request ID: <Request ID>
    at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:158)
    at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:101)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1568)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:707)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:662)
    at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:457)
    at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:471)
    at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:388)
    at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:360)
    at org.apache.hadoop.mapreduce.lib.output.DirectFileOutputCommitter.commitJob(DirectFileOutputCommitter.java:111)
    at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:166)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:187)
    ... 33 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: <Request ID>; S3 Extended Request ID: <Request ID>), S3 Extended Request ID: <Request ID>
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4921)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4867)
    at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1320)
    at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1294)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:904)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1553)
    ... 42 more

回溯（最近一次呼叫最后一次）：
文件“/home/sparkProcessor/controller.py”，第58行，在
process（）处理器
文件“/home/sparkProcessor/processor.py”，第32行，正在处理中
自写入输出（df1、df2、df3）
写入输出中的文件“/home/sparkProcessor/processor.py”，第168行
df1.coalesce（1）.write.csv（self.configurations['OUTPUT\u DIRECTORY']+“/”+dir\u name+“/OUTPUT”，mode=“overwrite”，header=“true”）
csv格式的文件“/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py”，第931行
文件“/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py”，第1257行，在__
文件“/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py”，第63行，deco格式
文件“/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py”，第328行，在get_return_值中
py4j.protocol.Py4JJavaError:调用o199.csv时出错。
：org.apache.spark.sparkeexception:作业已中止。
位于org.apache.spark.sql.execution.datasources.FileFormatWriter$.write（FileFormatWriter.scala:198）
位于org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run（InsertIntoHadoopFsRelationCommand.scala:159）
位于org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute（commands.scala:104）
位于org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult（commands.scala:102）
位于org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute（commands.scala:122）
位于org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply（SparkPlan.scala:131）
位于org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply（SparkPlan.scala:127）
位于org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply（SparkPlan.scala:156）
位于org.apache.spark.rdd.RDDOperationScope$.withScope（RDDOperationScope.scala:151）
位于org.apache.spark.sql.execution.SparkPlan.executeQuery（SparkPlan.scala:152）
位于org.apache.spark.sql.execution.SparkPlan.execute（SparkPlan.scala:127）
位于org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute（QueryExecution.scala:80）
位于org.apache.spark.sql.execution.QueryExecution.toRdd（QueryExecution.scala:80）
位于org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply（DataFrameWriter.scala:676）
位于org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply（DataFrameWriter.scala:676）
位于org.apache.spark.sql.execution.SQLExecution$$anonfun$和newexecutionid$1.apply（SQLExecution.scala:78）
在org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated（SQLExecution.scala:125）
位于org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId（SQLExecution.scala:73）
位于org.apache.spark.sql.DataFrameWriter.runCommand（DataFrameWriter.scala:676）
位于org.apache.spark.sql.DataFrameWriter.saveToV1Source（DataFrameWriter.scala:285）
位于org.apache.spark.sql.DataFrameWriter.save（DataFrameWriter.scala:271）
位于org.apache.spark.sql.DataFrameWriter.save（DataFrameWriter.scala:229）
位于org.apache.spark.sql.DataFrameWriter.csv（DataFrameWriter.scala:664）
在sun.reflect.NativeMethodAccessorImpl.invoke0（本机方法）处
位于sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:62）
在sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）中
位于java.lang.reflect.Method.invoke（Method.java:498）
位于py4j.reflection.MethodInvoker.invoke（MethodInvoker.java:244）
位于py4j.reflection.ReflectionEngine.invoke（ReflectionEngine.java:357）
在py4j.Gateway.invoke处（Gateway.java:282）
位于py4j.commands.AbstractCommand.invokeMethod（AbstractCommand.java:132）
在py4j.commands.CallCommand.execute（CallCommand.java:79）
在py4j.GatewayConnection.run处（GatewayConnection.java:238）
运行（Thread.java:748）
原因：java.nio.file.AccessDeniedException:s3a://sparkOutputBucket/output/_temporary/0/task\u 2020040217332\u m\u000000/part-00000-47cc7efd-8dfe-4d5c-8e20-c5b3ab08278b-c000.csv:s3a://sparkoutputbuctbucket/output/_temporary/0/task\u 2020040217332\u m\u000000/part-00000-47EFD-8E-C58E-B8CSV:com.amazonaws.services.s3.model.amazons3异常：禁止（服务：Amazon s3；状态代码：403；错误代码：403禁止；请求ID:；s3扩展请求ID:），s3扩展请求ID:
位于org.apache.hadoop.fs.s3a.S3AUtils.translateException（S3AUtils.java:158）
位于org.apache.hadoop.fs.s3a.S3AUtils.translateException（S3AUtils.java:101）
位于org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus（S3AFileSystem.java:1568）
位于org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename（S3AFileSystem.java:707）
位于org.apache.hadoop.fs.s3a.S3AFileSystem.rename（S3AFileSystem.java:662）
位于org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePath（FileOutputCommitter.java:457）
位于org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePath（FileOutputCommitter.java:471）
位于org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal（FileOutputCommitter.java:388）
位于org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob（FileOutputCommitter.java:360）
位于org.apache.hadoop.mapreduce.lib.output.DirectFileOutputCommitter.commitJob（DirectFileOutputCommitter.java:111）
在org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob（HadoopMapReduceCommitProtocol.scala:166）
位于org.apache.spark.sql.execution.datasources.FileFormatWriter$.write（FileFormatWriter.scala:187）
...