Amazon web services AmazonS3异常:尝试使用S3AFilesystem时请求错误400

Amazon web services AmazonS3异常:尝试使用S3AFilesystem时请求错误400,amazon-web-services,hadoop,amazon-s3,aws-sdk,Amazon Web Services,Hadoop,Amazon S3,Aws Sdk,我有一段java代码,它试图使用coniguration从S3初始化远程文件系统(这是以前在HDFS上实现的,我尝试将其移动到S3,而不需要对代码进行太多修改) 这是配置文件: fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain fs.defaultFS=s3a://mybucket.devrun.algo-resources/ 然后,在我使用的设置中 hdfsFileSys

我有一段java代码,它试图使用coniguration从S3初始化远程文件系统(这是以前在HDFS上实现的,我尝试将其移动到S3,而不需要对代码进行太多修改)

这是配置文件:

fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain
fs.defaultFS=s3a://mybucket.devrun.algo-resources/
然后,在我使用的设置中

hdfsFileSystem = FileSystem.get(conf);
这将导致以下异常:

org.apache.hadoop.fs.s3a.AWSS3IOException: doesBucketExist on mybucket.devrun.algo-resources: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 1E91F85FA3751C44), S3 Extended Request ID: 5KDgH7lsaIX7l5DQcdBdUjeg/qxYgOEU4WJBOL0p090kqNNlYOAie31zuYUQw+R3LN4CvavdVJk=: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 1E91F85FA3751C44)
    at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:178)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:282)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:236)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181)
    at pipeline.HdfsSyncUtils.<init>(HdfsSyncUtils.java:32)
    at pipeline.QueriesForArticles$QFAMapper.setup(QueriesForArticles.java:158)
    at AWSPipeline.C2SIndexSearchingAlgo.<init>(C2SIndexSearchingAlgo.java:41)
    at AWSPipeline.ABTestMainRunner.main(ABTestMainRunner.java:27)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 1E91F85FA3751C44), S3 Extended Request ID: 5KDgH7lsaIX7l5DQcdBdUjeg/qxYgOEU4WJBOL0p090kqNNlYOAie31zuYUQw+R3LN4CvavdVJk=
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
    at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1107)
    at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1070)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:276)
    ... 11 more
org.apache.hadoop.fs.s3a.AWSS3IOException:doesbucketexiston-mybucket.devrun.algo-resources:com.amazonaws.services.s3.model.AmazonS3Exception:Bad请求(服务:Amazon s3;状态代码:400;错误代码:400错误请求;请求ID:1E91F85FA3751C44),S3扩展请求ID:5KDgH7lsaIX7l5DQcdBdUjeg/qxYgOEU4WJBOL0p090kqNNlYOAie31zuYUQw+R3LN4CvavdVJk=:错误请求(服务:Amazon S3;状态代码:400;错误代码:400错误请求;请求ID:1E91F85FA3751C44)
位于org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:178)
位于org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:282)
位于org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:236)
位于org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811)
位于org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
位于org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848)
位于org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830)
位于org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
位于org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181)
位于pipeline.HdfsSyncUtils.(HdfsSyncUtils.java:32)
位于pipeline.querysforarticles$QFAMapper.setup(querysforarticles.java:158)
在AWSPipeline.C2SIndexSearchingAlgo。(C2SIndexSearchingAlgo.java:41)
位于AWSPipeline.abtestmainlunner.main(abtestmainlunner.java:27)
原因:com.amazonaws.services.s3.model.amazons3异常:错误请求(服务:Amazon s3;状态代码:400;错误代码:400错误请求;请求ID:1E91F85FA3751C44),s3扩展请求ID:5KDgH7lsaIX7l5DQcdBdUjeg/QXYGOEU4WJBOL0P090KQNNLYOIE31ZUYUQW+R3LN4CvavdVJk=
在com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
在com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)上
在com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
在com.amazonaws.http.AmazonHttpClient.execute上(AmazonHttpClient.java:310)
位于com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
位于com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1107)
位于com.amazonaws.services.s3.AmazonS3Client.doesbuckeest(AmazonS3Client.java:1070)
位于org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:276)
... 还有11个
我使用的是hadoop aws.jar
version
2.8.0

我使用
awsfed
~\.aws
下创建凭据文件,因此我认为我的凭据是正确的

你知道这个错误是什么意思吗?没有详细的错误消息

编辑:

对于任何感兴趣的人,我解决了这个问题: 随后,我得出结论,这与地区有关。我的桶在美国东部2区。我试着在另一个地区打开一个水桶,结果成功了


这可能与可能的情况有关-us-east-2中的S3只支持版本4签名,而我的代码(
hadoop-aws.jar 2.8.0
)可能使用的是旧版本。

非常感谢。我花了几个小时调试我的bukcet在美国东部2区的位置。在另一个地区重新创建了水桶,效果非常好!!