在pandas.read\u拼花地板中使用AWS\u配置文件

我正在本地测试它，其中有一个~/.aws/config文件

~/.aws/config看起来像：

[profile a] 
...
[profile b]
...

我还将AWS_PROFILE环境变量设置为“a”

我想读一个文件，其中可以使用pandas访问概要文件b

我可以通过执行以下操作通过

s3fs

访问它：

import s3fs
fs = s3fs.S3FileSystem(profile="b")
fs.get("BUCKET/FILE.parquet", "FILE.parquet")
pd.read_parquet("FILE.parquet")

但是，如果我尝试使用存储选项将其传递给pd.read\u拼花地板，我会得到一个

许可错误：禁止

pd.read_parquet(
    "s3://BUCKET/FILE.parquet",
    storage_options={"profile": "b"},
)

下面是完整的回溯

Traceback (most recent call last):
  File "/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site-packages/s3fs/core.py", line 233, in _call_s3
    out = await method(**additional_kwargs)
  File "/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site-packages/aiobotocore/client.py", line 154, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site-packages/pandas/io/parquet.py", line 459, in read_parquet
    return impl.read(
  File "/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site-packages/pandas/io/parquet.py", line 221, in read
    return self.api.parquet.read_table(
  File "/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site-packages/pyarrow/parquet.py", line 1672, in read_table
    dataset = _ParquetDatasetV2(
  File "/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site-packages/pyarrow/parquet.py", line 1504, in __init__
    if filesystem.get_file_info(path_or_paths).is_file:
  File "pyarrow/_fs.pyx", line 438, in pyarrow._fs.FileSystem.get_file_info
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/_fs.pyx", line 1004, in pyarrow._fs._cb_get_file_info
  File "/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site-packages/pyarrow/fs.py", line 226, in get_file_info
    info = self.fs.info(path)
  File "/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site-packages/fsspec/asyn.py", line 72, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site-packages/fsspec/asyn.py", line 53, in sync
    raise result[0]
  File "/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site-packages/fsspec/asyn.py", line 20, in _runner
    result[0] = await coro
  File "/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site-packages/s3fs/core.py", line 911, in _info
    out = await self._call_s3(
  File "/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site-packages/s3fs/core.py", line 252, in _call_s3
    raise translate_boto_error(err)
PermissionError: Forbidden

回溯（最近一次呼叫最后一次）：
文件“/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site packages/s3fs/core.py”，第233行，在调用s3中
out=等待方法（**额外的_-kwargs）
文件“/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site packages/aiobotocore/client.py”，第154行，在api调用中
引发错误\u类（解析的\u响应、操作\u名称）
botocore.exceptions.ClientError:调用HeadObject操作时出错（403）：禁止
上述异常是以下异常的直接原因：
回溯（最近一次呼叫最后一次）：
文件“”，第1行，在
文件“/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site-packages/pandas/io/parquet.py”，第459行，在read_-parquet中
返回impl.read(
文件“/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site packages/pandas/io/parquet.py”，第221行，已读
返回self.api.parquet.read_表(
文件“/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site packages/pyarrow/parquet.py”，第1672行，在read_表中
数据集=_ParquetDatasetV2(
文件“/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site packages/pyarrow/parquet.py”，第1504行，在__
如果filesystem.get_file_info（路径或路径）。is_file:
文件“pyarrow/_fs.pyx”，第438行，在pyarrow.\u fs.FileSystem.get\u File\u info中
文件“pyarrow/error.pxi”，第122行，在pyarrow.lib.pyarrow\u internal\u check\u状态中
文件“pyarrow/_fs.pyx”，第1004行，在pyarrow.\u fs.\u cb\u获取文件信息
文件“/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site packages/pyarrow/fs.py”，第226行，在get_File_info中
info=self.fs.info（路径）
文件“/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site packages/fsspec/asyn.py”，第72行，在包装器中
返回同步（self.loop、func、*args、**kwargs）
文件“/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site packages/fsspec/asyn.py”，第53行，同步
提升结果[0]
文件“/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site packages/fsspec/asyn.py”，第20行，在
结果[0]=等待coro
文件“/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site packages/s3fs/core.py”，第911行，在_info中
out=等待self.\u呼叫\u s3(
文件“/home/ray/local/bin/anaconda3/envs/main/lib/python3.8/site packages/s3fs/core.py”，第252行，在调用s3中
提升转换错误（err）
许可错误：禁止

注意：有一个老问题与此相关，但没有帮助：