Amazon web services 使用EMRCreateJobFlowOperator创建EMR群集

我正在尝试使用气流中的EMRCreateJobFlowOperator启动EMR群集，但任务始终失败，返回代码-11（请参阅下面的错误日志）：

我已将我的AWS凭据（管理员用户）添加到Airflow中的AWS_default和emr_default中。我还使用CLI创建了默认角色

我已经测试过使用CLI创建集群（请参见下面的代码）：

aws emr创建集群--名称测试\u集群--使用默认角色--发布标签emr-5.28.0--实例计数2--应用程序名称=Spark name=齐柏林飞艇--ec2属性KeyName=XXX--实例类型m5.xlarge--实例计数3--自动终止

…这很好用。我正在寻找一些想法来帮助这项工作，我真的被卡住了

谢谢

> [2020-06-20 10:39:24,054] {logging_mixin.py:112} INFO - Running %s on
> host %s <TaskInstance: test_emr_cluster_creation.create_job_flow
> 2020-06-20T00:00:00+00:00 [running]> richards-macbook-air.local
> [2020-06-20 10:39:24,087] {emr_create_job_flow_operator.py:66} INFO -
> Creating JobFlow using aws-conn-id: aws_default, emr-conn-id:
> emr_default [2020-06-20 10:39:33,907] {logging_mixin.py:112} INFO -
> [2020-06-20 10:39:33,906] {local_task_job.py:103} INFO - Task exited
> with return code -11

from datetime import datetime, timedelta
from airflow import DAG
from airflow.contrib.operators.emr_create_job_flow_operator import (
    EmrCreateJobFlowOperator,
)

DEFAULT_ARGS = {
    "owner": "XXX",
    "start_date": datetime(2020, 6, 20),
    "end_date": datetime(2020, 6, 20),
    "depends_on_past": False,
    "retries": 0,
    "retry_delay": timedelta(minutes=5),
    "catchup": False,
    "email": ["xxxxx@hotmail.com"],
    "email_on_failure": False,
    "email_on_retry": False,
}

JOB_FLOW_OVERRIDES = {
    'Name': 'test_cluster',
    'ReleaseLabel': 'emr-5.28.0',
    'Instances': {
        'Ec2KeyName': 'XXX',
        'InstanceGroups': [
            {
                'Name': 'Master node',
                'Market': 'SPOT',
                'InstanceRole': 'MASTER',
                'InstanceType': 'm5.xlarge',
                'InstanceCount': 3,
            }
        ],
        'KeepJobFlowAliveWhenNoSteps': True,
        'TerminationProtected': True,
    },
    'JobFlowRole': 'EMR_EC2_DefaultRole',
    'ServiceRole': 'EMR_DefaultRole',
}

with DAG(
    dag_id="test_emr_cluster_creation",
    default_args=DEFAULT_ARGS,
    dagrun_timeout=timedelta(hours=2),
    schedule_interval="@once",
) as dag:

    # Start the cluster
    cluster_creator = EmrCreateJobFlowOperator(
        task_id="create_job_flow",
        job_flow_overrides=JOB_FLOW_OVERRIDES,
        aws_conn_id="aws_default",
        emr_conn_id="emr_default",
    )

cluster_creator