Airflow 如何设置在Cloud Composer中运行KubernetesPodOperator的适当权限?

Airflow 如何设置在Cloud Composer中运行KubernetesPodOperator的适当权限?,airflow,google-cloud-composer,Airflow,Google Cloud Composer,我正试图按照这里的文档在我的Composer环境中运行一个简单的KubernetesPodOperator 由于缺少用户“默认”的权限,气流运行时失败 也就是说,如何正确地创建环境或设置默认用户权限以使此代码正常工作 达格: price\u analysis=KubernetesPodOperator( 任务_id='price-analysis', name='price-analysis', namespace='default', image='bash', 图像\u pull\u pol

我正试图按照这里的文档在我的Composer环境中运行一个简单的KubernetesPodOperator

由于缺少用户“默认”的权限,气流运行时失败

也就是说,如何正确地创建环境或设置默认用户权限以使此代码正常工作

达格:

price\u analysis=KubernetesPodOperator(
任务_id='price-analysis',
name='price-analysis',
namespace='default',
image='bash',
图像\u pull\u policy='Always',
cmds=['echo'],
参数=['something'],
环境变量={
“暴露阈值”:“5”,
“估计窗口”:“3,7”,
},
in_cluster=True,
)
日志:

-------------------------------------------------------------------------------
开始尝试1次,共次
-------------------------------------------------------------------------------
[2019-04-0314:54:15611]{models.py:1595}INFO-执行日期2019-04-03T14:53:59.658367+00:00
[2019-04-0314:54:15612]{base_task_runner.py:118}信息-运行:[bash','-c',u'气流运行增值税分析价格分析2019-04-03T14:53:59.658367+00:00--job_id 54--raw-sd DAGS_文件夹/vatu analysis_dag.py--cfg_路径/tmp/tmp3RdZOV']
[2019-04-03 14:54:18375]{base_task_runner.py:101}信息-作业54:子任务价格分析[2019-04-03 14:54:18374]{settings.py:176}信息-设置。配置_orm():使用池设置。池大小=5,池回收=1800
[2019-04-03 14:54:19652]{base_task_runner.py:101}信息-作业54:子任务价格分析[2019-04-03 14:54:19651]{default_芹菜.py:80}警告-您已经配置了redis://airflow-redis-service.default.svc.cluster.local:6379/0,强烈建议使用另一个result_后端(即数据库)。
[2019-04-03 14:54:19659]{base_task_runner.py:101}信息-作业54:子任务价格分析[2019-04-03 14:54:19659]{{uuuu init_uuu.py:51}信息-使用executor CeleryExecutor
[2019-04-03 14:54:19826]{base_task_runner.py:101}信息-作业54:子任务价格分析[2019-04-03 14:54:19825]{app.py:51}警告-使用默认编写器环境变量。覆盖尚未应用。
[2019-04-03 14:54:19842]{base_task_runner.py:101}信息-作业54:子任务价格分析[2019-04-03 14:54:19842]{configuration.py:516}信息-从/etc/aiffair/aiffaift.cfg读取配置
[2019-04-03 14:54:19868]{base_task_runner.py:101}信息-作业54:子任务价格分析[2019-04-03 14:54:19867]{configuration.py:516}信息-从/etc/aiffair/aiffaift.cfg读取配置
[2019-04-03 14:54:20380]{base_task_runner.py:101}信息-作业54:子任务价格分析[2019-04-03 14:54:20378]{models.py:271}信息-从/home/aiffair/gcs/dags/vat_analysis_dag.py填充数据包
[2019-04-03 14:54:21490]{base_task_runner.py:101}信息-作业54:子任务价格分析[2019-04-03 14:54:21490]{cli.py:484}信息-在主机上运行-worker-5b6d7c75c9-w6995
[2019-04-03 14:54:22093]{base_task_runner.py:101}信息-作业54:子任务价格分析[2019-04-03 14:54:21822]{pod_launcher.py:58}错误-尝试创建命名空间pod时出现异常。
[2019-04-0314:54:22103]{base_task_runner.py:101}信息-作业54:子任务价格分析回溯(最后一次调用):
[2019-04-03 14:54:22107]{base_task_runner.py:101}信息-作业54:子任务价格分析文件“/usr/local/lib/aiffair/aiffair/contrib/kubernetes/pod_launcher.py”,第55行,在run_pod_async中
[2019-04-03 14:54:22113]{base_task_runner.py:101}信息-作业54:子任务价格分析resp=self.\u client.create_namespaced_pod(body=req,namespace=pod.namespace)
[2019-04-03 14:54:22116]{base_task_runner.py:101}信息-作业54:子任务价格分析文件“/usr/local/lib/python2.7/dist packages/kubernetes/client/api/core_v1_api.py”,第6115行,在create_namespaced_pod中
[2019-04-03 14:54:22122]{base_task_runner.py:101}信息-作业54:子任务价格分析(数据)=自我。使用http_信息(名称空间、主体、**kwargs)创建名称空间的pod_
[2019-04-03 14:54:22126]{base_task_runner.py:101}信息-作业54:子任务价格分析文件“/usr/local/lib/python2.7/dist packages/kubernetes/client/api/core_v1_api.py”,第6206行,在create_namespaced_pod_with_http_INFO中
[2019-04-0314:54:22129]{base_task_runner.py:101}信息-作业54:子任务价格分析收集格式=收集格式)
[2019-04-03 14:54:22134]{base_task_runner.py:101}信息-作业54:子任务价格分析文件“/usr/local/lib/python2.7/dist packages/kubernetes/client/api_client.py”,第321行,在call_api中
[2019-04-03 14:54:22150]{base_task_runner.py:101}信息-作业54:子任务价格分析(返回)(仅http(数据),收集(格式),预加载(内容),请求(超时)
[2019-04-03 14:54:22155]{base_task_runner.py:101}信息-作业54:子任务价格分析文件“/usr/local/lib/python2.7/dist packages/kubernetes/client/api_client.py”,第155行,在调用api中
[2019-04-0314:54:22159]{base_task_runner.py:101}信息-作业54:子任务价格分析{u request_timeout={u request_timeout]
[2019-04-0314:54:22138]{models.py:1760}错误-(403)
理由:禁止
HTTP响应头:HTTPHeaderDict({'Date':'Wed,03 Apr 2019 14:54:21 GMT','Audit Id':'c027d4cb-5186-498a-a9b5-0e6c4420b816','Content Length':'284','Content Type':'application/json','X-Content-Type-Options':'nosniff'}
HTTP响应正文:{“种类”:“状态”,“apiVersion”:“v1”,“元数据”:{},“状态”:“失败”,“消息”:“pods被禁止:用户\”系统:serviceaccount:composer-1-6-0-airflow-1-10-1-ea0745b4:默认\“无法在命名空间\“默认\”,“原因”:“禁止”,“详细信息”:{“种类”:“pods”},“代码”:403

我收到了Google Groups中Google Composer讨论组的回复。 一个细节:服务帐户默认值:默认值必须与您在e中看到的服务帐户相同
-------------------------------------------------------------------------------
Starting attempt 1 of 
-------------------------------------------------------------------------------

[2019-04-03 14:54:15,611] {models.py:1595} INFO - Executing <Task(KubernetesPodOperator): price-analysis> on 2019-04-03T14:53:59.658367+00:00
[2019-04-03 14:54:15,612] {base_task_runner.py:118} INFO - Running: ['bash', '-c', u'airflow run vat-analysis price-analysis 2019-04-03T14:53:59.658367+00:00 --job_id 54 --raw -sd DAGS_FOLDER/vat_analysis_dag.py --cfg_path /tmp/tmp3RdZOV']
[2019-04-03 14:54:18,375] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:18,374] {settings.py:176} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2019-04-03 14:54:19,652] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,651] {default_celery.py:80} WARNING - You have configured a result_backend of redis://airflow-redis-service.default.svc.cluster.local:6379/0, it is highly recommended to use an alternative result_backend (i.e. a database).
[2019-04-03 14:54:19,659] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,659] {__init__.py:51} INFO - Using executor CeleryExecutor
[2019-04-03 14:54:19,826] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,825] {app.py:51} WARNING - Using default Composer Environment Variables. Overrides have not been applied.
[2019-04-03 14:54:19,842] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,842] {configuration.py:516} INFO - Reading the config from /etc/airflow/airflow.cfg
[2019-04-03 14:54:19,868] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,867] {configuration.py:516} INFO - Reading the config from /etc/airflow/airflow.cfg
[2019-04-03 14:54:20,380] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:20,378] {models.py:271} INFO - Filling up the DagBag from /home/airflow/gcs/dags/vat_analysis_dag.py
[2019-04-03 14:54:21,490] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:21,490] {cli.py:484} INFO - Running <TaskInstance: vat-analysis.price-analysis 2019-04-03T14:53:59.658367+00:00 [running]> on host airflow-worker-5b6d7c75c9-w6995
[2019-04-03 14:54:22,093] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:21,822] {pod_launcher.py:58} ERROR - Exception when attempting to create Namespaced Pod.
[2019-04-03 14:54:22,103] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis Traceback (most recent call last):
[2019-04-03 14:54:22,107] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 55, in run_pod_async
[2019-04-03 14:54:22,113] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
[2019-04-03 14:54:22,116] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6115, in create_namespaced_pod
[2019-04-03 14:54:22,122] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
[2019-04-03 14:54:22,126] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6206, in create_namespaced_pod_with_http_info
[2019-04-03 14:54:22,129] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     collection_formats=collection_formats)
[2019-04-03 14:54:22,134] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 321, in call_api
[2019-04-03 14:54:22,150] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     _return_http_data_only, collection_formats, _preload_content, _request_timeout)
[2019-04-03 14:54:22,155] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 155, in __call_api
[2019-04-03 14:54:22,159] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     _request_timeout=_request_timeout)
[2019-04-03 14:54:22,138] {models.py:1760} ERROR - (403
Reason: Forbidde
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 03 Apr 2019 14:54:21 GMT', 'Audit-Id': 'c027d4cb-5186-498a-a9b5-0e6c4420b816', 'Content-Length': '284', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'}
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:composer-1-6-0-airflow-1-10-1-ea0745b4:default\" cannot create pods in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403