我在KubernetesPodOperator的DAG设置中做错了什么
我在这里发现了以下气流DAG: 在我尝试添加任何自定义内容之前。。。试图按原样运行它。但是,在我的环境中,代码似乎超时 根据文档,我试图将我在KubernetesPodOperator的DAG设置中做错了什么,kubernetes,airflow,Kubernetes,Airflow,我在这里发现了以下气流DAG: 在我尝试添加任何自定义内容之前。。。试图按原样运行它。但是,在我的环境中,代码似乎超时 根据文档,我试图将启动\u超时\u秒设置为类似10m的荒谬值。。。但仍然收到文档中描述的超时消息: [2019-01-04 11:13:33,360] {pod_launcher.py:112} INFO - Event: fail-7dd76b92 had an event of type Pending Traceback (most recent call last):
启动\u超时\u秒
设置为类似10m的荒谬值。。。但仍然收到文档中描述的超时消息:
[2019-01-04 11:13:33,360] {pod_launcher.py:112} INFO - Event: fail-7dd76b92 had an event of type Pending
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/usr/local/lib/airflow/airflow/bin/airflow", line 27, in <module>
args.func(args)
File "/usr/local/lib/airflow/airflow/bin/cli.py", line 392, in run
pool=args.pool,
File "/usr/local/lib/airflow/airflow/utils/db.py", line 50, in wrapper
result = func(*args, **kwargs)
File "/usr/local/lib/airflow/airflow/models.py", line 1492, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/airflow/airflow/contrib/operators/kubernetes_pod_operator.py", line 123, in execute
raise AirflowException('Pod Launching failed: {error}'.format(error=ex))
airflow.exceptions.AirflowException: Pod Launching failed: Pod took too long to start
[2019-01-04 11:13:33360]{pod_launcher.py:112}INFO-事件:fail-7dd76b92有一个挂起类型的事件
回溯(最近一次呼叫最后一次):
文件“/usr/local/bin/afflow”,第6行,在
exec(编译(打开(\uuuuu文件\uuuuu).read(),\uuuuu文件\uuuuuuuu,'exec'))
文件“/usr/local/lib/afflow/afflow/bin/afflow”,第27行,in
args.func(args)
文件“/usr/local/lib/afflow/afflow/bin/cli.py”,第392行,运行中
pool=args.pool,
文件“/usr/local/lib/afflow/afflow/utils/db.py”,第50行,在包装器中
结果=函数(*args,**kwargs)
文件“/usr/local/lib/aiffair/aiffair/models.py”,第1492行,在原始任务中
结果=任务\复制.执行(上下文=上下文)
文件“/usr/local/lib/aiffair/aiffair/contrib/operators/kubernetes_pod_operator.py”,执行中的第123行
raise AirflowException('Pod启动失败:{error}'。格式(error=ex))
airflow.Exception.AirflowException:吊舱启动失败:吊舱启动时间过长
任何输入都将不胜感激。由于此代码未使用完全限定的图像,这意味着Airflow正在从中提取图像,
“Python:3.6”
和“ubuntu:1604”
不适用于或在中的docker图像名称
“Python”命令也不应该大写
具有有效docker图像名称的工作代码为:
from airflow import DAG
from datetime import datetime, timedelta
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from airflow.operators.dummy_operator import DummyOperator
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.utcnow(),
'email': ['airflow@example.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5)
}
dag = DAG(
'kubernetes_sample', default_args=default_args, schedule_interval=timedelta(minutes=10))
start = DummyOperator(task_id='run_this_first', dag=dag)
passing = KubernetesPodOperator(namespace='default',
image="python:3.6-stretch",
cmds=["python","-c"],
arguments=["print('hello world')"],
labels={"foo": "bar"},
name="passing-test",
task_id="passing-task",
get_logs=True,
dag=dag
)
failing = KubernetesPodOperator(namespace='default',
image="ubuntu:16.04",
cmds=["python","-c"],
arguments=["print('hello world')"],
labels={"foo": "bar"},
name="fail",
task_id="failing-task",
get_logs=True,
dag=dag
)
passing.set_upstream(start)
failing.set_upstream(start)
是的,我是处理你案件的GCP支持人员;)我在这里也作了完整的回答(我知道,泰;)遗憾的是,没有日志记录可以帮助调试此类错误<代码>Pod启动时间太长帮不了什么忙…@LondonRob-同意。您知道如何在使用KubernetesPodOperator时获取更详细的日志吗?
from airflow import DAG
from datetime import datetime, timedelta
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from airflow.operators.dummy_operator import DummyOperator
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.utcnow(),
'email': ['airflow@example.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5)
}
dag = DAG(
'kubernetes_sample', default_args=default_args, schedule_interval=timedelta(minutes=10))
start = DummyOperator(task_id='run_this_first', dag=dag)
passing = KubernetesPodOperator(namespace='default',
image="python:3.6-stretch",
cmds=["python","-c"],
arguments=["print('hello world')"],
labels={"foo": "bar"},
name="passing-test",
task_id="passing-task",
get_logs=True,
dag=dag
)
failing = KubernetesPodOperator(namespace='default',
image="ubuntu:16.04",
cmds=["python","-c"],
arguments=["print('hello world')"],
labels={"foo": "bar"},
name="fail",
task_id="failing-task",
get_logs=True,
dag=dag
)
passing.set_upstream(start)
failing.set_upstream(start)