Python 气流重试显示错误号码请尝试4次中的5次
我发现气流重试不一致,或者我遗漏了什么。基本上,我在默认参数中将Python 气流重试显示错误号码请尝试4次中的5次,python,airflow,Python,Airflow,我发现气流重试不一致,或者我遗漏了什么。基本上,我在默认参数中将retries设置为3 default_args = { 'owner': 'airflow', 'depends_on_past': False, 'start_date': datetime(2015, 6, 1), 'email': ['airflow@example.com'], 'email_on_failure': True, 'email_on_retry': True,
retries
设置为3
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2015, 6, 1),
'email': ['airflow@example.com'],
'email_on_failure': True,
'email_on_retry': True,
'retries': 3
}
并在PythonOperators
中设置为3
t1 = PythonOperator(
task_id='task_1',
python_callable=task_1_process,
retries=3,
dag=dag)
t2 = SparkSubmitOperator(
task_id="spark_submit_task",
conn_id="spark_default",
java_class="${JAVA_CLASS}",
application='/path/to/myjar-0.1.jar',
application_args=["${ARG1}"],
conf=SPARK_CONF,
dag=dag)
t3 = PythonOperator(
task_id='task_3',
python_callable=task_3_process,
retries=3,
dag=dag)
我假设这意味着如果第一次运行失败,作业将重试3次,这意味着,我应该:
Try 1 out of 4 [up_for_retry]
Try 2 out of 4 [up_for_retry]
Try 3 out of 4 [up_for_retry]
Try 4 out of 4 [failed]
但在电子邮件中,我收到:
Airflow alert: <TaskInstance: MyJob.spark_submit_task 2018-12-06T00:00:00+00:00 [up_for_retry]>
Try 2 out of 4
Try 3 out of 4
Try 4 out of 4
Airflow alert: <TaskInstance: Airflow alert: <TaskInstance: MyJob.spark_submit_task 2018-12-06T00:00:00+00:00 [failed]>
Try 5 out of 4
气流警报:
试试四分之二
四选三
四选四
气流警报:我以前在清除任务以重新运行时看到过这些状态。您是否已验证具有实例访问权限的任何人尚未清除并重新运行任务?@andscoop我可以确认任务未清除并重新运行