Airflow “U上的气流成功回调函数因”失败;收到一个学期的通知。“终止子流程”;

Airflow “U上的气流成功回调函数因”失败;收到一个学期的通知。“终止子流程”;,airflow,airflow-scheduler,Airflow,Airflow Scheduler,我们的on_success回调函数在完成之前被气流异常终止。但这项任务被标记为成功。 我们的on_success回调函数是一个简单的python函数,没有多线程。我复制了相同的错误,用sleep替换了我们的实际逻辑(10)。 请参考以下代码重新创建问题 如果我把睡眠时间减少到2,那么它就会成功完成。如果我的on_success功能需要2秒以上,则气流会中止该功能 感谢任何人的帮助 气流版本:v1.10.0 代码: 日志: [2019-12-11 18:32:23826]{base_task_ru

我们的on_success回调函数在完成之前被气流异常终止。但这项任务被标记为成功。 我们的on_success回调函数是一个简单的python函数,没有多线程。我复制了相同的错误,用sleep替换了我们的实际逻辑(10)。 请参考以下代码重新创建问题

如果我把睡眠时间减少到2,那么它就会成功完成。如果我的on_success功能需要2秒以上,则气流会中止该功能

感谢任何人的帮助

气流版本:v1.10.0

代码:

日志:

[2019-12-11 18:32:23826]{base_task_runner.py:107}信息-作业75401:Subtask run_this[2019-12-11 18:32:23825]{cli.py:492}信息-在主机xyz上运行
[2019-12-11 18:32:23933]{logging_mixin.py:95}信息-{'dag':,'next_ds':'2019-06-14','prev_ds':'2019-06-13','ds_nodash':'20190614','ts':'2019-06-14T00:00+00:00','etc
[2019-12-1118:32:23933]{logging_mixin.py:95}INFO-2019-06-14
[2019-12-11 18:32:33943]{python_operator.py:96}信息-完成。返回值为:返回的内容将打印在日志中
[2019-12-11 18:32:33995]{logging_mixin.py:95}信息-内部通知成功
[2019-12-11 18:32:37379]{logging_mixin.py:95}INFO-[2019-12-11 18:32:37379]{jobs.py:2680}警告-此实例的状态已从外部设置为成功。服用毒丸。
[2019-12-11 18:32:37417]{helpers.py:240}INFO-发送信号.SIGTERM到GPID105969
[2019-12-11 18:32:37418]{models.py:1610}错误-收到SIGTERM。终止子进程。
[2019-12-11 18:32:37418]{models.py:1689}错误-执行成功回调时失败
[2019-12-11 18:32:37418]{models.py:1690}错误-任务收到SIGTERM信号
回溯(最近一次呼叫最后一次):
文件“/usr/local/lib/python3.6/site packages/afflow/models.py”,第1687行,在原始任务中
task.on_success_回调(上下文)
文件“/home/apps/afflow/dags/folder1/testing_on_success.py”,第25行,在notify_success中
时间。睡眠(10)#实际逻辑执行时间由睡眠到10秒表示
文件“/usr/local/lib/python3.6/site packages/afflow/models.py”,第1612行,在信号处理器中
引发气流异常(“任务收到SIGTERM信号”)
airflow.Exception.AirflowException:任务收到SIGTERM信号
[2019-12-11 18:32:37420]{base_task_runner.py:107}INFO-作业75401:子任务运行\u this/usr/local/lib/python3.6/site packages/aiffort/utils/helpers.py:346:不推荐警告:直接从“aiffort.operators”导入“PythonOperator”已被不推荐。请改为从“airflow.operators.[operator\u module]”导入。气流2.0将完全取消对直接导入的支持。
[2019-12-11 18:32:37420]{base_task_runner.py:107}INFO-Job 75401:Subtask run_this DeprecationWarning)
[2019-12-11 18:32:37550]{helpers.py:230}INFO-进程psutil.Process(pid=105969(已终止))(105969)以退出代码0终止
[2019-12-11 18:32:37552]{logging_mixin.py:95}INFO-[2019-12-11 18:32:37551]{jobs.py:2612}INFO-任务已退出,返回代码为0

我知道这是个老问题,但你解决问题了吗?我和另一个运营商有同样的问题。你能解决这个问题吗?我也面临着这个问题。谢谢
dag = DAG(
    dag_id='testing_on_success',
    default_args=default_args,
    schedule_interval='0 * * * *',
    max_active_runs=1,
    concurrency=5,
    catchup=True)


def notify_success(context):
    print("inside notify_success")
    time.sleep(10) #actual logic execution time is represented by sleeping to 10 seconds 
    print("sleeping done")
    return


def print_context(ds, **kwargs):
    print(kwargs)
    print(ds)
    time.sleep(10)
    return 'Whatever you return gets printed in the logs'


run_this = PythonOperator(
    task_id='run_this',
    provide_context=True,
    python_callable=print_context,
    on_success_callback=notify_success,
    dag=dag,
)
[2019-12-11 18:32:23,826] {base_task_runner.py:107} INFO - Job 75401: Subtask run_this [2019-12-11 18:32:23,825] {cli.py:492} INFO - Running <TaskInstance: testing_on_success.run_this 2019-06-14T00:00:00+00:00 [running]> on host xyz
[2019-12-11 18:32:23,933] {logging_mixin.py:95} INFO - {'dag': <DAG: testing_on_success>, 'next_ds': '2019-06-14', 'prev_ds': '2019-06-13', 'ds_nodash': '20190614', 'ts': '2019-06-14T00:00:00+00:00', 'etc..'}

[2019-12-11 18:32:23,933] {logging_mixin.py:95} INFO - 2019-06-14

[2019-12-11 18:32:33,943] {python_operator.py:96} INFO - Done. Returned value was: Whatever you return gets printed in the logs
[2019-12-11 18:32:33,995] {logging_mixin.py:95} INFO - inside notify_success

[2019-12-11 18:32:37,379] {logging_mixin.py:95} INFO - [2019-12-11 18:32:37,379] {jobs.py:2680} WARNING - State of this instance has been externally set to success. Taking the poison pill.

[2019-12-11 18:32:37,417] {helpers.py:240} INFO - Sending Signals.SIGTERM to GPID 105969
[2019-12-11 18:32:37,418] {models.py:1610} ERROR - Received SIGTERM. Terminating subprocesses.
[2019-12-11 18:32:37,418] {models.py:1689} ERROR - Failed when executing success callback
[2019-12-11 18:32:37,418] {models.py:1690} ERROR - Task received SIGTERM signal
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1687, in _run_raw_task
    task.on_success_callback(context)
  File "/home/apps/airflow/dags/folder1/testing_on_success.py", line 25, in notify_success
    time.sleep(10) #actual logic execution time is represented by sleeping to 10 seconds
  File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1612, in signal_handler
    raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
[2019-12-11 18:32:37,420] {base_task_runner.py:107} INFO - Job 75401: Subtask run_this /usr/local/lib/python3.6/site-packages/airflow/utils/helpers.py:346: DeprecationWarning: Importing 'PythonOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
[2019-12-11 18:32:37,420] {base_task_runner.py:107} INFO - Job 75401: Subtask run_this   DeprecationWarning)
[2019-12-11 18:32:37,550] {helpers.py:230} INFO - Process psutil.Process(pid=105969 (terminated)) (105969) terminated with exit code 0
[2019-12-11 18:32:37,552] {logging_mixin.py:95} INFO - [2019-12-11 18:32:37,551] {jobs.py:2612} INFO - Task exited with return code 0