Python 3.x 利用ExternalTaskSensor的DAG持续运行

我正在使用ExternalTaskSensor在完成第一个DAG（DAG 1）后执行第二个DAG（DAG 2）

DAG 1中的最终任务是（传感器等待的任务）定义为：

end_of_global_dim = BashOperator(
        task_id='Global_dim_end',
        bash_command="echo 'Global dimensions DAG is finished'",
        dag=dag
    )

DAG 1的定义如下-

yesterday = datetime.datetime.combine(
    datetime.datetime.today() - datetime.timedelta(1),
    datetime.datetime.min.time()
)

default_dag_args = {
        'start_date': yesterday,
        # Email whenever an Operator in the DAG fails.
        'email': 'xyz@abc.com',
        'email_on_failure': True,
        'email_on_retry': False,
        'retries': 5,
        'retry_delay': timedelta(minutes=1),
        'project_id': 'dummy',
    }
    
    
    with models.DAG('Global_dimensions_DAG', default_args=default_dag_args, catchup=False,
                    schedule_interval=timedelta(days=1)) as dag:
     ...

DAG 2定义（默认参数与上述相同）：

外部任务传感器定义：

Global_dimensions_dag_sensor = ExternalTaskSensor(
        task_id='GLOBAL_DIM_EXTERNAL_DAG',
        external_dag_id='Global_dimensions_DAG',
        external_task_id='Global_dim_end',
        timeout=1000,
        #execution_delta = timedelta(days=1),
        dag=te_dag
    )

我的问题是外部任务传感器根本不工作。它一直在运行，但日志不会得到更新，直到我杀死DAG并重新启动DAG。只有在那时，我才看到传感器不断拨动“全局变暗结束”任务，但最终超时

[2021-04-22 16:38:29,847] {taskinstance.py:902} INFO - Executing <Task(ExternalTaskSensor): GLOBAL_DIM_EXTERNAL_DAG> on 2021-04-21T00:00:00+00:00
[2021-04-22 16:38:29,853] {standard_task_runner.py:54} INFO - Started process 168844 to run task
[2021-04-22 16:38:29,901] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'Term_enrollment_DAG', 'GLOBAL_DIM_EXTERNAL_DAG', '2021-04-21T00:00:00+00:00', '--job_id', '7143', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/Term_enrollment_DAG.py', '--cfg_path', '/tmp/tmpgyn8zzw6']
[2021-04-22 16:38:29,903] {standard_task_runner.py:78} INFO - Job 7143: Subtask GLOBAL_DIM_EXTERNAL_DAG
[2021-04-22 16:38:30,545] {logging_mixin.py:112} INFO - Running <TaskInstance: Term_enrollment_DAG.GLOBAL_DIM_EXTERNAL_DAG 2021-04-21T00:00:00+00:00 [running]> on host airflow-worker-65949b9449-h766l
[2021-04-22 16:38:30,693] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:39:30,777] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:40:30,855] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:41:30,928] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:42:30,970] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:43:31,142] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:44:31,259] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:45:31,423] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:46:31,540] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:47:31,646] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:48:31,751] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:49:31,840] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:50:31,945] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:51:32,048] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:52:32,177] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:53:32,266] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:54:32,372] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:55:32,517] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:55:32,632] {taskinstance.py:1152} ERROR - Snap. Time is OUT.
Traceback (most recent call last)
  File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 985, in _run_raw_tas
    result = task_copy.execute(context=context
  File "/usr/local/lib/airflow/airflow/sensors/base_sensor_operator.py", line 116, in execut
    raise AirflowSensorTimeout('Snap. Time is OUT.'
airflow.exceptions.AirflowSensorTimeout: Snap. Time is OUT
[2021-04-22 16:55:32,635] {taskinstance.py:1196} INFO - Marking task as UP_FOR_RETRY. dag_id=Term_enrollment_DAG, task_id=GLOBAL_DIM_EXTERNAL_DAG, execution_date=20210421T000000, start_date=20210422T163829, end_date=20210422T165532
[2021-04-22 16:55:36,800] {local_task_job.py:102} INFO - Task exited with return code 1

[2021-04-22 16:38:29847]{taskinstance.py:902}INFO-在2021-04-21T00:00:00+00:00执行
[2021-04-22 16:38:29853]{standard_task_runner.py:54}INFO-已启动进程168844以运行任务
[2021-04-22 16:38:29901]{standard_task_runner.py:77}运行信息：['aiffair'、'run'、'Term_registration_DAG'、'GLOBAL_DIM_EXTERNAL_DAG'、'2021-04-21T00:00:00+00:00'、'--job_id'、'7143'、'--pool'、'default_pool'、'default_pool'、'-raw'、'-sd'、'-sd'、'DAGS_文件夹/Term_registration_DAG.py'、'cfg'、'cfg路径'、'tmp/tmp/
[2021-04-22 16:38:29903]{standard_task_runner.py:78}信息-作业7143：子任务全局图外部图
[2021-04-22 16:38:30545]{logging_mixin.py:112}信息-在主机上运行airflow-worker-65949b9449-h766l
[2021-04-22 16:38:30693]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:39:30777]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:40:30855]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:41:30928]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:42:30970]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:43:31142]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:44:31259]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:45:31423]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:46:31540]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:47:31646]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:48:31751]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:49:31840]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:50:31945]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:51:32048]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:52:32177]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:53:32266]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:54:32372]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:55:32517]{external_task_sensor.py:117}信息-搜索全局尺寸_DAG.Global_dim_结束于2021-04-21T00:00:00+00:00。。。
[2021-04-22 16:55:32632]{taskinstance.py:1152}错误-快照。时间到了。
回溯（最近一次呼叫最后一次）
文件“/usr/local/lib/aiffair/aiffair/models/taskinstance.py”，第985行，在原始状态下运行
结果=任务\复制.执行（上下文=上下文
文件“/usr/local/lib/aiffair/aiffair/sensors/base\u sensor\u operator.py”，第116行，执行部分
升起气流传感器超时（'捕捉。超时。'
airflow.exceptions.AirflowSensorTimeout:捕捉。超时
[2021-04-22 16:55:32635]{taskinstance.py:1196}信息-将任务标记为“准备重试”。dag_id=Term_registration_dag，task_id=GLOBAL_DIM_EXTERNAL__dag，execution_date=20210421T000000，start_date=20210422T163829，end_date=20210422T165532
[2021-04-22 16:55:36800]{local_task_job.py:102}INFO-任务已退出，返回代码为1

我正在Google Cloud Composer（Composer-1.16.0-Airflow-1.10.14）上使用气流

注意：两个DAG都存在于单独的.py文件中。DAG 1有许多任务，需要35分钟才能完成。因此，当我将传感器的超时设置为2400秒（40分钟）时，传感器也会超时

[2021-04-22 16:38:29,847] {taskinstance.py:902} INFO - Executing <Task(ExternalTaskSensor): GLOBAL_DIM_EXTERNAL_DAG> on 2021-04-21T00:00:00+00:00
[2021-04-22 16:38:29,853] {standard_task_runner.py:54} INFO - Started process 168844 to run task
[2021-04-22 16:38:29,901] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'Term_enrollment_DAG', 'GLOBAL_DIM_EXTERNAL_DAG', '2021-04-21T00:00:00+00:00', '--job_id', '7143', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/Term_enrollment_DAG.py', '--cfg_path', '/tmp/tmpgyn8zzw6']
[2021-04-22 16:38:29,903] {standard_task_runner.py:78} INFO - Job 7143: Subtask GLOBAL_DIM_EXTERNAL_DAG
[2021-04-22 16:38:30,545] {logging_mixin.py:112} INFO - Running <TaskInstance: Term_enrollment_DAG.GLOBAL_DIM_EXTERNAL_DAG 2021-04-21T00:00:00+00:00 [running]> on host airflow-worker-65949b9449-h766l
[2021-04-22 16:38:30,693] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:39:30,777] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:40:30,855] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:41:30,928] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:42:30,970] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:43:31,142] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:44:31,259] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:45:31,423] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:46:31,540] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:47:31,646] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:48:31,751] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:49:31,840] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:50:31,945] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:51:32,048] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:52:32,177] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:53:32,266] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:54:32,372] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:55:32,517] {external_task_sensor.py:117} INFO - Poking for Global_dimensions_DAG.Global_dim_end on 2021-04-21T00:00:00+00:00 ... 
[2021-04-22 16:55:32,632] {taskinstance.py:1152} ERROR - Snap. Time is OUT.
Traceback (most recent call last)
  File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 985, in _run_raw_tas
    result = task_copy.execute(context=context
  File "/usr/local/lib/airflow/airflow/sensors/base_sensor_operator.py", line 116, in execut
    raise AirflowSensorTimeout('Snap. Time is OUT.'
airflow.exceptions.AirflowSensorTimeout: Snap. Time is OUT
[2021-04-22 16:55:32,635] {taskinstance.py:1196} INFO - Marking task as UP_FOR_RETRY. dag_id=Term_enrollment_DAG, task_id=GLOBAL_DIM_EXTERNAL_DAG, execution_date=20210421T000000, start_date=20210422T163829, end_date=20210422T165532
[2021-04-22 16:55:36,800] {local_task_job.py:102} INFO - Task exited with return code 1