Airflow 具有不同计划间隔的气流DAG相关性
我需要与具有不同计划间隔的气流DAG保持依赖关系 示例 DAG 1-任务1、任务2(周一至周六上午9点运行) DAG 2-任务1、任务2(每周周五运行) DAG 3-任务1(在DAG 1或DAG 2之后立即触发,每日周一至周六无特定时间运行) 要求:Airflow 具有不同计划间隔的气流DAG相关性,airflow,airflow-scheduler,directed-acyclic-graphs,Airflow,Airflow Scheduler,Directed Acyclic Graphs,我需要与具有不同计划间隔的气流DAG保持依赖关系 示例 DAG 1-任务1、任务2(周一至周六上午9点运行) DAG 2-任务1、任务2(每周周五运行) DAG 3-任务1(在DAG 1或DAG 2之后立即触发,每日周一至周六无特定时间运行) 要求: with DAG('DAG 1', schedule_interval='0 9 * * 1-6' ) as dag: TASK1 = BashOperator(task_id='TASK1',
with DAG('DAG 1',
schedule_interval='0 9 * * 1-6'
) as dag:
TASK1 = BashOperator(task_id='TASK1',
bash_command='sample')
TASK2 = BashOperator(task_id='TASK2',
bash_command='sample')
triggerDAG2TASK2 = TriggerDagRunOperator(
task_id="TASK2",
trigger_dag_id="DAG 2"
)
triggerDAG3TASK1 = TriggerDagRunOperator(
task_id="TASK1",
trigger_dag_id="DAG 3"
)
TASK1 >> TASK2 >> *triggerDAG2TASK2 OR triggerDAG3TASK1*
with DAG('DAG 2',
schedule_interval='0 9 * * 5'
) as dag:
TASK1 = BashOperator(task_id='TASK1',
bash_command='sample')
TASK2 = BashOperator(task_id='TASK2',
bash_command='sample')
triggerDAG3TASK1 = TriggerDagRunOperator(
task_id="TASK1",
trigger_dag_id="DAG 3"
)
TASK1 >> TASK2 >> triggerDAG3TASK1
with DAG('DAG 3',
schedule_interval='@daily' ## No specific time, immediately after DAG 1 OR DAG 2
) as dag:
TASK1 = BashOperator(task_id='TASK1',
bash_command='sample')
TASK2 = BashOperator(task_id='TASK2',
bash_command='sample')
TASK1
- DAG 1从Mon-Sat和
- DAG 1和DAG 2在周五成功
with DAG('DAG 1',
schedule_interval='0 9 * * 1-6'
) as dag:
TASK1 = BashOperator(task_id='TASK1',
bash_command='sample')
TASK2 = BashOperator(task_id='TASK2',
bash_command='sample')
triggerDAG2TASK2 = TriggerDagRunOperator(
task_id="TASK2",
trigger_dag_id="DAG 2"
)
triggerDAG3TASK1 = TriggerDagRunOperator(
task_id="TASK1",
trigger_dag_id="DAG 3"
)
TASK1 >> TASK2 >> *triggerDAG2TASK2 OR triggerDAG3TASK1*
with DAG('DAG 2',
schedule_interval='0 9 * * 5'
) as dag:
TASK1 = BashOperator(task_id='TASK1',
bash_command='sample')
TASK2 = BashOperator(task_id='TASK2',
bash_command='sample')
triggerDAG3TASK1 = TriggerDagRunOperator(
task_id="TASK1",
trigger_dag_id="DAG 3"
)
TASK1 >> TASK2 >> triggerDAG3TASK1
with DAG('DAG 3',
schedule_interval='@daily' ## No specific time, immediately after DAG 1 OR DAG 2
) as dag:
TASK1 = BashOperator(task_id='TASK1',
bash_command='sample')
TASK2 = BashOperator(task_id='TASK2',
bash_command='sample')
TASK1
DAG 2:
with DAG('DAG 1',
schedule_interval='0 9 * * 1-6'
) as dag:
TASK1 = BashOperator(task_id='TASK1',
bash_command='sample')
TASK2 = BashOperator(task_id='TASK2',
bash_command='sample')
triggerDAG2TASK2 = TriggerDagRunOperator(
task_id="TASK2",
trigger_dag_id="DAG 2"
)
triggerDAG3TASK1 = TriggerDagRunOperator(
task_id="TASK1",
trigger_dag_id="DAG 3"
)
TASK1 >> TASK2 >> *triggerDAG2TASK2 OR triggerDAG3TASK1*
with DAG('DAG 2',
schedule_interval='0 9 * * 5'
) as dag:
TASK1 = BashOperator(task_id='TASK1',
bash_command='sample')
TASK2 = BashOperator(task_id='TASK2',
bash_command='sample')
triggerDAG3TASK1 = TriggerDagRunOperator(
task_id="TASK1",
trigger_dag_id="DAG 3"
)
TASK1 >> TASK2 >> triggerDAG3TASK1
with DAG('DAG 3',
schedule_interval='@daily' ## No specific time, immediately after DAG 1 OR DAG 2
) as dag:
TASK1 = BashOperator(task_id='TASK1',
bash_command='sample')
TASK2 = BashOperator(task_id='TASK2',
bash_command='sample')
TASK1
DAG 3:
with DAG('DAG 1',
schedule_interval='0 9 * * 1-6'
) as dag:
TASK1 = BashOperator(task_id='TASK1',
bash_command='sample')
TASK2 = BashOperator(task_id='TASK2',
bash_command='sample')
triggerDAG2TASK2 = TriggerDagRunOperator(
task_id="TASK2",
trigger_dag_id="DAG 2"
)
triggerDAG3TASK1 = TriggerDagRunOperator(
task_id="TASK1",
trigger_dag_id="DAG 3"
)
TASK1 >> TASK2 >> *triggerDAG2TASK2 OR triggerDAG3TASK1*
with DAG('DAG 2',
schedule_interval='0 9 * * 5'
) as dag:
TASK1 = BashOperator(task_id='TASK1',
bash_command='sample')
TASK2 = BashOperator(task_id='TASK2',
bash_command='sample')
triggerDAG3TASK1 = TriggerDagRunOperator(
task_id="TASK1",
trigger_dag_id="DAG 3"
)
TASK1 >> TASK2 >> triggerDAG3TASK1
with DAG('DAG 3',
schedule_interval='@daily' ## No specific time, immediately after DAG 1 OR DAG 2
) as dag:
TASK1 = BashOperator(task_id='TASK1',
bash_command='sample')
TASK2 = BashOperator(task_id='TASK2',
bash_command='sample')
TASK1
如何以不同的计划间隔保持这些依赖关系
我在某处读到,不建议使用externaltask传感器,因为它只检查当天的运行情况
因为这是一个示例,所以我有多个DAG/任务,并且我正在尝试分离这些内容。我还可以说case DAG 1每天晚上9点运行,DAG 2每周晚上10点运行,DAG 3第二天早上9点运行。还需要能够处理不同日期之间依赖关系的解决方案