Airflow 具有不同计划间隔的气流DAG相关性

Airflow 具有不同计划间隔的气流DAG相关性,airflow,airflow-scheduler,directed-acyclic-graphs,Airflow,Airflow Scheduler,Directed Acyclic Graphs,我需要与具有不同计划间隔的气流DAG保持依赖关系 示例 DAG 1-任务1、任务2(周一至周六上午9点运行) DAG 2-任务1、任务2(每周周五运行) DAG 3-任务1(在DAG 1或DAG 2之后立即触发,每日周一至周六无特定时间运行) 要求: with DAG('DAG 1', schedule_interval='0 9 * * 1-6' ) as dag: TASK1 = BashOperator(task_id='TASK1',

我需要与具有不同计划间隔的气流DAG保持依赖关系

示例

DAG 1-任务1、任务2(周一至周六上午9点运行)

DAG 2-任务1、任务2(每周周五运行)

DAG 3-任务1(在DAG 1或DAG 2之后立即触发,每日周一至周六无特定时间运行)

要求:

with DAG('DAG 1',
     schedule_interval='0 9 * * 1-6'
     ) as dag:

TASK1 = BashOperator(task_id='TASK1',
                bash_command='sample')

TASK2 = BashOperator(task_id='TASK2',
                bash_command='sample')

 triggerDAG2TASK2 = TriggerDagRunOperator(
        task_id="TASK2",
        trigger_dag_id="DAG 2"
    )

 triggerDAG3TASK1 = TriggerDagRunOperator(
        task_id="TASK1",
        trigger_dag_id="DAG 3"
    )

TASK1  >> TASK2 >> *triggerDAG2TASK2 OR triggerDAG3TASK1*  
with DAG('DAG 2',
     schedule_interval='0 9 * * 5'
     ) as dag:

TASK1 = BashOperator(task_id='TASK1',
                bash_command='sample')

TASK2 = BashOperator(task_id='TASK2',
                bash_command='sample')

 triggerDAG3TASK1 = TriggerDagRunOperator(
        task_id="TASK1",
        trigger_dag_id="DAG 3"
    )

TASK1  >> TASK2 >> triggerDAG3TASK1
with DAG('DAG 3',
     schedule_interval='@daily' ## No specific time, immediately after DAG 1 OR DAG 2
     ) as dag:

TASK1 = BashOperator(task_id='TASK1',
                bash_command='sample')

TASK2 = BashOperator(task_id='TASK2',
                bash_command='sample')

TASK1 
  • DAG 1任务1和DAG 2任务1可以并行运行
  • 只有在DAG 1任务2成功时,DAG 2任务2才应在星期五每周触发
  • DAG 3仅在以下情况下运行:
    • DAG 1从Mon-Sat和

    • DAG 1和DAG 2在周五成功

    以下样本:

    DAG 1:

    with DAG('DAG 1',
         schedule_interval='0 9 * * 1-6'
         ) as dag:
    
    TASK1 = BashOperator(task_id='TASK1',
                    bash_command='sample')
    
    TASK2 = BashOperator(task_id='TASK2',
                    bash_command='sample')
    
     triggerDAG2TASK2 = TriggerDagRunOperator(
            task_id="TASK2",
            trigger_dag_id="DAG 2"
        )
    
     triggerDAG3TASK1 = TriggerDagRunOperator(
            task_id="TASK1",
            trigger_dag_id="DAG 3"
        )
    
    TASK1  >> TASK2 >> *triggerDAG2TASK2 OR triggerDAG3TASK1*  
    
    with DAG('DAG 2',
         schedule_interval='0 9 * * 5'
         ) as dag:
    
    TASK1 = BashOperator(task_id='TASK1',
                    bash_command='sample')
    
    TASK2 = BashOperator(task_id='TASK2',
                    bash_command='sample')
    
     triggerDAG3TASK1 = TriggerDagRunOperator(
            task_id="TASK1",
            trigger_dag_id="DAG 3"
        )
    
    TASK1  >> TASK2 >> triggerDAG3TASK1
    
    with DAG('DAG 3',
         schedule_interval='@daily' ## No specific time, immediately after DAG 1 OR DAG 2
         ) as dag:
    
    TASK1 = BashOperator(task_id='TASK1',
                    bash_command='sample')
    
    TASK2 = BashOperator(task_id='TASK2',
                    bash_command='sample')
    
    TASK1 
    
    DAG 2:

    with DAG('DAG 1',
         schedule_interval='0 9 * * 1-6'
         ) as dag:
    
    TASK1 = BashOperator(task_id='TASK1',
                    bash_command='sample')
    
    TASK2 = BashOperator(task_id='TASK2',
                    bash_command='sample')
    
     triggerDAG2TASK2 = TriggerDagRunOperator(
            task_id="TASK2",
            trigger_dag_id="DAG 2"
        )
    
     triggerDAG3TASK1 = TriggerDagRunOperator(
            task_id="TASK1",
            trigger_dag_id="DAG 3"
        )
    
    TASK1  >> TASK2 >> *triggerDAG2TASK2 OR triggerDAG3TASK1*  
    
    with DAG('DAG 2',
         schedule_interval='0 9 * * 5'
         ) as dag:
    
    TASK1 = BashOperator(task_id='TASK1',
                    bash_command='sample')
    
    TASK2 = BashOperator(task_id='TASK2',
                    bash_command='sample')
    
     triggerDAG3TASK1 = TriggerDagRunOperator(
            task_id="TASK1",
            trigger_dag_id="DAG 3"
        )
    
    TASK1  >> TASK2 >> triggerDAG3TASK1
    
    with DAG('DAG 3',
         schedule_interval='@daily' ## No specific time, immediately after DAG 1 OR DAG 2
         ) as dag:
    
    TASK1 = BashOperator(task_id='TASK1',
                    bash_command='sample')
    
    TASK2 = BashOperator(task_id='TASK2',
                    bash_command='sample')
    
    TASK1 
    
    DAG 3:

    with DAG('DAG 1',
         schedule_interval='0 9 * * 1-6'
         ) as dag:
    
    TASK1 = BashOperator(task_id='TASK1',
                    bash_command='sample')
    
    TASK2 = BashOperator(task_id='TASK2',
                    bash_command='sample')
    
     triggerDAG2TASK2 = TriggerDagRunOperator(
            task_id="TASK2",
            trigger_dag_id="DAG 2"
        )
    
     triggerDAG3TASK1 = TriggerDagRunOperator(
            task_id="TASK1",
            trigger_dag_id="DAG 3"
        )
    
    TASK1  >> TASK2 >> *triggerDAG2TASK2 OR triggerDAG3TASK1*  
    
    with DAG('DAG 2',
         schedule_interval='0 9 * * 5'
         ) as dag:
    
    TASK1 = BashOperator(task_id='TASK1',
                    bash_command='sample')
    
    TASK2 = BashOperator(task_id='TASK2',
                    bash_command='sample')
    
     triggerDAG3TASK1 = TriggerDagRunOperator(
            task_id="TASK1",
            trigger_dag_id="DAG 3"
        )
    
    TASK1  >> TASK2 >> triggerDAG3TASK1
    
    with DAG('DAG 3',
         schedule_interval='@daily' ## No specific time, immediately after DAG 1 OR DAG 2
         ) as dag:
    
    TASK1 = BashOperator(task_id='TASK1',
                    bash_command='sample')
    
    TASK2 = BashOperator(task_id='TASK2',
                    bash_command='sample')
    
    TASK1 
    
    如何以不同的计划间隔保持这些依赖关系

    我在某处读到,不建议使用externaltask传感器,因为它只检查当天的运行情况

    因为这是一个示例,所以我有多个DAG/任务,并且我正在尝试分离这些内容。我还可以说case DAG 1每天晚上9点运行,DAG 2每周晚上10点运行,DAG 3第二天早上9点运行。还需要能够处理不同日期之间依赖关系的解决方案