Python 没有日期的例子

我不习惯使用。我正在尝试运行dag，不想做任何计划

我希望使用命令行参数运行管道，并覆盖所有当前输出。我没有开始日期，没有计划，没有计时，也没有重试逻辑，我只想按顺序运行一组函数来开始

文档中始终包含日期

airflow test tutorial print_date 2015-06-01

我希望运行dag，以便它执行所有函数并忽略以前的任何运行。如何从dag中删除所有日期和日期逻辑

我有教程dag文件的修改版本：

"""
Code that goes along with the Airflow tutorial located at:
https://github.com/airbnb/airflow/blob/master/airflow/example_dags/tutorial.py
"""
import os
import cPickle
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime


default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2015, 6, 1),
    'email': ['airflow@airflow.com'],
    'email_on_failure': False,
    'email_on_retry': False,
    'schedule_interval': '@once'
}

dag = DAG('tutorial_me', default_args=default_args)

def save_file(filenm):
    with open(filenm, 'wb') as pickle_file:
        cPickle.dump(['1','2',3], pickle_file)

def delete_file(filenm):
    print "************ THIS IS WHERE STDOUT GOES"
    if os.path.exists(filenm):
        os.path.remove(filenm)


# t1, t2 and t3 are examples of tasks created by instantiating operators
t1 = PythonOperator(
    task_id='save_file',
    python_callable=save_file,
    op_kwargs=dict(filenm='__myparamfile__.txt'),
    dag=dag)

t2 = PythonOperator(
    task_id='remove_file',
    python_callable=delete_file,
    op_kwargs=dict(filenm='__myparamfile__.txt'),
    dag=dag)

t1.set_upstream(t2)

我第一次运行它时：

airflow run tutorial_me remove_file 2015-01-04

它工作并打印

print“*********这是标准输出的地方”

行。我第二次运行它时，它没有

第二次运行后，日志文件看起来像这样

cat 2015-01-04T00\:00\:00
[2016-12-10 11:27:47,158] {models.py:154} INFO - Filling up the DagBag from /Users/user_01/airflow/dags
[2016-12-10 11:27:47,214] {models.py:1750} WARNING - schedule_interval is used for <Task(PythonOperator): save_file>, though it has been deprecated as a task parameter, you need to specify it as a DAG parameter instead
[2016-12-10 11:27:47,214] {models.py:1750} WARNING - schedule_interval is used for <Task(PythonOperator): remove_file>, though it has been deprecated as a task parameter, you need to specify it as a DAG parameter instead
[2016-12-10 11:27:47,227] {base_executor.py:36} INFO - Adding to queue: airflow run tutorial_me remove_file 2015-01-04T00:00:00 --local -sd DAGS_FOLDER/tutorial_01.py
[2016-12-10 11:27:47,234] {sequential_executor.py:26} INFO - Executing command: airflow run tutorial_me remove_file 2015-01-04T00:00:00 --local -sd DAGS_FOLDER/tutorial_01.py
[2016-12-10 11:27:48,050] {models.py:154} INFO - Filling up the DagBag from /Users/user_01/airflow/dags/tutorial_01.py
[2016-12-10 11:27:48,101] {models.py:1750} WARNING - schedule_interval is used for <Task(PythonOperator): save_file>, though it has been deprecated as a task parameter, you need to specify it as a DAG parameter instead
[2016-12-10 11:27:48,102] {models.py:1750} WARNING - schedule_interval is used for <Task(PythonOperator): remove_file>, though it has been deprecated as a task parameter, you need to specify it as a DAG parameter instead
[2016-12-10 11:27:48,942] {models.py:154} INFO - Filling up the DagBag from /Users/user_01/airflow/dags/tutorial_01.py
[2016-12-10 11:27:48,998] {models.py:1750} WARNING - schedule_interval is used for <Task(PythonOperator): save_file>, though it has been deprecated as a task parameter, you need to specify it as a DAG parameter instead
[2016-12-10 11:27:48,998] {models.py:1750} WARNING - schedule_interval is used for <Task(PythonOperator): remove_file>, though it has been deprecated as a task parameter, you need to specify it as a DAG parameter instead
[2016-12-10 11:27:49,020] {models.py:1196} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------

[2016-12-10 11:27:49,046] {models.py:1219} INFO - Executing <Task(PythonOperator): remove_file> on 2015-01-04 00:00:00
[2016-12-10 11:27:49,054] {python_operator.py:67} INFO - Done. Returned value was: None
[2016-12-10 11:27:55,168] {models.py:154} INFO - Filling up the DagBag from /Users/user_01/airflow/dags
[2016-12-10 11:27:55,219] {models.py:1750} WARNING - schedule_interval is used for <Task(PythonOperator): save_file>, though it has been deprecated as a task parameter, you need to specify it as a DAG parameter instead
[2016-12-10 11:27:55,220] {models.py:1750} WARNING - schedule_interval is used for <Task(PythonOperator): remove_file>, though it has been deprecated as a task parameter, you need to specify it as a DAG parameter instead
[2016-12-10 11:27:55,231] {base_executor.py:36} INFO - Adding to queue: airflow run tutorial_me remove_file 2015-01-04T00:00:00 --local -sd DAGS_FOLDER/tutorial_01.py
[2016-12-10 11:27:55,236] {sequential_executor.py:26} INFO - Executing command: airflow run tutorial_me remove_file 2015-01-04T00:00:00 --local -sd DAGS_FOLDER/tutorial_01.py
[2016-12-10 11:27:56,030] {models.py:154} INFO - Filling up the DagBag from /Users/user_01/airflow/dags/tutorial_01.py
[2016-12-10 11:27:56,082] {models.py:1750} WARNING - schedule_interval is used for <Task(PythonOperator): save_file>, though it has been deprecated as a task parameter, you need to specify it as a DAG parameter instead
[2016-12-10 11:27:56,082] {models.py:1750} WARNING - schedule_interval is used for <Task(PythonOperator): remove_file>, though it has been deprecated as a task parameter, you need to specify it as a DAG parameter instead
[2016-12-10 11:27:56,899] {models.py:154} INFO - Filling up the DagBag from /Users/user_01/airflow/dags/tutorial_01.py
[2016-12-10 11:27:56,950] {models.py:1750} WARNING - schedule_interval is used for <Task(PythonOperator): save_file>, though it has been deprecated as a task parameter, you need to specify it as a DAG parameter instead
[2016-12-10 11:27:56,951] {models.py:1750} WARNING - schedule_interval is used for <Task(PythonOperator): remove_file>, though it has been deprecated as a task parameter, you need to specify it as a DAG parameter instead
[2016-12-10 11:27:56,967] {models.py:1150} INFO - 

cat 2015-01-04T00 \：00 \：00
[2016-12-10 11:27:47158]{models.py:154}信息-从/Users/user_01/aiffort/dags填充DagBag
[2016-12-10 11:27:47214]{models.py:1750}警告-schedule_interval用于，尽管它已被弃用为任务参数，但您需要将其指定为DAG参数
[2016-12-10 11:27:47214]{models.py:1750}警告-schedule_interval用于，尽管它已被弃用为任务参数，但您需要将其指定为DAG参数
[2016-12-10 11:27:47227]{base_executor.py:36}信息-添加到队列：气流运行教程删除文件2015-01-04T00:00:00-本地-sd DAGS_文件夹/tutorial_01.py
[2016-12-10 11:27:47234]{sequential_executor.py:26}信息-执行命令：气流运行教程删除文件2015-01-04T00:00:00-本地-sd DAGS_文件夹/tutorial_01.py
[2016-12-10 11:27:48050]{models.py:154}信息-从/Users/user_01/aiffair/dags/tutorial_01.py填充DagBag
[2016-12-10 11:27:48101]{models.py:1750}警告-schedule_interval用于，尽管它已被弃用为任务参数，但您需要将其指定为DAG参数
[2016-12-10 11:27:48102]{models.py:1750}警告-schedule_interval用于，尽管它已被弃用为任务参数，但您需要将其指定为DAG参数
[2016-12-10 11:27:48942]{models.py:154}信息-从/Users/user_01/aiffair/dags/tutorial_01.py填充DagBag
[2016-12-10 11:27:48998]{models.py:1750}警告-schedule_interval用于，尽管它已被弃用为任务参数，但您需要将其指定为DAG参数
[2016-12-10 11:27:48998]{models.py:1750}警告-schedule_interval用于，尽管它已被弃用为任务参数，但您需要将其指定为DAG参数
[2016-12-10 11:27:49020]{models.py:1196}信息-
--------------------------------------------------------------------------------
开始尝试1次，共1次
--------------------------------------------------------------------------------
[2016-12-10 11:27:49046]{models.py:1219}信息-于2015-01-04 00:00:00执行
[2016-12-10 11:27:49054]{python_operator.py:67}信息-完成。返回值为：无
[2016-12-10 11:27:55168]{models.py:154}信息-从/Users/user_01/aiffort/dags填充DagBag
[2016-12-10 11:27:55219]{models.py:1750}警告-schedule_interval用于，尽管它已被弃用为任务参数，但您需要将其指定为DAG参数
[2016-12-10 11:27:55220]{models.py:1750}警告-schedule_interval用于，尽管它已被弃用为任务参数，但您需要将其指定为DAG参数
[2016-12-10 11:27:55231]{base_executor.py:36}信息-添加到队列：气流运行教程删除文件2015-01-04T00:00:00-本地-sd DAGS_文件夹/tutorial_01.py
[2016-12-10 11:27:55236]{sequential_executor.py:26}信息-执行命令：气流运行教程删除文件2015-01-04T00:00:00-本地-sd DAGS_文件夹/tutorial_01.py
[2016-12-10 11:27:56030]{models.py:154}信息-从/Users/user_01/aiffair/dags/tutorial_01.py填充DagBag
[2016-12-10 11:27:56082]{models.py:1750}警告-schedule_interval用于，尽管它已被弃用为任务参数，但您需要将其指定为DAG参数
[2016-12-10 11:27:56082]{models.py:1750}警告-schedule_interval用于，尽管它已被弃用为任务参数，但您需要将其指定为DAG参数
[2016-12-10 11:27:56899]{models.py:154}信息-从/Users/user_01/aiffair/dags/tutorial_01.py填充DagBag
[2016-12-10 11:27:56950]{models.py:1750}警告-schedule_interval用于，尽管它已被弃用为任务参数，但您需要将其指定为DAG参数
[2016-12-10 11:27:56951]{models.py:1750}警告-schedule_interval用于，尽管它已被弃用为任务参数，但您需要将其指定为DAG参数
[2016-12-10 11:27:56967]{models.py:1150}信息-

Airflow设计用于维护其DAG运行的历史记录，以便它可以按顺序处理批数据，并确保每个任务在其DAG运行时只运行一次

---

对于您尝试执行的操作，最简单的方法可能是忽略调度程序，并从外部触发执行日期为“now”的DagRun，包括完整日期和时间。这可以确保您调用的每次运行都会精确地执行所有任务一次，并且任务的每次运行都独立于以前的任何运行。您需要dependens_on_pass=False，并且您可能还需要max_active_runs成为某个非常大的值，因为任何失败的DAGRunts都将保持“活动”，但您不希望它们干扰新的调用。

我相信您的请求类似于：

“如果我们只需要运行一系列任务实例运行中最新的一个，并将其他任务实例标记为已跳过。例如，我们可能需要每天执行一个DB快照。如果DAG暂停5天，然后取消暂停，我们不想运行所有5个，只想运行最晚的一个

from airflow.operators.latest_only_operator import LatestOnlyOperator

dag = DAG(
    dag_id='latest_only_with_trigger',
    schedule_interval=dt.timedelta(hours=4),
    start_date=dt.datetime(2016, 9, 20),
)

latest_only = LatestOnlyOperator(task_id='latest_only', dag=dag)