Airflow 在运行时显示打印语句(不是在任务完成后)

Airflow 在运行时显示打印语句(不是在任务完成后),airflow,Airflow,假设我有以下DAG(为了清楚起见省略了内容) #dag.py 从afflow.operators.python导入PytonOperator def main(): 打印(“任务1”) #一些代码 打印(“任务2”) #还有代码吗 打印(“完成”) 返回0 t1=PythonOperator(python_callable=main) t1 假设程序在处失败#由于RAM问题,还有一些代码,我只是在日志中发现一个错误,例如 [2021-05-25 12:49:54,211] {process_u

假设我有以下DAG(为了清楚起见省略了内容)

#dag.py
从afflow.operators.python导入PytonOperator
def main():
打印(“任务1”)
#一些代码
打印(“任务2”)
#还有代码吗
打印(“完成”)
返回0
t1=PythonOperator(python_callable=main)
t1
假设程序在
处失败#由于RAM问题,还有一些代码
,我只是在日志中发现一个错误,例如

[2021-05-25 12:49:54,211] {process_utils.py:137} INFO - Output:
[2021-05-25 12:52:44,605] {taskinstance.py:1482} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.6/site-packages/airflow/operators/python.py", line 493, in execute
    super().execute(context=serializable_context)
  File "/usr/local/lib/python3.6/site-packages/airflow/operators/python.py", line 117, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python3.6/site-packages/airflow/operators/python.py", line 531, in execute_callable
    string_args_filename,
  File "/usr/local/lib/python3.6/site-packages/airflow/utils/process_utils.py", line 145, in execute_in_subprocess
    raise subprocess.CalledProcessError(exit_code, cmd)
subprocess.CalledProcessError: Command '['/tmp/venv2wbjnabi/bin/python', '/tmp/venv2wbjnabi/script.py', '/tmp/venv2wbjnabi/script.in', '/tmp/venv2wbjnabi/script.out', '/tmp/venv2wbjnabi/string_args.txt']' died with <Signals.SIGKILL: 9>.
[2021-05-25 13:00:55,733] {taskinstance.py:1532} INFO - Marking task as FAILED. dag_id=test_dag, task_id=clean_data, execution_date=20210525T105621, start_date=20210525T105732, end_date=20210525T110055
[2021-05-25 13:00:56,555] {local_task_job.py:146} INFO - Task exited with return code 1

[2021-05-25 12:49:54211]{process_utils.py:137}信息-输出:
[2021-05-25 12:52:44605]{taskinstance.py:1482}错误-任务失败,出现异常
回溯(最近一次呼叫最后一次):
文件“/usr/local/lib/python3.6/site packages/afflow/models/taskinstance.py”,第1138行,在原始任务中
self.\u准备\u和执行\u任务\u以及\u回调(上下文、任务)
文件“/usr/local/lib/python3.6/site packages/afflow/models/taskinstance.py”,第1311行,在带有回调的“准备”和“执行”任务中
结果=自我执行任务(上下文、任务、副本)
文件“/usr/local/lib/python3.6/site packages/afflow/models/taskinstance.py”,第1341行,在执行任务中
结果=任务\复制.执行(上下文=上下文)
文件“/usr/local/lib/python3.6/site packages/afflow/operators/python.py”,第493行,在execute中
super().execute(context=serializable_context)
文件“/usr/local/lib/python3.6/site packages/afflow/operators/python.py”,执行中的第117行
return\u value=self.execute\u callable()
文件“/usr/local/lib/python3.6/site packages/afflow/operators/python.py”,第531行,在execute\u callable中
字符串参数文件名,
文件“/usr/local/lib/python3.6/site packages/afflow/utils/process\u utils.py”,第145行,在执行子流程中
raise SUBSPROCESS.CalledProcessError(退出代码,cmd)
subprocess.CalledProcessError:Command“['/tmp/venv2wbjnabi/bin/python',“/tmp/venv2wbjnabi/script.py','/tmp/venv2wbjnabi/script.in','/tmp/venv2wbjnabi/script.out',“/tmp/venv2wbjnabi/string_args.txt']”与一起死亡。
[2021-05-25 13:00:55733]{taskinstance.py:1532}信息-将任务标记为失败。dag\u id=测试\u dag,任务\u id=清理\u数据,执行日期=20210525T105621,开始日期=20210525T105732,结束日期=20210525T110055
[2021-05-25 13:00:56555]{local_task_job.py:146}INFO-任务已退出,返回代码为1
但是没有打印语句,因此我不知道程序在哪里失败(由于调试,我现在知道了)


因此,我假设在任务被标记为“成功”之前,气流不会流动。有没有办法让气流在运行时刷新/在运行时打印?

没有可复制的代码/stacktrace,很难提供帮助。我的错误-我已经补充说,现在你是否尝试使用导入日志记录。信息(“你想要的任何日志”)我没有让气流记录工作-我使用它作为标准,但它实际上打印了很多wierd字符,因此,用于调试目的的打印声明考虑使用它用于在运行时显示输出,而不仅仅在日志中显示输出。