现在，如何使用BashOperator通过python name.py运行.py文件

我使用芹菜和Redis来运行气流，它工作得很好，但我在工人方面遇到了问题。 我有两个docker compose文件，一个是在服务器上运行的主文件，另一个是在其他PC上运行的工作文件

我有运行

pythonscript.py

的dag，但总是失败，因为它找不到脚本。看起来Airflow Base task runner只是将dag文件复制到tmp文件夹中

测试_dag.py

from airflow.models import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta
default_args = {
        'owner': 'airflow',
        'start_date': datetime(2018, 9, 3)
        }
dag = DAG('test', default_args=default_args, schedule_interval='*/20 * * * *', catchup=False)

curl = BashOperator(
    task_id='testingbash',
    bash_command="python test.py",
    dag=dag)

test.py

print('it worked')

错误：

[2018-09-06 22:30:34,832] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash [2018-09-06 22:30:34,832] {cli.py:492} INFO - Running <TaskInstance: test.testingbash 2018-09-06T22:00:00+00:00 [running]> on host e91df5a905de
[2018-09-06 22:30:39,066] {bash_operator.py:74} INFO - Tmp dir root location: 
 /tmp
[2018-09-06 22:30:39,067] {bash_operator.py:87} INFO - Temporary script location: /tmp/airflowtmprgxtunoa/testingbash_7opnm28
[2018-09-06 22:30:39,067] {bash_operator.py:97} INFO - Running command: python test.py
[2018-09-06 22:30:39,079] {bash_operator.py:106} INFO - Output:
[2018-09-06 22:30:39,164] {bash_operator.py:110} INFO - python: can't open file 'test.py': [Errno 2] No such file or directory
[2018-09-06 22:30:39,165] {bash_operator.py:114} INFO - Command exited with return code 2
[2018-09-06 22:30:40,400] {models.py:1736} ERROR - Bash command failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1633, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.6/site-packages/airflow/operators/bash_operator.py", line 118, in execute
    raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed
[2018-09-06 22:30:40,408] {models.py:1764} INFO - Marking task as FAILED.
[2018-09-06 22:30:42,132] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash Traceback (most recent call last):
[2018-09-06 22:30:42,132] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash   File "/usr/local/bin/airflow", line 32, in <module>
[2018-09-06 22:30:42,132] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash     args.func(args)
[2018-09-06 22:30:42,133] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash   File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
[2018-09-06 22:30:42,133] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash     return f(*args, **kwargs)
[2018-09-06 22:30:42,133] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash   File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 498, in run
[2018-09-06 22:30:42,133] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash     _run(args, dag, ti)
[2018-09-06 22:30:42,133] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash   File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 402, in _run
[2018-09-06 22:30:42,134] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash     pool=args.pool,
[2018-09-06 22:30:42,134] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash   File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper
[2018-09-06 22:30:42,134] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash     return func(*args, **kwargs)
[2018-09-06 22:30:42,134] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash   File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1633, in _run_raw_task
[2018-09-06 22:30:42,134] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash     result = task_copy.execute(context=context)
[2018-09-06 22:30:42,134] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash   File "/usr/local/lib/python3.6/site-packages/airflow/operators/bash_operator.py", line 118, in execute
[2018-09-06 22:30:42,134] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash     raise AirflowException("Bash command failed")
[2018-09-06 22:30:42,135] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash airflow.exceptions.AirflowException: Bash command failed
[2018-09-06 22:30:46,945] {logging_mixin.py:95} INFO - [2018-09-06 22:30:46,944] {jobs.py:2612} INFO - Task exited with return code 1

[2018-09-06 22:30:34832]{base_task_runner.py:107}信息-作业4:Subtask testingbash[2018-09-06 22:30:34832]{cli.py:492}信息-在主机e91df5a905de上运行
[2018-09-06 22:30:39066]{bash_operator.py:74}INFO-Tmp dir根位置：
/tmp
[2018-09-06 22:30:39067]{bash_operator.py:87}信息-临时脚本位置：/tmp/airrowtmprgxtunoa/testingbash_7opnm28
[2018-09-06 22:30:39067]{bash_operator.py:97}INFO-运行命令：python test.py
[2018-09-06 22:30:39079]{bash_operator.py:106}信息-输出：
[2018-09-06 22:30:39164]{bash_operator.py:110}INFO-python:无法打开文件'test.py'：[Errno 2]没有这样的文件或目录
[2018-09-06 22:30:39165]{bash_operator.py:114}INFO-命令退出，返回代码为2
[2018-09-06 22:30:40400]{models.py:1736}错误-Bash命令失败
回溯（最近一次呼叫最后一次）：
文件“/usr/local/lib/python3.6/site packages/afflow/models.py”，第1633行，在原始任务中
结果=任务\复制.执行（上下文=上下文）
文件“/usr/local/lib/python3.6/site packages/afflow/operators/bash_operator.py”，执行中的第118行
raise AirflowException（“Bash命令失败”）
airflow.Exception.AirflowException:Bash命令失败
[2018-09-06 22:30:40408]{models.py:1764}信息-将任务标记为失败。
[2018-09-06 22:30:42132]{base_task_runner.py:107}信息-作业4：子任务testingbash回溯（最后一次调用）：
[2018-09-06 22:30:42132]{base_task_runner.py:107}信息-作业4：子任务testingbash文件“/usr/local/bin/aiffair”，第32行，in
[2018-09-06 22:30:42132]{base_task_runner.py:107}信息-作业4：子任务testingbash args.func（args）
[2018-09-06 22:30:42133]{base_task_runner.py:107}信息-作业4：子任务testingbash文件“/usr/local/lib/python3.6/site packages/afflow/utils/cli.py”，包装中的第74行
[2018-09-06 22:30:42133]{base_task_runner.py:107}信息-作业4：子任务testingbash返回f（*args，**kwargs）
[2018-09-06 22:30:42133]{base_task_runner.py:107}信息-作业4：子任务testingbash文件“/usr/local/lib/python3.6/site packages/afflow/bin/cli.py”，第498行，正在运行
[2018-09-06 22:30:42133]{base_task_runner.py:107}信息-作业4：子任务testingbash_运行（args、dag、ti）
[2018-09-06 22:30:42133]{base_task_runner.py:107}信息-作业4：子任务testingbash文件“/usr/local/lib/python3.6/site packages/afflow/bin/cli.py”，第402行，在运行中
[2018-09-06 22:30:42134]{base_task_runner.py:107}INFO-作业4：子任务testingbash pool=args.pool，
[2018-09-06 22:30:42134]{base_task_runner.py:107}信息-作业4：子任务testingbash文件“/usr/local/lib/python3.6/site packages/aiffort/utils/db.py”，第74行，在包装器中
[2018-09-06 22:30:42134]{base_task_runner.py:107}信息-作业4：子任务testingbash return func（*args，**kwargs）
[2018-09-06 22:30:42134]{base_task_runner.py:107}信息-作业4：子任务testingbash文件“/usr/local/lib/python3.6/site packages/aiffort/models.py”，第1633行，在_run_raw_task中
[2018-09-06 22:30:42134]{base_task_runner.py:107}INFO-Job 4:Subtask testingbash result=task_copy.execute（context=context）
[2018-09-06 22:30:42134]{base_task_runner.py:107}信息-作业4：子任务testingbash文件“/usr/local/lib/python3.6/site packages/aiffort/operators/bash_operator.py”，第118行，执行
[2018-09-06 22:30:42134]{base_task_runner.py:107}信息-作业4：子任务testingbash引发空气流量异常（“Bash命令失败”）
[2018-09-06 22:30:42135]{base_task_runner.py:107}信息-作业4：子任务testingbash airflow.exceptions.AirflowException:Bash命令失败
[2018-09-06 22:30:46945]{logging_mixin.py:95}信息-[2018-09-06 22:30:46944]{jobs.py:2612}信息-任务退出，返回代码为1

解决方案：

---

几天前修复了，我没有考虑docker正在构建的结构，现在我使用

python~/scripts/test.py运行，它运行得很好。
如果您只想运行一个python脚本，使用PythonOperator可能会更容易

如果设置为使用BashOperator，则只需包含文件的绝对文件路径-默认情况下，它创建并在几天前修复的，我没有考虑docker正在构建的临时结构，现在我使用python~/scripts/test.py运行，它运行得很好。
你能发布你是如何编写DAG的吗？您可能只需要包含.py的绝对文件路径file@VirajParekh添加了更多信息我不知道所有这些“FooOperator”的东西，但一般来说，您总是希望从
python-m
开始，或者使用setuptools生成的
入口点
脚本。
[2018-09-06 22:30:34,832] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash [2018-09-06 22:30:34,832] {cli.py:492} INFO - Running <TaskInstance: test.testingbash 2018-09-06T22:00:00+00:00 [running]> on host e91df5a905de
[2018-09-06 22:30:39,066] {bash_operator.py:74} INFO - Tmp dir root location: 
 /tmp
[2018-09-06 22:30:39,067] {bash_operator.py:87} INFO - Temporary script location: /tmp/airflowtmprgxtunoa/testingbash_7opnm28
[2018-09-06 22:30:39,067] {bash_operator.py:97} INFO - Running command: python test.py
[2018-09-06 22:30:39,079] {bash_operator.py:106} INFO - Output:
[2018-09-06 22:30:39,164] {bash_operator.py:110} INFO - python: can't open file 'test.py': [Errno 2] No such file or directory
[2018-09-06 22:30:39,165] {bash_operator.py:114} INFO - Command exited with return code 2
[2018-09-06 22:30:40,400] {models.py:1736} ERROR - Bash command failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1633, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.6/site-packages/airflow/operators/bash_operator.py", line 118, in execute
    raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed
[2018-09-06 22:30:40,408] {models.py:1764} INFO - Marking task as FAILED.
[2018-09-06 22:30:42,132] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash Traceback (most recent call last):
[2018-09-06 22:30:42,132] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash   File "/usr/local/bin/airflow", line 32, in <module>
[2018-09-06 22:30:42,132] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash     args.func(args)
[2018-09-06 22:30:42,133] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash   File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
[2018-09-06 22:30:42,133] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash     return f(*args, **kwargs)
[2018-09-06 22:30:42,133] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash   File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 498, in run
[2018-09-06 22:30:42,133] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash     _run(args, dag, ti)
[2018-09-06 22:30:42,133] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash   File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 402, in _run
[2018-09-06 22:30:42,134] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash     pool=args.pool,
[2018-09-06 22:30:42,134] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash   File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper
[2018-09-06 22:30:42,134] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash     return func(*args, **kwargs)
[2018-09-06 22:30:42,134] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash   File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1633, in _run_raw_task
[2018-09-06 22:30:42,134] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash     result = task_copy.execute(context=context)
[2018-09-06 22:30:42,134] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash   File "/usr/local/lib/python3.6/site-packages/airflow/operators/bash_operator.py", line 118, in execute
[2018-09-06 22:30:42,134] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash     raise AirflowException("Bash command failed")
[2018-09-06 22:30:42,135] {base_task_runner.py:107} INFO - Job 4: Subtask testingbash airflow.exceptions.AirflowException: Bash command failed
[2018-09-06 22:30:46,945] {logging_mixin.py:95} INFO - [2018-09-06 22:30:46,944] {jobs.py:2612} INFO - Task exited with return code 1