Python “操作员抛出”;没有这样的文件或目录;错误

Python “操作员抛出”;没有这样的文件或目录;错误,python,python-2.7,airflow,airbnb,Python,Python 2.7,Airflow,Airbnb,我无法找出我提出的问题,日志如下所示。 下面还显示了我创建的DAG、connection和pig脚本 达格: 猪字: rmf /onlyvinish/sample_out; a_load = load '/onlyvinish/sample.txt' using PigStorage(','); a_gen = foreach a_load generate (int)$0 as a; b_gen = foreach a_gen generate a, a+1, a+2, a+3, a+4, a

我无法找出我提出的问题,日志如下所示。 下面还显示了我创建的DAG、connection和pig脚本

达格:

猪字:

rmf /onlyvinish/sample_out;
a_load = load '/onlyvinish/sample.txt' using PigStorage(',');
a_gen = foreach a_load generate (int)$0 as a;
b_gen = foreach a_gen generate a, a+1, a+2, a+3, a+4, a+5;
store b_gen into '/onlyvinish/sample_out' using PigStorage(',');
连接:

失败任务的日志:

[2017-01-24 00:03:27,199] {models.py:168} INFO - Filling up the DagBag from /home/hadoop/airflow/dags/ETL.py
[2017-01-24 00:03:27,276] {jobs.py:2042} INFO - Subprocess PID is 8532
[2017-01-24 00:03:29,410] {models.py:168} INFO - Filling up the DagBag from /home/hadoop/airflow/dags/ETL.py
[2017-01-24 00:03:29,487] {models.py:1078} INFO - Dependencies all met for <TaskInstance: ETL-DEMO.fly_task_2 2017-01-24 00:03:07.199790 [queued]>
[2017-01-24 00:03:29,496] {models.py:1078} INFO - Dependencies all met for <TaskInstance: ETL-DEMO.fly_task_2 2017-01-24 00:03:07.199790 [queued]>
[2017-01-24 00:03:29,496] {models.py:1266} INFO - 
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------

[2017-01-24 00:03:29,533] {models.py:1289} INFO - Executing <Task(PigOperator): fly_task_2> on 2017-01-24 00:03:07.199790
[2017-01-24 00:03:29,550] {pig_operator.py:64} INFO - Executing: rmf /onlyvinish/sample_out;

a_load = load '/onlyvinish/sample.txt' using PigStorage(',');

a_gen = foreach a_load generate (int)$0 as a;
b_gen = foreach a_gen generate a, a+1, a+2, a+3, a+4, a+5;

store b_gen into '/onlyvinish/sample_out' using PigStorage(',');
[2017-01-24 00:03:29,612] {pig_hook.py:67} INFO - pig -f /tmp/airflow_pigop_sm5bjE/tmpNP0ZXM
[2017-01-24 00:03:29,620] {models.py:1364} ERROR - [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/airflow-1.7.2.dev0-py2.7.egg/airflow/models.py", line 1321, in run
    result = task_copy.execute(context=context)
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/airflow-1.7.2.dev0-py2.7.egg/airflow/operators/pig_operator.py", line 66, in execute
    self.hook.run_cli(pig=self.pig)
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/airflow-1.7.2.dev0-py2.7.egg/airflow/hooks/pig_hook.py", line 72, in run_cli
    cwd=tmp_dir)
  File "/home/hadoop/anaconda2/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/home/hadoop/anaconda2/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
[2017-01-24 00:03:29,623] {models.py:1388} INFO - Marking task as FAILED.
[2017-01-24 00:03:29,636] {models.py:1409} ERROR - [Errno 2] No such file or directory
[2017-01-24 00:03:27199]{models.py:168}INFO-从/home/hadoop/aiffort/dags/ETL.py填充DagBag
[2017-01-24 00:03:27276]{jobs.py:2042}信息-子流程PID为8532
[2017-01-24 00:03:29410]{models.py:168}信息-从/home/hadoop/aiffort/dags/ETL.py填充DagBag
[2017-01-24 00:03:29487]{models.py:1078}信息-所有依赖项都满足
[2017-01-24 00:03:29496]{models.py:1078}信息-所有依赖项都满足
[2017-01-24 00:03:29496]{models.py:1266}信息-
--------------------------------------------------------------------------------
开始尝试1次,共1次
--------------------------------------------------------------------------------
[2017-01-24 00:03:29533]{models.py:1289}信息-于2017-01-24 00:03:07.199790执行
[2017-01-24 00:03:29550]{pig_operator.py:64}信息-执行:rmf/onlyvinish/sample_out;
a_load=load'/onlyvinish/sample.txt',使用PigStorage(',');
a_gen=每个a_load generate(int)$0作为a;
b_gen=每个a_gen生成a、a+1、a+2、a+3、a+4、a+5;
使用PigStorage(',)将b_gen存储到'/onlyfinish/sample_out'中;
[2017-01-24 00:03:29612]{pig_hook.py:67}INFO-pig-f/tmp/aiffair_pigop_sm5bjE/tmpNP0ZXM
[2017-01-24 00:03:29620]{models.py:1364}错误-[Errno 2]没有这样的文件或目录
回溯(最近一次呼叫最后一次):
文件“/home/hadoop/anaconda2/lib/python2.7/site packages/afflow-1.7.2.dev0-py2.7.egg/afflow/models.py”,第1321行,运行中
结果=任务\复制.执行(上下文=上下文)
文件“/home/hadoop/anaconda2/lib/python2.7/site packages/afflow-1.7.2.dev0-py2.7.egg/afflow/operators/pig_operator.py”,第66行,执行
self.hook.run_cli(pig=self.pig)
文件“/home/hadoop/anaconda2/lib/python2.7/site packages/afflow-1.7.2.dev0-py2.7.egg/afflow/hooks/pig_hook.py”,第72行,在run_cli中
cwd=tmp_目录)
文件“/home/hadoop/anaconda2/lib/python2.7/subprocess.py”,第711行,在__
错误读取,错误写入)
文件“/home/hadoop/anaconda2/lib/python2.7/subprocess.py”,第1343行,在执行子进程中
引发子对象异常
OSError:[Errno 2]没有这样的文件或目录
[2017-01-24 00:03:29623]{models.py:1388}信息-将任务标记为失败。
[2017-01-24 00:03:29636]{models.py:1409}错误-[Errno 2]没有这样的文件或目录
  • 气流:1.7.2
  • Python:2.7
  • 瑞尔:6.7

请让我知道我做错了什么。

pig脚本必须是一个模板字符串,而不是pig脚本本身的路径。

我要说的是,
pig
不在您的路径中。如果是这样,pig内容在日志中的显示方式@Jean François Fabrecan你能详细说明一下吗?
[2017-01-24 00:03:27,199] {models.py:168} INFO - Filling up the DagBag from /home/hadoop/airflow/dags/ETL.py
[2017-01-24 00:03:27,276] {jobs.py:2042} INFO - Subprocess PID is 8532
[2017-01-24 00:03:29,410] {models.py:168} INFO - Filling up the DagBag from /home/hadoop/airflow/dags/ETL.py
[2017-01-24 00:03:29,487] {models.py:1078} INFO - Dependencies all met for <TaskInstance: ETL-DEMO.fly_task_2 2017-01-24 00:03:07.199790 [queued]>
[2017-01-24 00:03:29,496] {models.py:1078} INFO - Dependencies all met for <TaskInstance: ETL-DEMO.fly_task_2 2017-01-24 00:03:07.199790 [queued]>
[2017-01-24 00:03:29,496] {models.py:1266} INFO - 
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------

[2017-01-24 00:03:29,533] {models.py:1289} INFO - Executing <Task(PigOperator): fly_task_2> on 2017-01-24 00:03:07.199790
[2017-01-24 00:03:29,550] {pig_operator.py:64} INFO - Executing: rmf /onlyvinish/sample_out;

a_load = load '/onlyvinish/sample.txt' using PigStorage(',');

a_gen = foreach a_load generate (int)$0 as a;
b_gen = foreach a_gen generate a, a+1, a+2, a+3, a+4, a+5;

store b_gen into '/onlyvinish/sample_out' using PigStorage(',');
[2017-01-24 00:03:29,612] {pig_hook.py:67} INFO - pig -f /tmp/airflow_pigop_sm5bjE/tmpNP0ZXM
[2017-01-24 00:03:29,620] {models.py:1364} ERROR - [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/airflow-1.7.2.dev0-py2.7.egg/airflow/models.py", line 1321, in run
    result = task_copy.execute(context=context)
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/airflow-1.7.2.dev0-py2.7.egg/airflow/operators/pig_operator.py", line 66, in execute
    self.hook.run_cli(pig=self.pig)
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/airflow-1.7.2.dev0-py2.7.egg/airflow/hooks/pig_hook.py", line 72, in run_cli
    cwd=tmp_dir)
  File "/home/hadoop/anaconda2/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/home/hadoop/anaconda2/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
[2017-01-24 00:03:29,623] {models.py:1388} INFO - Marking task as FAILED.
[2017-01-24 00:03:29,636] {models.py:1409} ERROR - [Errno 2] No such file or directory