Warning: file_get_contents(/data/phpspider/zhask/data//catemap/0/hadoop/6.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Hadoop 错误2997:遇到IOException。找不到要替换参数的文件_Hadoop_Apache Pig - Fatal编程技术网

Hadoop 错误2997:遇到IOException。找不到要替换参数的文件

Hadoop 错误2997:遇到IOException。找不到要替换参数的文件,hadoop,apache-pig,Hadoop,Apache Pig,使用-dryrun选项运行pig脚本时,出现错误2997。 清管器脚本、参数文件和输入文件都放在hdfs位置 pig -param_file hdfs://xxx.xx.xx.xxx:8020/user/hdfs/pig/data/part-m-00000 --param "input_file=/user/hdfs/pig/data/input_file.txt" --param "timeband_file=/user/hdfs/pig/data/file2.txt" --param

使用-dryrun选项运行pig脚本时,出现错误2997。 清管器脚本、参数文件和输入文件都放在hdfs位置

pig -param_file hdfs://xxx.xx.xx.xxx:8020/user/hdfs/pig/data/part-m-00000    --param "input_file=/user/hdfs/pig/data/input_file.txt" --param "timeband_file=/user/hdfs/pig/data/file2.txt"  --param "output_file=hdfs://xxx.xx.xx.xxx:8020/user/hdfs/pig/data/output"    -dryrun -f hdfs://xxx.xx.xx.xxx:8020/user/hdfs/pig/scripts/Generate_Agg_Flow_ParamFile_Store.pig
WARNING: Use "yarn jar" to launch YARN applications.
16/02/01 21:33:41 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
16/02/01 21:33:41 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
16/02/01 21:33:41 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
2016-02-01 21:33:41,983 [main] INFO  org.apache.pig.Main - Apache Pig version 0.15.0.2.3.4.0-3485 (rexported) compiled Dec 16 2015, 04:30:33
2016-02-01 21:33:41,983 [main] INFO  org.apache.pig.Main - Logging error messages to: /home/hdfs/pig_1454380421982.log
2016-02-01 21:33:43,743 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /home/hdfs/.pigbootup not found
2016-02-01 21:33:43,769 [main] ERROR org.apache.pig.Main - ERROR 2997: Encountered IOException. Could not find file to substitute parameters for: hdfs://xxx.xx.xx.xxx:8020/user/hdfs/pig/scripts/Generate_Agg_Cash_Flow_ParamFile_Store.pig.substituted
Details at logfile: /home/hdfs/pig_1454380421982.log
2016-02-01 21:33:43,792 [main] INFO  org.apache.pig.Main - Pig script completed in 1 second and 962 milliseconds (1962 ms)
日志文件位于/home/hdfs/pig_1454380421982.Log

 Error before Pig is launched
    ----------------------------
    ERROR 2997: Encountered IOException. Could not find file to substitute parameters for: hdfs://xxx.xx.xx.xxx:8020/user/hdfs/pig/scripts/Generate_Agg_Flow_ParamFile_Store.pig.substituted

    java.io.IOException: Could not find file to substitute parameters for: hdfs://172.31.29.201:8020/user/hdfs/pig/scripts/Generate_Agg_Cash_Flow_ParamFile_Store.pig.substituted
        at org.apache.pig.impl.PigContext.doParamSubstitutionOutputToFile(PigContext.java:441)
        at org.apache.pig.Main.runParamPreprocessor(Main.java:829)
        at org.apache.pig.Main.run(Main.java:466)
        at org.apache.pig.Main.main(Main.java:177)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
    ================================================================================
    Close

如果是本地文件系统中的脚本和参数文件。dryrun选项有效。pig-param_file/TestDivya/pig/GroupByParam_file.txt-f/TestDivya/pig/GroupParamTest.pig-dryrun使用wasb时也会失败://