Hadoop 无法在Psudo分布式模式下运行Pig脚本

Hadoop 无法在Psudo分布式模式下运行Pig脚本,hadoop,apache-pig,hdfs,Hadoop,Apache Pig,Hdfs,我正在尝试编写一个pig脚本。我成功地使它在本地模式下工作,但在Psudo分布式模式下启动pig时出错。 以下是脚本: register 'myfolder/target/myfunc-with-dependencies.jar'; SET mapred.cache.files /tmp/scripts#scripts,/tmp/my_rules#my_rules; SET mapred.create.symlink yes; %default INPUT 'test.seq' %defaul

我正在尝试编写一个pig脚本。我成功地使它在本地模式下工作,但在Psudo分布式模式下启动pig时出错。 以下是脚本:

register 'myfolder/target/myfunc-with-dependencies.jar';
SET mapred.cache.files /tmp/scripts#scripts,/tmp/my_rules#my_rules;
SET mapred.create.symlink yes;

%default INPUT 'test.seq'
%default OUTPUT 'final.out'
%default TIMEOUT_MS '180000'
%default USE_DEVSHM 'true'

data = load '$INPUT' using com.pack.loaders.MyTextLoader('$TIMEOUT_MS');
STORE data INTO '$OUTPUT'
我得到以下错误:

2013-09-03 08:08:44,314 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2013-09-03 08:08:44,316 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_201309030605_0019 has failed! Stop running all dependent jobs
2013-09-03 08:08:44,319 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2013-09-03 08:08:44,323 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2013-09-03 08:08:44,323 [main] WARN  org.apache.pig.tools.pigstats.ScriptState - unable to read pigs manifest file
2013-09-03 08:08:44,325 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: 

HadoopVersion   PigVersion  UserId  StartedAt   FinishedAt  Features
1.2.1       vmuser  2013-09-03 08:07:57 2013-09-03 08:08:44 UNKNOWN

Failed!

Failed Jobs:
JobId   Alias   Feature Message Outputs
job_201309030605_0019   data    MAP_ONLY    Message: Job failed! Error - JobCleanup Task Failure, Task: task_201309030605_0019_m_000001 hdfs://localhost:9000/user/vmuser/final.out,

Input(s):
Failed to read data from "hdfs://localhost:9000/user/vmuser/test.seq"

Output(s):
Failed to produce result in "hdfs://localhost:9000/user/vmuser/final.out"
该文件存在于HDFS上,另一个不注册任何jar并使用
PigStorage()
代替UDF的脚本工作正常。有没有其他方法来注册jar并调用UDF

清管器日志文件的内容包括:

清管器堆迹
此链接可能会帮助您:
并给出jar文件的完整路径,以便hadoop可以在任何地方找到它。

您可以粘贴Pig日志文件吗?我猜你的MyTextLoader出现了一些异常。UDF MyTextLoader在本地模式下工作正常。你能检查一下你的Pig版本是否与Hadoop版本兼容吗?也许你的Pig是用错误的Hadoop版本编译的。结果证明我使用的工具与我的Pig版本不兼容。我安装了一个较低的pig和hadoop版本,它运行正常。谢谢你的帮助
ERROR 2244: Job failed, hadoop does not return any error message

org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job failed, hadoop does not return any error message
    at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:145)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
    at org.apache.pig.Main.run(Main.java:604)
    at org.apache.pig.Main.main(Main.java:157)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:160)