Pig 0.7.0错误2118:无法在Hadoop 1.2.1上创建输入拆分
我从MapReduce程序中获得了输出文件(存储在HDFS上)。现在,我尝试使用PIG 0.7.0加载该文件 我得到以下错误。我试着将这个文件复制到本地机器上,并在本地模式下运行pig,效果很好。但是我想跳过这一步,让它在MapReduce模式下工作 我尝试过的选项:Pig 0.7.0错误2118:无法在Hadoop 1.2.1上创建输入拆分,hadoop,apache-pig,Hadoop,Apache Pig,我从MapReduce程序中获得了输出文件(存储在HDFS上)。现在,我尝试使用PIG 0.7.0加载该文件 我得到以下错误。我试着将这个文件复制到本地机器上,并在本地模式下运行pig,效果很好。但是我想跳过这一步,让它在MapReduce模式下工作 我尝试过的选项: LOAD 'file://log/part-00000', LOAD '/log/part-00000', LOAD 'hdfs:/log/part-00000', LOAD 'hdfs://localhost:50070/
LOAD 'file://log/part-00000',
LOAD '/log/part-00000',
LOAD 'hdfs:/log/part-00000',
LOAD 'hdfs://localhost:50070/log/part-00000',
清管器(在mapreduce模式下运行) 清管器堆迹
... 6更多您应该尝试升级到Pig的最新版本。0.7.0已经有好几年的历史了。0.12.0是当前的稳定版本。如何
REC=LOAD'/log'as(创建:chararray,消息类型:chararray,消息来源:chararray,消息文本:chararray)
感谢您的快速回复。同样的错误…我将尝试升级的pig 0.12.0,并将我的发现反馈给大家。原因:org.apache.hadoop.mapreduce.lib.input.InvalidInputException:输入路径不存在:文件:/Log对于在查找此帖子时发现此帖子的人,这里是一个。感谢您的快速回复…我将试用升级后的pig 0.12.0,并将我的发现回复给大家。谢谢,您的技巧奏效了。Pig 0.7只在Hadoop 0.20上工作。我试过使用Pig 0.12.0,它成功了!:)-->REC=LOAD'hdfs:/log/part-00000'…我很高兴新版本适合您。大多数Hadoop生态系统都是特定于版本的。如果您有幸能够运行其中一个发行版(,),您将省去确保所有工具版本兼容的麻烦。
hadoop dfs -ls /log/
Warning: $HADOOP_HOME is deprecated.
Found 3 items
-rw-r--r-- 3 supergroup 0 2014-02-07 07:56 /log/_SUCCESS
drwxr-xr-x - supergroup 0 2014-02-07 07:55 /log/_logs
-rw-r--r-- 3 supergroup 10021 2014-02-07 07:56 /log/part-00000
grunt> REC = LOAD 'file://log/part-00000' as (CREATE_TMSTP:chararray, MESSAGE_TYPE:chararray, MESSAGE_FROM:chararray, MESSAGE_TEXT:chararray);
grunt> DUMP REC;
Backend error message during job submission
-------------------------------------------
org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Unable to create input splits for: file:///log/part-00000
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:269)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:695)
Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/log/part-00000
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:258)
... 7 more
ERROR 2997: Unable to recreate exception from backend error:org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Unable to create input splits for: file:///log/part-00000
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias REC
at org.apache.pig.PigServer.openIterator(PigServer.java:521)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:544)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:162)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:138)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:75)
at org.apache.pig.Main.main(Main.java:357)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Unable to create input splits for: file:///log/part-00000
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:169)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:268)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:308)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:835)
at org.apache.pig.PigServer.store(PigServer.java:569)
at org.apache.pig.PigServer.openIterator(PigServer.java:504)