Java Pig错误:无法找到或加载主类org.apache.Pig.main
我是猪的初学者 我在WIKI之后编写了一个程序,将文件中的单词转换为大写 --cat UPPER.javaJava Pig错误:无法找到或加载主类org.apache.Pig.main,java,hadoop,mapreduce,apache-pig,bigdata,Java,Hadoop,Mapreduce,Apache Pig,Bigdata,我是猪的初学者 我在WIKI之后编写了一个程序,将文件中的单词转换为大写 --cat UPPER.java package com.bigdata.myUdf; import java.io.IOException; import org.apache.pig.EvalFunc; import org.apache.pig.data.Tuple; import org.apache.pig.impl.util.WrappedIOException; public class UPPER ex
package com.bigdata.myUdf;
import java.io.IOException;
import org.apache.pig.EvalFunc;
import org.apache.pig.data.Tuple;
import org.apache.pig.impl.util.WrappedIOException;
public class UPPER extends EvalFunc<String> {
public String exec(Tuple input) throws IOException {
if (input == null || input.size() == 0)
return null;
try{
String str = (String)input.get(0);
return str.toUpperCase();
}catch(Exception e){
throw WrappedIOException.wrap("Caught exception processing input row ", e);
}
}
}
但当我运行它时,我得到了错误
java -cp com.bigdata.myUdf.UPPER.jar org.apache.pig.Main -x local /home/hduser/lab/mydata/myscript.pig
错误
猫
export PIG_INSTALL=/home/hduser/software/pig
export PATH="${PATH}:${PIG_INSTALL}/bin"
export PIG_CLASSPATH=$HADOOP_CONF_DIR:${PIG_INSTALL}:.
export CLASSPATH=.:${PIG_CLASSPATH}
pig脚本位于:
/home/hduser/lab/mydata/myscript.pig
REGISTER /home/hduser/software/myUdfs/UPPER.jar
std_det = LOAD '/pigdata/udf1.txt' USING PigStorage(',') as (name:chararray);
B = FOREACH std_det GENERATE com.bigdata.myUdf.UPPER(name);
dump B;
JAR文件位于:
/home/hduser/software/myUdfs/UPPER.jar
请帮助我理解我做错了什么。提前谢谢。
遵循Shivashakti的指示后。程序运行了,但没有给出任何输出
pig -x local myScript.pig
15/01/05 04:47:57 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
15/01/05 04:47:57 INFO pig.ExecTypeProvider: Picked LOCAL as the ExecType
2015-01-05 04:47:57,920 [main] INFO org.apache.pig.Main - Apache Pig version 0.14.0 (r1640057) compiled Nov 16 2014, 18:02:05
2015-01-05 04:47:57,921 [main] INFO org.apache.pig.Main - Logging error messages to: /home/hduser/lab/piglog/pig_1420462077918.log
2015-01-05 04:47:57,959 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - user.name is deprecated. Instead, use mapreduce.job.user.name
2015-01-05 04:47:58,314 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-01-05 04:47:58,315 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-01-05 04:47:58,318 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2015-01-05 04:47:58,463 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-01-05 04:47:59,070 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-01-05 04:47:59,227 [main] INFO org.apache.pig.Main - Pig script completed in 2 seconds and 505 milliseconds (2505 ms)
该错误指示apache jar不在类路径中<代码>-cp com.bigdata.myUdf.UPPER.jar不包括所需的jar。它只包括'UPPER.jar'。您可以看看如何在类路径中正确地包含所有必需的JAR
另外,我认为您应该使用命令行中的pig命令,而不是按自己的方式执行。但是我自己没有用过,所以这只是一个猜测。你能按照以下步骤操作吗 1.从以下链接下载3个jar文件(pig-0.11.1.jar、hadoop-common-0.21.0.jar和piggybank.jar) 2。将上述3个jar文件全部设置为类路径
export CLASSPATH=/tmp/pig-0.11.1.jar:/tmp/hadoop-common-0.21.0.jar:/tmp/piggybank.jar
3。从当前目录创建目录名“com/bigdata/myUdf/”
>>mkdir -p com/bigdata/myUdf/
4。编译UPPER.java文件并确保java_HOME设置正确,并且上述三个jars文件都包含在类路径中,否则编译问题将出现
>>javac UPPER.java
5。将编译后的UPPER.class文件移到“com/bigdata/myUdf/”文件夹中
http://www.java2s.com/Code/Jar/p/Downloadpig0111jar.htm
http://www.java2s.com/Code/Jar/h/Downloadhadoopcommon0210jar.htm
http://www.java2s.com/Code/Jar/p/Downloadpiggybankjar.htm
>>mv UPPER.class com/bigdata/myUdf/
>>jar -cvf UPPER.jar com/
6。创建一个jar文件名UPPER.jar
http://www.java2s.com/Code/Jar/p/Downloadpig0111jar.htm
http://www.java2s.com/Code/Jar/h/Downloadhadoopcommon0210jar.htm
http://www.java2s.com/Code/Jar/p/Downloadpiggybankjar.htm
>>mv UPPER.class com/bigdata/myUdf/
>>jar -cvf UPPER.jar com/
7。现在将UPPER.jar包含到pig脚本中,并运行下面的命令
>>pig -x local myscript.pig
运行上述命令后,您将获得实际输出
示例输入
hello
world
myscript.pig
REGISTER UPPER.jar;
std_det = LOAD 'input' USING PigStorage(',') as (name:chararray);
B = FOREACH std_det GENERATE com.bigdata.myUdf.UPPER(name);
dump B;
输出:
(HELLO)
(WORLD)
$ ls
UPPER.java input myscript.pig
$ mkdir -p com/bigdata/myUdf/
$ javac UPPER.java
$ mv UPPER.class com/bigdata/myUdf/
$ jar -cvf UPPER.jar com/
$ pig -x local myscript.pig
示例命令:
(HELLO)
(WORLD)
$ ls
UPPER.java input myscript.pig
$ mkdir -p com/bigdata/myUdf/
$ javac UPPER.java
$ mv UPPER.class com/bigdata/myUdf/
$ jar -cvf UPPER.jar com/
$ pig -x local myscript.pig
我想java-cp不是必需的。您不是在执行嵌入java的pig,而是需要执行UDF。请尝试pig-x local file.pigHi-Sivasakthi,谢谢您的时间。现在,程序不会抱怨丢失了罐子。。。但是它没有给出任何有效的输出。我已经发布了上面的输出。很抱歉,我忘记了pig脚本中的转储。。现在工作正常了。谢谢你的帮助,谢谢你抽出时间。