Apache pig Pig UDF正在从UDF抛出一个错误捕获错误：GetCounty，越界访问[1]

我正在编写一个pig程序，它读取包含city、zip的文件，然后将city传递给UDF。UDF将加载哈希映射中包含县、市的文件。UDF然后从哈希映射中找到城市的县并返回它

请让我知道我在这里做错了什么；我在运行程序时遇到以下错误：

2014-12-28 16:15:16,506 WARN org.apache.hadoop.mapred.Child: Error running child
org.apache.pig.backend.executionengine.ExecException: ERROR 2078: Caught error from UDF: GetCounty, Out of bounds access [1]
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:370)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:434)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:340)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:283)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at GetCounty.exec(GetCounty.java:33)
at GetCounty.exec(GetCounty.java:1)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
... 15 more
2014-12-28 16:15:16,510 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

输入文件包含以下数据：

File zipcity:
irving  75038
san francisco   94903
san rafael      94905
las vegas       98043
coppel  75063

File citycounty:
irving  dallas
las vegas       tarrant
san francisco   san francisco
coppel  dallas

public class GetCounty extends EvalFunc<String> {
String lookupfile;
HashMap<String, String> lookup = null;

public String exec(Tuple input) throws IOException{
    if ( input.size() != 1 ){       
        return null;
    }

    if ( lookup == null ) {
        FileSystem fs = FileSystem.get(UDFContext.getUDFContext().getJobConf());
        DataInputStream in = fs.open(new Path(lookupfile));
        String line;
        while ( (line = in.readLine()) != null){
            String[] tok = new String[2];
            tok = line.split(":", 2);
            lookup.put(tok[0], tok[1]);
        }}

    String city = (String) input.get(0);        
    return lookup.get(city);        
}

public GetCounty(String f){
    lookupfile = f;
}
}

---

你能试试这个吗？。输入字段由制表符分隔

拉链

城市国家

笔迹：

输出：

---

使用pig可以很容易地解决这个问题，只是想知道为什么要使用UDF？。在UDF代码中，为什么要根据：作为分隔符拆分字符串，我在输入中没有看到：。谢谢Sivasakhthi！！这是由于代码复制粘贴习惯。但我几个小时都找不到它。我用的不是UDF，而是使用native pig发布的解决方案，请让我知道这是否适用于您。谢谢Sivasakhthi！我在UDF中用你的观点解决了这个问题，我改变了delim，它成功了。另一件事是我的代码由于HashMap lookup=null而失败；已声明，导致空指针异常。Pig book中的编程也有这个bug。我能够使用您提供的解决方案解决此问题。

grunt> register 'PigMyUDF.jar';
grunt> define GetCounty GetCounty('pig/citycounty');
grunt> a = load 'pig/zipcity' as ( city:chararray, zip:int );
grunt> b = foreach a generate city, zip, GetCounty(city);
grunt> dump b;

irving  75038
san francisco   94903
san rafael      94905
las vegas       98043
coppel  75063

irving  dallas
las vegas       tarrant
san francisco   san francisco
coppel  dallas

A = LOAD 'zipcity' AS (city:chararray, zip:int);
B = LOAD 'citycounty' AS  (city:chararray,country:chararray);
C = JOIN A BY city,B BY city;
D = FOREACH C GENERATE A::city AS city,A::zip AS zip,B::country AS country;
DUMP D;

(coppel,75063,dallas)
(irving,75038,dallas)
(las vegas,98043,tarrant)
(san francisco,94903,san francisco)