使用mapreduce解析twitter json:Java，Pig

我相信你可能会发现这个问题有些“重复”，但我相信我在发布同样的问题之前已经做了研究。我也很抱歉在一个线程中发布Java和Pig问题，但我不想为同一个问题创建另一个线程

我得到了一个json文件和一些twitter摘录。我也在尝试使用javamr&Pig执行解析，但面临一些问题。下面是我尝试编写的Java代码：

public class twitterDataStore {
    private static final ObjectMapper mapper = new ObjectMapper();
    public static abstract class Map extends MapReduceBase implements 
    Mapper<Object, Text, IntWritable, Reporter>{

        private static final IntWritable one = new IntWritable();
        private Text word = new Text();

        public void map(Object key, Text value, OutputCollector<Text, IntWritable> context, Reporter arg3)
        throws IOException{
            try{
                JSONObject jsonObj = new JSONObject(value.toString());

                String text = (String) jsonObj.get("retweet_count");
                StringTokenizer strToken = new StringTokenizer(text);

                while(strToken.hasMoreTokens()){
                    word.set(strToken.nextToken());
                    context.collect(word, one);
                }
            }catch(IOException e){
                e.printStackTrace();
            } catch (JSONException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }

        }


    }
    public static class Reduce extends MapReduceBase implements Reducer<Object, Text, IntWritable, Reporter>{

        @Override
        public void reduce(Object key, Iterator<Text> value,
                OutputCollector<IntWritable, Reporter> context, Reporter arg3)
                throws IOException {
                    while(value.hasNext()){
                        System.out.println(value);
                        if(value.equals("retweet_count"))
                        {
                            System.out.println(value.equals("id_str"));
                        }

                    }
            // TODO Auto-generated method stub

        }

    }
    public static void main(String[] args) throws IOException{

        JobConf conf = new JobConf(twitterDataStore.class);
        conf.setJobName("twitterDataStore");

        conf.setMapperClass(Map.class);
        conf.setReducerClass(Reducer.class);

        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(IntWritable.class);

        conf.setInputFormat(TextInputFormat.class);
        conf.setOutputFormat(TextOutputFormat.class);

        FileInputFormat.setInputPaths(conf, new Path(args[0]));
        FileOutputFormat.setOutputPath(conf,new Path(args[1]));

        JobClient.runJob(conf);
    }

}

谈到Pig（版本0.11）加载，我编写了一个JsonLoader加载来加载我的tweet数据：

data = LOAD '/tmp/twitter.txt' using JsonLoader('in_reply_to_screen_name:chararray,text:chararray,id_str:long,place:chararray,in_reply_to_status_id:chararray, contributors:chararray,retweet_count:CHARARRAY,favorited:chararray,truncated:chararray,source:chararray,in_reply_to_status_id_str:chararray,created_at:chararray, in_reply_to_user_id_str:chararray,in_reply_to_user_id:chararray,user:{(lang:chararray,profile_background_image_url:chararray,id_str:long,default_profile_image: chararray,statuses_count:chararray,profile_link_color:chararray,favourites_count:chararray,profile_image_url_https:chararray,following:chararray, profile_background_color:chararray,description:chararray,notifications:chararray,profile_background_tile:chararray,time_zone:chararray, profile_sidebar_fill_color:chararray,listed_count:chararray,contributors_enabled:chararray,geo_enabled:chararray,created_at:chararray,screen_name:chararray, follow_request_sent:chararray,profile_sidebar_border_color:chararray,protected:chararray,url:chararray,default_profile:chararray,name:chararray, is_translator:chararray,show_all_inline_media:chararray,verified:chararray,profile_use_background_image:chararray,followers_count:chararray,profile_image_url:chararray, id:long,profile_background_image_url_https:chararray,utc_offset:chararray,friends_count:chararray,profile_text_color:chararray,location:chararray)},retweeted:chararray, id:long,coordinates:chararray,geo:chararray');

很抱歉在这里粘贴了所有不必要的内容，但我不想错过解释中的任何内容

我在将一些字段声明为“integer”时遇到了问题，但当我将所有整数转换为chararray时，该命令通过了检查。我这里的错误是

2014-04-14 21:19:24,977 [JobControl] WARN  org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2014-04-14 21:19:24,982 [JobControl] WARN  org.apache.hadoop.mapred.JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

解析也存在同样的问题。我尝试在加载之前注册json jar，但仍然存在相同的问题。有人能帮我解决这个问题吗

提前谢谢。

---

-Adil

您使用的是TextInputFormat，因此当您执行value.toString（）时，会得到一行作为输入。不是整个JSON对象

我建议您使用以下方法：

2014-04-14 21:19:24,977 [JobControl] WARN  org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2014-04-14 21:19:24,982 [JobControl] WARN  org.apache.hadoop.mapred.JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).