Warning: file_get_contents(/data/phpspider/zhask/data//catemap/0/hadoop/6.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Hadoop 无法在mapreduce 2.x中运行wordcount示例_Hadoop_Mapreduce - Fatal编程技术网

Hadoop 无法在mapreduce 2.x中运行wordcount示例

Hadoop 无法在mapreduce 2.x中运行wordcount示例,hadoop,mapreduce,Hadoop,Mapreduce,我试图在java中的mapreduce 2.x中执行mapreduce字数示例。。。。我已经创建了jar,但在执行时显示了错误,比如在我的包中找不到WordMapper类,但我已经声明在我的包中……请帮助我解决这个问题 这是我的WordCount驱动程序代码: package com.mapreduce2.x; public class WordCount { public static void main(String args[]) throws IOException, ClassNo

我试图在java中的mapreduce 2.x中执行mapreduce字数示例。。。。我已经创建了jar,但在执行时显示了错误,比如在我的包中找不到WordMapper类,但我已经声明在我的包中……请帮助我解决这个问题

这是我的WordCount驱动程序代码:

package com.mapreduce2.x;

public class WordCount {

public static void main(String args[]) throws IOException, ClassNotFoundException, InterruptedException
{
    Configuration conf=new Configuration();

    org.apache.hadoop.mapreduce.Job job= new org.apache.hadoop.mapreduce.Job(conf, "Word_Count");


    job.setMapperClass(WordMapper.class);
    job.setReducerClass(WordReducer.class);

    job.setMapOutputKeyClass(Text.class);
    job.setMapOutputValueClass(IntWritable.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
    org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));

    job.waitForCompletion(true);

}}
这是我的WordMapper类:-

public class WordMapper extends Mapper<LongWritable, Text, Text,IntWritable>{

private final static IntWritable one=new IntWritable(1);
private Text word=new Text();

public void map(LongWritable key, Text value, org.apache.hadoop.mapreduce.Reducer.Context context) throws IOException, InterruptedException
{
    String line=value.toString();
    StringTokenizer tokenizer=new StringTokenizer(line);

    while(tokenizer.hasMoreTokens())
    {
        word.set(tokenizer.nextToken());
        context.write(word, one);

    }


}}

在运行JAR文件时包含类名,或者可以在创建JAR文件时指定主类名

如果运行时没有类名,则在运行JAR时指定类名

使用命令 hadoop jar word.jar com.mapreduce2.x.WordMapper/input/output

这里word.jar是jar文件名

您还可以在创建jar文件时包含主类名。 步骤: 文件-->导出-->JAR-->位置-->然后单击下一步-->它要求选择主类-->选择类并单击确定

之后,您可以使用命令运行jar文件

hadoop jar word.jar/input/output


希望这能解决您的问题。

尝试添加下面的注释行 Job Job=新作业(conf,“wordcount”); //job.setJarByClass(WordCount.class)

它对我很管用

你可以试试这个:(在Linux/Unix中)

  • 删除java代码中的包名

  • 在包含java程序的目录中,创建一个名为classes的新目录。例如:
    Hadoop Wordcount->classes,Wordcount.java

  • 编译:
    javac-classpath$HADOOP\u HOME/HADOOP-common-2.7.1.jar:$HADOOP\u HOME/HADOOP-mapreduce-client-core-2.7.1.jar:$HADOOP\u HOME/HADOOP-annotations-2.7.1.jar:$HADOOP\u HOME/commons-cli-1.2.jar-d./classes WordCount.java

  • 创建一个jar
    jar-cvf wordcount.jar-C./classes/


  • 5.run
    bin/hadoop jar$hadoop\u HOME/hadoop WordCount/WordCount.jar WordCount输入输出是我的主类,我在执行时指定了。。。。hadoop jar wordcount.jar com.mapreduce2.x.wordcount/input/output,但它不工作
    
    public class WordReducer extends Reducer<Text, IntWritable, Text, IntWritable>{
    
    
    public void reduce(Text key, Iterator<IntWritable> values,Context context) throws IOException, InterruptedException
    {
        int sum =0;
    
        while(values.hasNext())
        {
            sum= sum+values.next().get();
        }
    
        context.write(key, new IntWritable(sum));
    }}
    
    15/05/29 10:12:26 INFO mapreduce.Job:  map 0% reduce 0%
    15/05/29 10:12:33 INFO mapreduce.Job: Task Id : attempt_1432876892622_0005_m_000000_0, Status : FAILED
    Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.mapreduce2.x.WordMapper not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2076)
        at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:742)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
    Caused by: java.lang.ClassNotFoundException: Class com.mapreduce2.x.WordMapper not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2074)
        ... 8 more