Warning: file_get_contents(/data/phpspider/zhask/data//catemap/9/java/352.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Java 是否需要复制MapReduce中的Reducer值,否则会进行修改?_Java_Hadoop_Mapreduce - Fatal编程技术网

Java 是否需要复制MapReduce中的Reducer值,否则会进行修改?

Java 是否需要复制MapReduce中的Reducer值,否则会进行修改?,java,hadoop,mapreduce,Java,Hadoop,Mapreduce,在MapReduce应用程序中,我有一个名为AnonymousPair的任意可写可比实现,我注意到了这一点 import com.google.common.collect.MinMaxPriorityQueue; public static class MyReducer extends Reducer<LongWritable, AnonymousPair, LongWritable, Text> { @Override protected void reduc

在MapReduce应用程序中,我有一个名为AnonymousPair的任意可写可比实现,我注意到了这一点

import com.google.common.collect.MinMaxPriorityQueue;

public static class MyReducer extends Reducer<LongWritable, AnonymousPair, LongWritable, Text> {
    @Override
    protected void reduce(LongWritable key, Iterable<AnonymousPair> values, Context context) throws IOException, InterruptedException { 
        // ...
        MinMaxPriorityQueue<AnonymousPair> pQueue = MinMaxPriorityQueue
                .orderedBy(new AnonymousPair().comparator())
                .maximumSize(Constants.MaxKeywords)
                .create();

        for(AnonymousPair val : values) {
            pQueue.add(new AnonymousPair(val)); // No problem with copy constructor
            // pQueue.add(val);                 // Wrong! Every element in pQueue will be the same 
        }
    }
}
如果我不使用“复制构造函数”,PQUE中的每个元素最终都是相同的。有人能帮我理解吗?谢谢 我的猜测是

参考要修改的元素。它在医生的某个地方,但我没找到。 我用Google Guava MinMaxPriorityQueue时出错了 或者我的可写可比实现有问题 我的匿名对实现

    public static class AnonymousPair implements WritableComparable<AnonymousPair> {
        private String a = "";
        private Float b = 0f;
        public AnonymousPair() {}
        public AnonymousPair(String a, Float b) {this.a = a; this.b = b;}
        public AnonymousPair(AnonymousPair o) {this.a = o.a; this.b = o.b;}

        public Comparator<AnonymousPair> comparator() {return new AnonymousPairComparator();}

        class AnonymousPairComparator implements Comparator<AnonymousPair> {
            @Override
            public int compare(AnonymousPair o1, AnonymousPair o2) {
                Float diff = o1.b - o2.b;
                if(diff == 0) {
                    return 0;
                }
                if(diff < 0) {
                    return 1;   // Reverse order
                } else {
                    return -1;
                }

            }
        }

        @Override
        public int compareTo(AnonymousPair o) {
            int temp = this.a.compareTo(o.a);
            if(temp == 0) {
                return -this.b.compareTo(o.b);
            } else {
                return temp;
            }
        }

        // More overriding...
    }
见:

框架将重用传递的键和值对象 因此,应用程序应该克隆对象 他们想保留一份。在许多情况下,所有值都是组合的 转换为零或一个值


显示您的AnonymousPair类以及PQUE的实现/定义位置我看不到ret变量的声明位置。抱歉。应该是“返回”,谢谢!正是我要找的。但是重用如何帮助框架呢?WritableComparable对我来说似乎是轻量级的,因为它只保留了一些引用。是的,但将其与处理非常大的文件结合起来,在这些文件中可以有数十亿行,每次reduce调用每个键可以处理数千(如果不是数百万)个值。