Warning: file_get_contents(/data/phpspider/zhask/data//catemap/0/hadoop/6.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
将mongoDB的所有集合作为输入传递给mapreduce hadoop_Mongodb_Hadoop_Mapreduce_Mongo Collection - Fatal编程技术网

将mongoDB的所有集合作为输入传递给mapreduce hadoop

将mongoDB的所有集合作为输入传递给mapreduce hadoop,mongodb,hadoop,mapreduce,mongo-collection,Mongodb,Hadoop,Mapreduce,Mongo Collection,我需要将MongoDB中数据库的所有集合作为输入传递给Hadoop MR作业。有一种方法允许多个输入: MultiCollectionSplitBuilder mcsb = new MultiCollectionSplitBuilder(); mcsb.add(new MongoURI("mongodb://localhost:27017/mongo_hadoop.yield_historical.in"), (MongoURI)null, // authuri

我需要将MongoDB中数据库的所有集合作为输入传递给Hadoop MR作业。有一种方法允许多个输入:

MultiCollectionSplitBuilder mcsb = new MultiCollectionSplitBuilder();
mcsb.add(new MongoURI("mongodb://localhost:27017/mongo_hadoop.yield_historical.in"),
        (MongoURI)null, // authuri
        true, // notimeout
        (DBObject)null, // fields
        (DBObject)null, // sort
        (DBObject)null, // query
        false,
        MultiMongoCollectionSplitter.class)
.add(new MongoURI("mongodb://localhost:27017/mongo_hadoop.yield_historical.in"),
        (MongoURI)null, // authuri
        true, // notimeout
        (DBObject)null, // fields
        (DBObject)null, // sort
        new BasicDBObject("_id", new BasicDBObject("$gt", new Date(883440000000L))),
        false, // range query
        MultiMongoCollectionSplitter.class);
但我的数据库中有大约10个集合。上述方法仅允许2个集合参数。 我所需要做的就是用mapper Methods单独获取所有集合。我的减速机对他们来说都是一样的


感谢您的帮助

您可以继续添加到MultiCollectionSplitBuilder

    MultiCollectionSplitBuilder mcsb = new MultiCollectionSplitBuilder();
    mcsb
            .add(new MongoURI("mongodb://localhost:27017/mongo_hadoop.yield_historical.in"),
                    (MongoURI) null, // authuri
                    true, // notimeout
                    (DBObject) null, // fields
                    (DBObject) null, // sort
                    (DBObject) null, // query
                    false,
                    MultiMongoCollectionSplitter.class
            )
            .add(new MongoURI("mongodb://localhost:27017/mongo_hadoop.yield_historical.in"),
                    (MongoURI) null, // authuri
                    true, // notimeout
                    (DBObject) null, // fields
                    (DBObject) null, // sort
                    new BasicDBObject("_id", new BasicDBObject("$gt", new Date(883440000000L))),
                    false, // range query
                    MultiMongoCollectionSplitter.class
            )
            .add(new MongoURI("mongodb://localhost:27017/mongo_hadoop.yield_historical.in"),
                    (MongoURI) null, // authuri
                    true, // notimeout
                    (DBObject) null, // fields
                    (DBObject) null, // sort
                    new BasicDBObject("_id", new BasicDBObject("$gt", new Date(883440000000L))),
                    false, // range query
                    MultiMongoCollectionSplitter.class
            )
    ;

谢谢你,艾伦。但是我的程序无法使用此选项连接。仅当我使用MongoConfigUtil.setInputURI()只传递一个集合时,它才起作用