Warning: file_get_contents(/data/phpspider/zhask/data//catemap/9/ssl/3.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Parallel processing pymongo-执行并行查询_Parallel Processing_Pymongo_Joblib - Fatal编程技术网

Parallel processing pymongo-执行并行查询

Parallel processing pymongo-执行并行查询,parallel-processing,pymongo,joblib,Parallel Processing,Pymongo,Joblib,下面是我想并行化的伪代码,但不知道从哪里开始 from pymongo import MongoClient client = MongoClient('localhost', 27017) db = client['myDB'] collection = db.myCollection test_list = ['foo', 'bar'] result_list = list() for el in test_list: result_list.append(collection

下面是我想并行化的伪代码,但不知道从哪里开始

from pymongo import MongoClient
client = MongoClient('localhost', 27017)
db = client['myDB']
collection = db.myCollection

test_list = ['foo', 'bar']
result_list = list()

for el in test_list:
     result_list.append(collection.distinct('attrib',{'version': el}))

我知道如何使用joblib创建并行循环,但我不确定如何并行查询MongoDB,我应该创建多个客户机还是集合?如果我简单地使用joblib重新编写上述代码,而不关心MongoDB,那么上述代码是否有效?

您可以在单独的线程中运行请求:

from multiprocessing.dummy import Pool as ThreadPool 

from pymongo import MongoClient


client = MongoClient('localhost', 27017)
db = client['myDB']
collection = db.myCollection

thread_pool_size = 4
pool = ThreadPool(thread_pool_size) 


def my_function(el):
    return collection.distinct('attrib', {'version': el}))


test_list = ['foo', 'bar']
result_list = pool.map(my_function, test_list)