Mongodb 使用asyncio运行并发mongoengine查询

Mongodb 使用asyncio运行并发mongoengine查询,mongodb,python-3.x,pymongo,mongoengine,python-asyncio,Mongodb,Python 3.x,Pymongo,Mongoengine,Python Asyncio,我有4个函数,基本上构建查询并执行它们。我希望使用asyncio使它们同时运行。我对asyncio的实现似乎是正确的,因为非mongodb任务可以正常运行(例如asyncio.sleep()。代码如下: loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) tasks = [ service.async_get_associate_opportunity_count_by_user(me, cri

我有4个函数,基本上构建查询并执行它们。我希望使用asyncio使它们同时运行。我对asyncio的实现似乎是正确的,因为非mongodb任务可以正常运行(例如asyncio.sleep()。代码如下:

loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
tasks = [
                service.async_get_associate_opportunity_count_by_user(me, criteria),
                service.get_new_associate_opportunity_count_by_user(me, criteria),
                service.async_get_associate_favorites_count(me, criteria=dict()),
                service.get_group_matched_opportunities_count_by_user(me, criteria)
            ]

available, new, favorites, group_matched = loop.run_until_complete(asyncio.gather(*tasks))

stats['opportunities']['available'] = available
stats['opportunities']['new'] = new
stats['opportunities']['favorites'] = favorites
stats['opportunities']['group_matched'] = group_matched

loop.close()


# functions written in other file

@asyncio.coroutine
def async_get_ass(self, user, criteria=None, **kwargs):
    start_time = time.time()
    query = **query that gets built from some other functions**

    opportunities = Opportunity.objects(query).count()
    run_time = time.time() - start_time
    print("runtime of available: {}".format(run_time))
    yield from asyncio.sleep(2)
    return opportunities

@asyncio.coroutine
def get_new_associate_opportunity_count_by_user(self, user, criteria=None, **kwargs):
    start_time = time.time()
    query = **query that gets built from some other functions**
    opportunities = Opportunity.objects(query).count()
    run_time = time.time() - start_time
    print("runtime of new: {}".format(run_time))
    yield from asyncio.sleep(2)
    return opportunities

@asyncio.coroutine
def async_get_associate_favorites_count(self, user, criteria={}, **kwargs):
    start_time = time.time()
    query = **query that gets built from some other functions**
    favorites = Opportunity.objects(query).count()
    run_time = time.time() - start_time
    print("runtime of favorites: {}".format(run_time))
    yield from asyncio.sleep(2)
    return favorites

@asyncio.coroutine
def get_group_matched_opportunities_count_by_user(self, user, criteria=None, **kwargs):
    start_time = time.time()
    query = **query that gets built from some other functions**
    opportunities = Opportunity.objects(query).count()
    run_time = time.time() - start_time
    print("runtime of group matched: {}".format(run_time))
    yield from asyncio.sleep(2)
    return opportunities
asyncio.sleep(2)的
结果只是表明函数是异步运行的。以下是终端上的输出:

匹配组的运行时:0.11431598663330078 收藏夹的运行时间:0.0029871463775634766 时间戳函数运行时间:0.0004897117614746094 新版本的运行时间:0.15225648880004883 可用的运行时间:0.13006806373596191 总运行时间:2403.2700061798096 根据我的理解,除了由于sleep函数而增加到总运行时间的2000ms之外,它不应该超过155-160ms,因为所有函数中的最大运行时间就是这个值


我目前正在研究motorengine(mongoengine 0.9.0的一个端口),它显然支持异步mongodb查询,但我认为我无法使用它,因为我的模型是使用mongoengine定义的。有解决此问题的方法吗?

查询没有并行运行的原因是,每当您在协同例程中运行
Opportunity.objects(query).count()
时,整个事件循环都会阻塞,因为这些方法正在执行阻塞IO

因此,您需要一个mongodb驱动程序,它可以执行异步/非阻塞IO。您尝试使用
motorengine
的方法是正确的,但据我所知,它是为Tornado异步框架编写的。要使它与
asyncio
一起工作,您必须连接Tornado和
asyncio
。看,关于如何做到这一点

另一个选择是使用,但它没有兼容的ORM,因此您可能需要重写大部分代码