Python Queue.asyncio ValueError:task_done（）调用次数太多-编码错误或检测到错误？

我实现了一段代码，从一个队列中获取一个元素，并从队列列表中将相同的对象放入每个队列中。问题是，当我运行一个特定的测试时，我得到了一个

ValueError:task_done（）调用次数过多的异常。此错误发生在测试代码中，而不是正在测试的代码中

我正在使用
asyncio.Queue
并使用协同程序进行编程。
我匹配了每个
队列。用一个
队列获得
。任务完成
调用。
我正在用pytest测试代码

我正在使用以下LIB：


Python 3.7
pytest==3.10.0
pytest asyncio==0.9.0

我有两个文件：
middleware.py
包含我的类实现和
test\u middleware.py
实现pytest测试

文件
middlware.py
：

import asyncio

class DistributorMiddleware:

    def __init__(self, in_queue, out_list_queue):
        self._in = in_queue
        self._out = out_list_queue

    async def distribute(self):

        while True:
            ele = await self._in.get()
            count=0
            for queue in self._out:
                await queue.put(ele)
                count+=1
                print(f'inserted ele in {count}')
            queue.task_done()
            if ele == None:
                break
        for queue in self._out:
            await queue.join()

import pytest
import asyncio                
from asyncio import Queue
from middleware import DistributorMiddleware
import random
import os


@pytest.mark.asyncio                                                                                     
async def test_distribution(request, event_loop):                                                        
    q_list = [ Queue() for _ in range(10) ]                                                              
    _in = Queue()
    distrib = DistributorMiddleware(_in, q_list)                                                         
    event_loop.create_task(distrib.distribute())                                                         
    num_ele = random.randint(1, 10)
    ele_set = set()
    for _ in range(num_ele):                                                                             
        ele = os.urandom(4)                                                                              
        ele_set.add(ele)
        await _in.put(ele)
    await _in.put(None)                                                                                  
    await asyncio.sleep(1)                                                                               

    for i,q in enumerate(q_list):
        assert q.qsize() == num_ele + 1
        c_set = ele_set.copy()
        count= 0
        while True:
            e = await q.get()
            count+=1
            print(f'Queue {i}: element: "{e}" number {count} extracted of {q.qsize()}!')
            q.task_done()
            if e == None:
                break
            assert e in c_set
            c_set.remove(e)

文件
test\u middleware.py
：

import asyncio

class DistributorMiddleware:

    def __init__(self, in_queue, out_list_queue):
        self._in = in_queue
        self._out = out_list_queue

    async def distribute(self):

        while True:
            ele = await self._in.get()
            count=0
            for queue in self._out:
                await queue.put(ele)
                count+=1
                print(f'inserted ele in {count}')
            queue.task_done()
            if ele == None:
                break
        for queue in self._out:
            await queue.join()

import pytest
import asyncio                
from asyncio import Queue
from middleware import DistributorMiddleware
import random
import os


@pytest.mark.asyncio                                                                                     
async def test_distribution(request, event_loop):                                                        
    q_list = [ Queue() for _ in range(10) ]                                                              
    _in = Queue()
    distrib = DistributorMiddleware(_in, q_list)                                                         
    event_loop.create_task(distrib.distribute())                                                         
    num_ele = random.randint(1, 10)
    ele_set = set()
    for _ in range(num_ele):                                                                             
        ele = os.urandom(4)                                                                              
        ele_set.add(ele)
        await _in.put(ele)
    await _in.put(None)                                                                                  
    await asyncio.sleep(1)                                                                               

    for i,q in enumerate(q_list):
        assert q.qsize() == num_ele + 1
        c_set = ele_set.copy()
        count= 0
        while True:
            e = await q.get()
            count+=1
            print(f'Queue {i}: element: "{e}" number {count} extracted of {q.qsize()}!')
            q.task_done()
            if e == None:
                break
            assert e in c_set
            c_set.remove(e)

在测试中，中间件应该从输入队列中获取元素，并将它们从列表中放入10个队列中。它正确地完成了工作

测试代码从10个队列中的每个队列中获取所有元素，并检查它们是否存在于原始队列中。对于前9个队列，一切正常，没有错误，但是当测试尝试从第十个列表中获取第一个元素时，会引发
ValueError
：

request = <FixtureRequest for <Function 'test_distribution'>>, event_loop = <_UnixSelectorEventLoop running=False closed=False debug=False>

    @pytest.mark.asyncio
    async def test_distribution(request, event_loop):
        q_list = [ Queue() for _ in range(10) ]
        _in = Queue()
        distrib = DistributorMiddleware(_in, q_list)
        event_loop.create_task(distrib.distribute())
        num_ele = random.randint(1, 10)
        ele_set = set()
        for _ in range(num_ele):
            ele = os.urandom(4)
            ele_set.add(ele)
            await _in.put(ele)
        await _in.put(None)
        await asyncio.sleep(1)

        for i,q in enumerate(q_list):
            assert q.qsize() == num_ele + 1
            c_set = ele_set.copy()
            count= 0
            while True:
                e = await q.get()
                count+=1
                print(f'Queue {i}: element: "{e}" number {count} extracted of {q.qsize()}!')
>               q.task_done()

test_middlewares.py:34: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Queue at 0x7f7af5b9d828 maxsize=0 _queue=[b'\x15\xad\t\xaf', b'\x8b\xa2M=', None]>

    def task_done(self):
        """Indicate that a formerly enqueued task is complete.

        Used by queue consumers. For each get() used to fetch a task,
        a subsequent call to task_done() tells the queue that the processing
        on the task is complete.

        If a join() is currently blocking, it will resume when all items have
        been processed (meaning that a task_done() call was received for every
        item that had been put() into the queue).

        Raises ValueError if called more times than there were items placed in
        the queue.
        """
        if self._unfinished_tasks <= 0:
>           raise ValueError('task_done() called too many times')
E           ValueError: task_done() called too many times

/usr/lib/python3.7/asyncio/queues.py:202: ValueError

这样做，我可以看到，即使有许多
ValueError
被抛出，元素仍会不断从队列中检索。测试成功：

platform linux -- Python 3.7.1, pytest-3.10.0, py-1.7.0, pluggy-0.8.0
rootdir: /tmp/stack, inifile:
plugins: asyncio-0.9.0
collected 1 item                                                                                                                                                                                                  

test_middlewares.py .                                                                                                                                                                                       [100%]

============================================================================================ 1 passed in 1.04 seconds =============================================================================================

为了确保测试正在使用所有列表中的所有元素，我在测试结束时添加了一个错误断言：

             assert e in c_set
             c_set.remove(e)

+    assert False == True
+

结果输出显示从所有列表检索所有元素，但在最后一个队列上完成的每个任务都会生成一个
ValueError

Queue 7: element: "b'\x9b\xf8m\x02'" number 1 extracted of 3!
Queue 7: element: "b'\x15\xad\t\xaf'" number 2 extracted of 2!
Queue 7: element: "b'\x8b\xa2M='" number 3 extracted of 1!
Queue 7: element: "None" number 4 extracted of 0!
Queue 8: element: "b'\x9b\xf8m\x02'" number 1 extracted of 3!
Queue 8: element: "b'\x15\xad\t\xaf'" number 2 extracted of 2!
Queue 8: element: "b'\x8b\xa2M='" number 3 extracted of 1!
Queue 8: element: "None" number 4 extracted of 0!
Queue 9: element: "b'\x9b\xf8m\x02'" number 1 extracted of 3!
============================================================================================ 1 failed in 1.06 seconds ==

问题是，我是否遗漏了什么，代码中有错误，或者我发现了一个bug？
您的代码中有错误。实际上，
queue.task_done（）

但是您的中间件类正在它刚刚使用的队列上调用它。
.put（）
，用于
self.\u out
列表中的最后一个队列；从DistributorMiddleware.distribute（）中删除队列.task_done（）
调用：

当删除该行时，测试通过

您在测试中看到引发异常的原因是，只有到那时队列才知道
task\u done（）
调用得太频繁。
DistributorMiddleware.distribute（）
中的
queue.task_done（）
调用将未完成的任务计数器递减1，但只有当该计数器降至零以下时，才能检测到异常。只有在
test_distribution（）
中将最后一个任务从队列中取出时，才能到达该点，此时未完成的任务计数器至少提前一步到达0

也许这是为了调用
self.\u in.task\u done（）
？您只是在
循环中从该队列中获取了一个元素：

async def distribute(self):

    while True:
        ele = await self._in.get()
        # getting an element from self._in
        count=0
        for queue in self._out:
            await queue.put(ele)
            count+=1
            print(f'inserted ele in {count}')
        self._in.task_done()
        # done with ele, so decrement the self._in unfinished tasks counter

旁注：不要使用
==None
，它是一个单例，因此测试它的python方法是使用
是None
。这正是问题所在。当我想在
中使用
self.\u时，我使用了
queue
。非常感谢。