Python 3.x aiohttp异步请求的任务异常

我正在尝试使用asyncio和aiohttp加速对web服务的多个get请求

为此，我在函数中使用psycopg2 module.fetchmany（）从postgresql数据库获取数据，并构建一个包含100条记录的字典，以字典URL列表的形式发送给名为batch（）的异步函数。分批处理

我在batch（）函数中遇到的问题是，尽管脚本继续运行并且没有失败，但有些请求正在记录下面的消息，但我无法捕获并记录此异常，以便稍后重新处理它们

Task exception was never retrieved
future: <Task finished coro=<batch.<locals>.fetch() done, defined at C:/PythonProjects/bindings/batch_fetch.py:34> exception=ClientOSError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)>
Traceback (most recent call last):
  File "C:/PythonProjects/bindings/batch_fetch.py", line 36, in fetch
    async with session.get(url) as resp:
  File "C:\Miniconda3\lib\site-packages\aiohttp\client.py", line 1005, in __aenter__
    self._resp = await self._coro
  File "C:\Miniconda3\lib\site-packages\aiohttp\client.py", line 497, in _request
    await resp.start(conn)
  File "C:\Miniconda3\lib\site-packages\aiohttp\client_reqrep.py", line 844, in start
    message, payload = await self._protocol.read()  # type: ignore  # noqa
  File "C:\Miniconda3\lib\site-packages\aiohttp\streams.py", line 588, in read
    await self._waiter
aiohttp.client_exceptions.ClientOSError: [WinError 10054] An existing connection was forcibly closed by the remote host
Task exception was never retrieved
future: <Task finished coro=<batch.<locals>.fetch() done, defined at C:/PythonProjects/bindings/batch_fetch.py:34> exception=ClientConnectorError(10060, "Connect call failed ('xx.xxx.xx.xxx', 80)")>
Traceback (most recent call last):
  File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 924, in _wrap_create_connection
    await self._loop.create_connection(*args, **kwargs))
  File "C:\Miniconda3\lib\asyncio\base_events.py", line 778, in create_connection
    raise exceptions[0]
  File "C:\Miniconda3\lib\asyncio\base_events.py", line 765, in create_connection
    yield from self.sock_connect(sock, address)
  File "C:\Miniconda3\lib\asyncio\selector_events.py", line 450, in sock_connect
    return (yield from fut)
  File "C:\Miniconda3\lib\asyncio\selector_events.py", line 480, in _sock_connect_cb
    raise OSError(err, 'Connect call failed %s' % (address,))
TimeoutError: [Errno 10060] Connect call failed ('xx.xxx.xx.xxx', 80)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:/PythonProjects/bindings/batch_fetch.py", line 36, in fetch
    async with session.get(url) as resp:
  File "C:\Miniconda3\lib\site-packages\aiohttp\client.py", line 1005, in __aenter__
    self._resp = await self._coro
  File "C:\Miniconda3\lib\site-packages\aiohttp\client.py", line 476, in _request
    timeout=real_timeout
  File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 522, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 854, in _create_connection
    req, traces, timeout)
  File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 992, in _create_direct_connection
    raise last_exc
  File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 974, in _create_direct_connection
    req=req, client_error=client_error)
  File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 931, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host cms-uat.cme.in.here.com:80 ssl:None [Connect call failed ('xx.xxx.xx.xxx', 80)]

从未检索到任务异常

当您创建某个任务时，您会看到此警告，该任务完成时出现异常，但您从未显式检索（等待）其结果。这里是相关的

我打赌在你的情况下，问题在于线路

loop.run_until_complete(asyncio.wait(tasks))

asyncio.wait（）

只要在所有

任务完成后等待即可。它不区分正常完成的任务和异常完成的任务，它只是阻塞，直到所有任务都完成。在这种情况下，您的任务是从已完成的任务中检索异常，而以下部分不会帮助您，因为
asyncio.wait（）
永远不会引发错误：

try:
    loop.run_until_complete(asyncio.wait(tasks))
except Exception:
    print('...')  # You will probably NEVER see this message

如果您想在其中一个任务发生错误时立即捕获错误，我可以建议您使用
asyncio.gather（）
。它将引发第一个发生的异常。但是请注意，如果您想让挂起的任务正常关闭，您的工作就是处理这些任务

try:
    loop.run_until_complete(asyncio.wait(tasks))
except Exception:
    print('...')  # You will probably NEVER see this message