Python 使用aiohttp的异步调用不可伸缩

我正在尝试使用aiohttp进行http调用。我能获得的最大吞吐量是1K请求/秒。服务器没有进行任何处理，只是立即确认请求。请求的RTT大约需要7毫秒

python进程的概要分析显示了类似的内容。我尝试异步发出2K请求，但花了2秒多的时间才完成

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     3720    0.905    0.000    0.905    0.000 {method 'poll' of 'select.epoll' objects}
     2000    0.142    0.000    0.158    0.000 {method 'feed_data' of 'aiohttp._http_parser.HttpParser' objects}
    11500    0.081    0.000    1.958    0.000 grpc.py:113(request)
     9500    0.080    0.000    1.760    0.000 /home/me/venv2/lib/python3.6/site-packages/aiohttp/client.py:180(_request)
     3720    0.069    0.000    3.634    0.001 /usr/lib/python3.6/asyncio/base_events.py:1330(_run_once)
     4000    0.064    0.000    0.064    0.000 {method 'send' of '_socket.socket' objects}
     4000    0.063    0.000    0.067    0.000 /home/me/venv2/lib/python3.6/site-packages/aiohttp/client.py:635(_prepare_headers)
     6000    0.053    0.000    0.215    0.000 /home/me/venv2/lib/python3.6/site-packages/idna/core.py:231(check_label)
    54000    0.053    0.000    0.104    0.000 /home/me/venv2/lib/python3.6/site-packages/idna/intranges.py:38(intranges_contain)
    26509    0.050    0.000    2.608    0.000 /usr/lib/python3.6/asyncio/events.py:125(_run)
     5500    0.045    0.000    0.364    0.000 /home/me/venv2/lib/python3.6/site-packages/aiohttp/connector.py:723(_create_connection)
182134/182126    0.045    0.000    0.067    0.000 {built-in method builtins.isinstance}
     1500    0.042    0.000    0.042    0.000 {method 'connect' of '_socket.socket' objects}
     2000    0.040    0.000    0.178    0.000 /home/me/venv2/lib/python3.6/site-packages/aiohttp/client_reqrep.py:477(send)
      382    0.040    0.000    0.040    0.000 {built-in method marshal.loads}
     9500    0.034    0.000    2.083    0.000 grpc.py:65(grpc)

如何提高吞吐量

这是我的实现

def _get_session_headers() -> dict:
    """Get session headers."""
    return {
        // Some key value for headers
    }

conn = aiohttp.TCPConnector(limit=1000)
_session = aiohttp.ClientSession(connector=conn,headers=_get_session_headers())

def _get_uri(path: str) -> str:
    """Get URI."""
    return urljoin("http://targethost:8080/", path)    

async def make_request(
               path: str,
               grequest: Message,
               index, int
               headers: dict = {},
               **kwargs: dict) -> None:

    async with request('POST', _get_uri(path), data=grequest, headers=headers, **kwargs) as response:
    print("Before " + str(index))
    buffer = await response.read()
    print("After " + str(index))

@async_contextmanager
async def request(method: str, path: str, **kwargs: dict) -> aiohttp.ClientResponse:
    async with _session.request(method, _get_uri(path), **kwargs) as res:
        yield res




async def test(req, index):
    await make_request('fetch', req, index)




grequest = initialize_data() // Some sample data to be sent 

start = time.time()
loop = asyncio.get_event_loop()
tasks = []

start = time.time()
for i in range(1000):
    tasks.append(asyncio.ensure_future(test(grequest, i)))


print("Starting the loop")
loop.run_until_complete(asyncio.wait(tasks))
end = time.time()

task=loop.create_task(_session.close())
loop.run_until_complete(task)
loop.close()
print(end-start)

这是一秒钟的打印。所以我的rps只有1k/秒

Before buffer 896
After buffer 896
Before buffer 976
After buffer 976
Before buffer 880
After buffer 880
Before buffer 898
After buffer 898
Before buffer 882
After buffer 882
Before buffer 978
After buffer 978
Before buffer 900
After buffer 900
Before buffer 884
After buffer 884
Before buffer 982
After buffer 982
Before buffer 902
After buffer 902
Before buffer 886
After buffer 886
Before buffer 904
After buffer 904
Before buffer 988
After buffer 988
Before buffer 888
After buffer 888
Before buffer 990
After buffer 990
Before buffer 994
After buffer 994

看起来他们是连续执行的。但是，如果我替换

buffer=await response.read（）

与

buffer=await asyncio.sleep（1）

混合打印

我还验证了本教程脚本的情况

import asyncio
from aiohttp import ClientSession

async def fetch(url, session, index):
    async with session.get(url) as response:
        print("Before " + str(index))
        buffer = await response.read()
        print("After " + str(index))

async def run(r):
    url = "http://google.com"
    tasks = []

    # Fetch all responses within one Client session,
    # keep connection alive for all requests.
    async with ClientSession() as session:
        for i in range(r):
            task = asyncio.ensure_future(fetch(url, session, i))
            tasks.append(task)

        responses = await asyncio.gather(*tasks)
        # you now have all response bodies in this variable
        print(responses)

def print_responses(result):
    print(result)

loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run(4))
loop.run_until_complete(future)

对于10000个请求，这是我从分析器中得到的

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    35901   10.792    0.000   10.792    0.000 {method 'poll' of 'select.epoll' objects}
    20000    0.607    0.000    0.607    0.000 /home/me/venv2/lib/python3.6/site-packages/aiohttp/client.py:635(_prepare_headers)
    35901    0.397    0.000   22.202    0.001 /usr/lib/python3.6/asyncio/base_events.py:1330(_run_once)
    33500    0.393    0.000    7.911    0.000 /home/me/venv2/lib/python3.6/site-packages/aiohttp/client.py:180(_request)
    63007    0.345    0.000    0.430    0.000 /usr/lib/python3.6/asyncio/base_events.py:594(_call_soon)
    20000    0.341    0.000    0.341    0.000 {method 'send' of '_socket.socket' objects}
    10000    0.311    0.000    0.311    0.000 {method 'copy' of 'multidict._multidict.CIMultiDict' objects}
   270000    0.282    0.000    0.557    0.000 /home/me/venv2/lib/python3.6/site-packages/idna/intranges.py:38(intranges_contain)
    30000    0.263    0.000    1.103    0.000 /home/me/venv2/lib/python3.6/site-packages/idna/core.py:231(check_label)
    10000    0.246    0.000    0.355    0.000 {method 'feed_data' of 'aiohttp._http_parser.HttpParser' objects}

---

重构您的协同程序，使其正常工作？从个人资料来看，你认为瓶颈在哪里？没有你的实施，怎么会有人知道问题出在哪里？@wwii。用我的实现更新了帖子。您能提供一些建议吗？将范围（1000）内i的

更改为范围（10000）内i的
，并检查性能。@Sraw。2000年为2.5秒，3000年为5秒，10000年为19秒seconds@Sraw2000年为2.5秒，3000年为5秒，10000年为19秒