Asynchronous 异步写入文件
我一直在尝试创建一个服务器进程,该进程从客户端进程异步接收输入文件路径和输出路径。服务器执行一些依赖于数据库的转换,但是为了简单起见,让我们假设它只是将所有内容放在大写形式。以下是服务器的一个玩具示例:Asynchronous 异步写入文件,asynchronous,python-3.5,python-asyncio,python-aiofiles,Asynchronous,Python 3.5,Python Asyncio,Python Aiofiles,我一直在尝试创建一个服务器进程,该进程从客户端进程异步接收输入文件路径和输出路径。服务器执行一些依赖于数据库的转换,但是为了简单起见,让我们假设它只是将所有内容放在大写形式。以下是服务器的一个玩具示例: import asyncio import aiofiles as aiof import logging import sys ADDRESS = ("localhost", 10000) logging.basicConfig(level=logging.DEBUG,
import asyncio
import aiofiles as aiof
import logging
import sys
ADDRESS = ("localhost", 10000)
logging.basicConfig(level=logging.DEBUG,
format="%(name)s: %(message)s",
stream=sys.stderr)
log = logging.getLogger("main")
loop = asyncio.get_event_loop()
async def server(reader, writer):
log = logging.getLogger("process at {}:{}".format(*ADDRESS))
paths = await reader.read()
in_fp, out_fp = paths.splitlines()
log.debug("connection accepted")
log.debug("processing file {!r}, writing output to {!r}".format(in_fp, out_fp))
async with aiof.open(in_fp, loop=loop) as inp, aiof.open(out_fp, "w", loop=loop) as out:
async for line in inp:
out.write(line.upper())
out.flush()
writer.write(b"done")
await writer.drain()
log.debug("closing")
writer.close()
return
factory = asyncio.start_server(server, *ADDRESS)
server = loop.run_until_complete(factory)
log.debug("starting up on {} port {}".format(*ADDRESS))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
log.debug("closing server")
server.close()
loop.run_until_complete(server.wait_closed())
log.debug("closing event loop")
loop.close()
客户:
import asyncio
import logging
import sys
import random
ADDRESS = ("localhost", 10000)
MESSAGES = ["/path/to/a/big/file.txt\n",
"/output/file_{}.txt\n".format(random.randint(0, 99999))]
logging.basicConfig(level=logging.DEBUG,
format="%(name)s: %(message)s",
stream=sys.stderr)
log = logging.getLogger("main")
loop = asyncio.get_event_loop()
async def client(address, messages):
log = logging.getLogger("client")
log.debug("connecting to {} port {}".format(*address))
reader, writer = await asyncio.open_connection(*address)
writer.writelines([bytes(line, "utf8") for line in messages])
if writer.can_write_eof():
writer.write_eof()
await writer.drain()
log.debug("waiting for response")
response = await reader.read()
log.debug("received {!r}".format(response))
writer.close()
return
try:
loop.run_until_complete(client(ADDRESS, MESSAGES))
finally:
log.debug("closing event loop")
loop.close()
我一次激活了服务器和几个客户端。服务器的日志:
asyncio: Using selector: KqueueSelector
main: starting up on localhost port 10000
process at localhost:10000: connection accepted
process at localhost:10000: processing file b'/path/to/a/big/file.txt', writing output to b'/output/file_79609.txt'
process at localhost:10000: connection accepted
process at localhost:10000: processing file b'/path/to/a/big/file.txt', writing output to b'/output/file_68917.txt'
process at localhost:10000: connection accepted
process at localhost:10000: processing file b'/path/to/a/big/file.txt', writing output to b'/output/file_2439.txt'
process at localhost:10000: closing
process at localhost:10000: closing
process at localhost:10000: closing
所有客户端都打印此文件:
asyncio: Using selector: KqueueSelector
client: connecting to localhost port 10000
client: waiting for response
client: received b'done'
main: closing event loop
输出文件已创建,但仍为空。我相信他们没有被冲。我能修好它吗 在
out.write()和out.flush()之前,您缺少一个wait
:
但是,我强烈建议尝试跳过所有文件,使用常规的、同步的磁盘I/O,并为网络活动保持异步:
with open(file, "w") as out: # regular file I/O
async for s in network_request(): # asyncio for slow network work. measure it!
out.write(s) # should be really quick, measure it!
使用同步文件IO有点破坏了异步服务器的整体功能。它不是在客户机连接到来时处理它们,而是将它们排列在队列中,并且只在前一项处理完成后接收下一项。这没关系,如果客户端没有达到超时,它可以。除非I/O非常快。缓存和SSD磁盘可能会产生很好的效果,而当前的Aiofile实现甚至可能会导致速度减慢(使用线程延迟io而不是事件)
with open(file, "w") as out: # regular file I/O
async for s in network_request(): # asyncio for slow network work. measure it!
out.write(s) # should be really quick, measure it!