Python 异步并联发电机

我有一个python脚本，它懒洋洋地收集数据，创建训练样本，并将其传递给我的ML模型进行学习。目前，我正在使用标准python生成器生成数据，据我所知，它是同步的。我正在寻找一种智能、干净的方法，使我的生成器真正异步，因此当我使用它作为迭代器时，在我取出最后一个样本后，下一个数据样本的处理将立即开始。考虑下面的例子：

def asyncgen():
    for i in range(5):
        print("I want this part to work asynchronously :(")
        i = 0;
        while(i<1e8):
            i+=1
        yield "Hi"

a = asyncgen()
for w in a:
    print(w)
    i = 0
    while (i < 1e8):
        i += 1

我如何使我的生成器在收到Hi后立即在不同的进程下开始异步处理内容？当前，处理仅在for循环调用next之后开始

---

我一直在研究政治公众人物525，但他们似乎只是同时工作，而不是并行该死的吉尔！。在Python中，这是一种很好的、最好是本机的方法。

绕过GIL的唯一方法是使用

---

因此，在上面的代码中，asyncgen作为并行进程独立运行了5次。然后在程序结束前将它们连接起来。保持一个列表p只是一个说明。

套接字也是解决这个问题的一个很好的方法。 基本上，与其让一个程序带有线程或多个进程，不如让一个外部程序执行您从外部连接到的异步程序

了解有关套接字的更多信息，请访问

下面是一个非常完整的python 2.7示例：

import argparse
import logging
import socket
import time
import random

description = """
This program sends data across a socket
"""
arg_parser = argparse.ArgumentParser(description=description)
arg_parser.add_argument('--AgentIP', '-i', action='store'    
    , default='localhost' , type=str
    , help="The IP address of this Server that client will connect to."
    )
arg_parser.add_argument('--AgentPort', '-p', action='store'     
    , default='13000' , type=int
    , help="The port number of the socket that client will connect to."
    )
arg_parser.add_argument('--log', action='store'     
    , default='INFO' , type=str
    , help="Log level"
    , choices=['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
    )
args = arg_parser.parse_args()

# use Logger for great justice and heavenly output
logging.basicConfig(level=getattr(logging, args.log, 'CRITICAL'))
log = logging.getLogger(name='SERVER')

### vvv YOUR CODE vvv ###
def asyncgen(log):
    for i in range(5):
        log.debug("I want this part to work asynchronously :(") 
        time.sleep(random.random())
        yield "Hi"
### ^^^ YOUR CODE ^^^ ###

def make_a_connection(server, log):
    # Accept outside connections
    (client_socket, address) = server.accept()
    log.info("Got a connection : {}:{}".format(client_socket,address))

    for value in asyncgen(log):
        client_socket.send(value)
        log.info("SEND:{}".format(value))

    client_socket.close()


def main(args, log):
    server   = socket.socket( socket.AF_INET , socket.SOCK_STREAM )
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

    hostname = socket.gethostbyname(args.AgentIP)
    port     = args.AgentPort
    address  = (hostname, port)

    server.bind(address)
    server.listen(1)
    log.info("Server started on {}:{}".format(hostname, port))

    try:
        while True:
          make_a_connection(server, log)

    except (KeyboardInterrupt, SystemExit):
       server.close()
       log.info("Server connections closed")


if __name__=='__main__':
    main(args, log)

---

你读过吗？@lalengua是的，我读过类似的博客和文档。线程、greenlet和asyncio都在单个进程上运行。虽然我的解析前端正在做一些io的事情，但它也在做很多处理，我不想在同一个过程中减慢我的计算速度。我的机器上有足够的内核来并行运行。我知道这个选项，我只是希望有更干净的东西：。。。所以我想我将要编写一个包装类，它是iterable，接收生成器并提前调用几个项，将它们准备到队列中，并在调用其uu next uu_u方法时逐个生成它们，而一些后台工作人员将在需要时填充队列。很遗憾，无法从多处理中逃脱。发电机看起来很优雅，请使用netcat查看此运行情况。请尝试nc localhost 13000

import argparse
import logging
import socket
import time
import random

description = """
This program sends data across a socket
"""
arg_parser = argparse.ArgumentParser(description=description)
arg_parser.add_argument('--AgentIP', '-i', action='store'    
    , default='localhost' , type=str
    , help="The IP address of this Server that client will connect to."
    )
arg_parser.add_argument('--AgentPort', '-p', action='store'     
    , default='13000' , type=int
    , help="The port number of the socket that client will connect to."
    )
arg_parser.add_argument('--log', action='store'     
    , default='INFO' , type=str
    , help="Log level"
    , choices=['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
    )
args = arg_parser.parse_args()

# use Logger for great justice and heavenly output
logging.basicConfig(level=getattr(logging, args.log, 'CRITICAL'))
log = logging.getLogger(name='SERVER')

### vvv YOUR CODE vvv ###
def asyncgen(log):
    for i in range(5):
        log.debug("I want this part to work asynchronously :(") 
        time.sleep(random.random())
        yield "Hi"
### ^^^ YOUR CODE ^^^ ###

def make_a_connection(server, log):
    # Accept outside connections
    (client_socket, address) = server.accept()
    log.info("Got a connection : {}:{}".format(client_socket,address))

    for value in asyncgen(log):
        client_socket.send(value)
        log.info("SEND:{}".format(value))

    client_socket.close()


def main(args, log):
    server   = socket.socket( socket.AF_INET , socket.SOCK_STREAM )
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

    hostname = socket.gethostbyname(args.AgentIP)
    port     = args.AgentPort
    address  = (hostname, port)

    server.bind(address)
    server.listen(1)
    log.info("Server started on {}:{}".format(hostname, port))

    try:
        while True:
          make_a_connection(server, log)

    except (KeyboardInterrupt, SystemExit):
       server.close()
       log.info("Server connections closed")


if __name__=='__main__':
    main(args, log)