Python多处理，函数的一个参数是迭代器，Get-TypeError

我有这样一个代码：

导入多处理
来自itertools进口产品、imap、ifilter
def测试（it）：
对于其中的x：
打印x
一无所获
mp\u pool=multiprocessing.pool（multiprocessing.cpu\u count（））
it=imap（λx:ifilter（λy:x+y>10，xrange（10）），xrange（10））
结果=mp_pool.map（测试，it）

我收到错误消息：

     File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
        self._target(*self._args, **self._kwargs)
      File "/usr/lib64/python2.7/multiprocessing/pool.py", line 102, in worker
        task = get()
      File "/usr/lib64/python2.7/multiprocessing/queues.py", line 376, in get
        return recv()
        task = get()
      File "/usr/lib64/python2.7/multiprocessing/queues.py", line 376, in get
    TypeError: ifilter expected 2 arguments, got 0
        return recv()

---

多处理不能使用带迭代器参数的函数？谢谢大家!

您的迭代器，

it

，必须生成单个值（每个值可以是“复杂的”，例如元组或列表）。现在我们有：

>>> it
<itertools.imap object at 0x000000000283DB70>
>>> list(it)
[<itertools.ifilter object at 0x000000000283DC50>, <itertools.ifilter object at 0x000000000283DF98>, <itertools.ifilter object at 0x000000000283DBE0>, <itertools.ifilter object at 0x000000000283DF60>, <itertools.ifilter object at 0x000000000283DB00>, <itertools.ifilter object at 0x000000000283DCC0>, <itertools.ifilter object at 0x000000000283DD30>, <itertools.ifilter object at 0x000000000283DDA0>, <itertools.ifilter object at 0x000000000283DE80>, <itertools.ifilter object at 0x000000000284F080>]

按照您的定义，打印的结果是：

[]
[]
['t = 9']
['t = 8', 't = 9']
['t = 7', 't = 8', 't = 9']
['t = 6', 't = 7', 't = 8', 't = 9']
['t = 5', 't = 6', 't = 7', 't = 8', 't = 9']
['t = 4', 't = 5', 't = 6', 't = 7', 't = 8', 't = 9']
['t = 3', 't = 4', 't = 5', 't = 6', 't = 7', 't = 8', 't = 9']
['t = 2', 't = 3', 't = 4', 't = 5', 't = 6', 't = 7', 't = 8', 't = 9']

但是，如果您想充分利用多处理（假设您有足够的处理器），则可以使用

map\u async

，以便一次提交所有作业：

import multiprocessing
from itertools import imap, ifilter
import sys


def test(t):
    return 't = ' + str(t) # return value rather than printing


if __name__ == '__main__': # required for Windows
    mp_pool = multiprocessing.Pool(multiprocessing.cpu_count())
    it = imap(lambda x: ifilter(lambda y: x+y > 10, xrange(10)), xrange(10))
    results = [mp_pool.map_async(test, the_iterator) for the_iterator in it]
    for result in results:
        print result.get()
    mp_pool.close() # needed to ensure all processes terminate
    mp_pool.join() # needed to ensure all processes terminate

或者您可以考虑使用<代码> My.Posi.IMAP，它不同于<代码> MyoPo.MasyAsYNC/<代码>，不首先将迭代参数转换为列表，以确定用于提交作业的最佳<代码> CukStase<代码>值（读取文档，这不是很好），但默认情况下，使用<代码> CukStase<代码>值为1，这通常不适用于非常大的可熔炉：

results = [mp_pool.imap(test, the_iterator) for the_iterator in it]
for result in results:
    print list(result) # to get a comparable printout as when using map_async

更新：使用多处理生成列表

import multiprocessing
from itertools import imap, ifilter
import sys


def test(t):
    return 't = ' + str(t) # return value rather than printing

def generate_lists(x):
    return list(ifilter(lambda y: x+y > 10, xrange(10)))

if __name__ == '__main__': # required for Windows
    mp_pool = multiprocessing.Pool(multiprocessing.cpu_count())
    lists = mp_pool.imap(generate_lists, xrange(10))
    # lists, returned by mp_pool.imap, is an iterable
    # as each element of lists becomes available it is passed to test:
    results = mp_pool.imap(test, lists)
    # as each result becomes available
    for result in results:
        print result
    mp_pool.close() # needed to ensure all processes terminate

印刷品：

t = []
t = []
t = [9]
t = [8, 9]
t = [7, 8, 9]
t = [6, 7, 8, 9]
t = [5, 6, 7, 8, 9]
t = [4, 5, 6, 7, 8, 9]
t = [3, 4, 5, 6, 7, 8, 9]
t = [2, 3, 4, 5, 6, 7, 8, 9]

---

线程可能是相关的。对不起，我的示例代码把你弄糊涂了！我的实际代码是迭代器的每次迭代都会产生另一个迭代器。在我的实际代码中，生成的迭代器产生值将非常耗时，因此我想将生成的迭代器放到一个进程中以产生值。我已经更新了答案。我不确定你的迭代器，

it

，是否产生你期望的结果。我的代码和你的代码之间的区别在于我把迭代器作为函数的参数。在我的实际代码中，迭代器生成值将非常耗时，因此我想将迭代器放到一个进程中以生成值。您的代码和我的代码之间的区别在于，您的代码是非法的

results=[mp\u pool.map\u async（test，其中的\u迭代器）]

（或者使用

mp\u pool.imap的下一个版本）将尽可能地并行处理（取决于实际拥有的CPU数量）。如果您说迭代器本身很耗时，那么代码中没有任何内容使用多处理来生成迭代器。你是说你想使用多重处理来生成迭代器吗？我想知道为什么我的代码是非法的。我想使用多进程来迭代许多进程中的许多迭代器。
t = []
t = []
t = [9]
t = [8, 9]
t = [7, 8, 9]
t = [6, 7, 8, 9]
t = [5, 6, 7, 8, 9]
t = [4, 5, 6, 7, 8, 9]
t = [3, 4, 5, 6, 7, 8, 9]
t = [2, 3, 4, 5, 6, 7, 8, 9]