Python 如何并行化复杂的for循环

我有一个复杂的

for

循环，它包含对循环中多个记录的多个操作。循环如下所示：

for i,j,k in zip(is,js,ks):
    #declare multiple lists.. like
    a = []
    b = []
    #...
    if i:
        for items in i:
            values = items['key'].split("--")
            #append the values to the declared lists
            a.append(values[0])
            b.append(values[1])
    # also other operations with j and k where are is a list of dicts. 
    if "substring" in k:
        for k, v in j["key"].items():
            l = "string"
            t = v
    else:
        for k, v in j["key2"].items():
            l = k
            t = v

            # construct an object with all the lists/params
            content = {
                'sub_content': {
                    "a":a,
                    "b":b,
                    .
                    .
                }
            }

            #form a tuple. We are interested in this tuple.
            data_tuple = (content,t,l)

考虑到上面的

for

循环，我如何将其并行化？我已经研究过多处理，但还没有能够并行化如此复杂的循环。我也愿意接受一些可能在这里表现更好的建议，包括OpenMP/MPI/OpenACC等并行语言范例。

您可以使用Python库。如中所述，您应该确定是需要多处理还是多线程

底线：如果需要多线程，则应使用multiprocessing.dummy。如果您只执行CPU密集型任务，而没有IO/依赖项，则可以使用多处理

multiprocessing.dummy与multiprocessing模块完全相同， 而是使用线程（一个重要的区别——使用多线程） CPU密集型任务的进程；IO（和IO期间）的线程：

设置zip对象 简单示例函数 设置多线程。 你的全部例子

---

您可以添加一个示例来演示相同的功能吗？@Eswar-请对此进行测试，看看return语句

return（content，t，l）

是否正确。如果my\u函数没有返回任何内容，则池是否返回一些内容？我可以运行pool.map（my_func，zip_object）吗？小心命名变量

is

，因为它已经是Python语句了。我在设置测试时遇到了这个问题：-）-Guimoute我在帖子中更改的变量。否则，名称将根据其相关性而定。

#!/usr/bin/env python3

import numpy as np

n = 2000
xs = np.arange(n)
ys = np.arange(n) * 2
zs = np.arange(n) * 3

zip_obj = zip(xs, ys, zs)

def my_function(my_tuple):
    iv, jv, kv = my_tuple
    return f"{str(iv)}-{str(jv)}-{str(kv)}"   

from multiprocessing.dummy import Pool as ThreadPool
pool = ThreadPool(4)
data_tuple = pool.map(my_function, zip_obj)

def my_function(my_tuple):
    i, j, k = my_tuple
    #declare multiple lists.. like
    a = []
    b = []
    #...
    if (i):
        for items in i:
            values = items['key'].split("--")
            #append the values to the declared lists
            a.append(values[0])
            b.append(values[1])
     #also other ooperations with j and k where are is a list of dicts. 
     if ("substring" in k):
           for k, v in j["key"].items():
               l = "string"
               t = v
      else:
           for k, v in j["key2"].items():
               l = k
               t = v
    #construct an object called content with all the lists/params like
           content = {
                'sub_content': {
                  "a":a,
                  "b":b,
                  .
                  .
                }
            }
    #form a tuple. We are interested in this tuple.
    return (content,t,l)


from multiprocessing.dummy import Pool as ThreadPool
pool = ThreadPool(4)
zip_obj = zip(is,js,ks)
data_tuple = pool.map(my_function, zip_obj)
# Do whatever you need to do w/ data_tuple here