Python 并行迭代

所以，我想并行地迭代一行，假设我有15行，然后我想并行地迭代它，而不是逐个迭代

df:-

因此，我在df上迭代并生成命令行，然后将输出存储在df中，进行数据过滤，最后将其存储到XDB中。问题是我在一个接一个地重复它。我想要在所有行上并行迭代的内容

到目前为止，我已经制作了20个脚本，并使用多处理并行地检查了所有脚本。当我不得不在所有20个脚本中进行更改时，这是一种痛苦。我的脚本如下所示：-

for index, row in dff.iterrows():
    domain = row['domain']
    duration = str(row['duration'])
    media_file = row['media_file']
    user = row['user']
    channel = row['channel']
    cmda = './vaa -s https://' + domain + '.www.vivox.com/api2/ -d ' + 
    duration + ' -f ' + media_file + ' -u .' + user + '. -c 
    sip:confctl-2@' + domain + '.localhost.com -ati 0ps-host -atk 0ps- 
    test'

    rows = [shlex.split(line) for line in os.popen(
    cmda).read().splitlines() if line.strip()]

    df = pd.DataFrame(rows)
    """
    Bunch of data filteration and pushing it into influx 
    """

import os
import time
from multiprocessing import Process
os.chdir('/Users/akumar/vivox-sdk-4.9.0002.30719.ebb523a9')
def run_program(cmd):
    # Function that processes will run
    os.system(cmd)

# Creating command to run
commands = ['python testv.py']
commands.extend(['python testv{}.py'.format(i) for i in range(1, 15)])

# Amount of times your programs will run
runs = 1

for run in range(runs):
    # Initiating Processes with desired arguments
    running_programs = []
    for command in commands:
        running_programs.append(Process(target=run_program, args=(command,)))
        running_programs[-1].daemon = True

    # Start our processes simultaneously
    for program in running_programs:
        program.start()

    # Wait untill all programs are done
    while any(program.is_alive() for program in running_programs):
        time.sleep(1)

到目前为止，如果我在df中获得15行并执行如下并行处理，则我有15个脚本：-

for index, row in dff.iterrows():
    domain = row['domain']
    duration = str(row['duration'])
    media_file = row['media_file']
    user = row['user']
    channel = row['channel']
    cmda = './vaa -s https://' + domain + '.www.vivox.com/api2/ -d ' + 
    duration + ' -f ' + media_file + ' -u .' + user + '. -c 
    sip:confctl-2@' + domain + '.localhost.com -ati 0ps-host -atk 0ps- 
    test'

    rows = [shlex.split(line) for line in os.popen(
    cmda).read().splitlines() if line.strip()]

    df = pd.DataFrame(rows)
    """
    Bunch of data filteration and pushing it into influx 
    """

import os
import time
from multiprocessing import Process
os.chdir('/Users/akumar/vivox-sdk-4.9.0002.30719.ebb523a9')
def run_program(cmd):
    # Function that processes will run
    os.system(cmd)

# Creating command to run
commands = ['python testv.py']
commands.extend(['python testv{}.py'.format(i) for i in range(1, 15)])

# Amount of times your programs will run
runs = 1

for run in range(runs):
    # Initiating Processes with desired arguments
    running_programs = []
    for command in commands:
        running_programs.append(Process(target=run_program, args=(command,)))
        running_programs[-1].daemon = True

    # Start our processes simultaneously
    for program in running_programs:
        program.start()

    # Wait untill all programs are done
    while any(program.is_alive() for program in running_programs):
        time.sleep(1)

---

问题：-我如何迭代df，使所有15行并行运行，并在for循环中执行所有操作。

使用线程并使用参数调用线程化函数，而不是启动15个进程

threading.Thread（target=func，args=（i，）

其中i是您的编号，

func

是包装整个代码的函数。然后遍历它。您不需要在15个项目上并行迭代。

我将从Reddit复制并粘贴我的答案到这里（以防有人在类似情况下偶然发现）：

---

您可能需要在

apply

方法中使用axis参数。

更新：为熊猫数据帧进行多处理的另一个好方法是modin项目：在最后一行，

lambda x:your_函数（x）

可以简化为

your_函数