Python joblib与大熊猫并行慢速.DataFrame

我正在尝试并行化一个for循环，它基本上在一个数据帧数组上循环。显然，我遗漏了一些东西，因为这对于相对较小的数据帧（~1e5行）来说效果很好，但对于较大的数据帧（~3e6行）来说却变得非常缓慢

最小（不是真的，但我尝试过）示例：

当然，当进程分叉并将元素作为参数传递给包装器时，有一些方法可以更好地处理内存，以避免复制self.sim的元素，但我认为问题不在于此，因为trip_manager（）中的for循环正在变慢

任何帮助或意见都将不胜感激，谢谢

import numpy as np
import pandas as pd
import multiprocessing
from joblib import Parallel, delayed

class Analysis:

    # ...
    # istantiate the pd.DataFrame self.data 

    # Checks for every person (pid) if the difference between consecutive values of lifetime is equal to 
    # some value (self.samp). If not, increase the 'trip #' of the corresponding row entries 
    def trip_manager(self):
      
        self.data.insert(0, 'trip #', 0)
        for pid in self.data.index.unique():
            datadf = self.data.loc[pid]

            newtrip = 0
            while(1):

                try:
                    arr = np.where(datadf['lifetime'].values[newtrip+1:] -
                                   datadf['lifetime'].values[newtrip:-1]
                                   != self.samp)[0]

                except AttributeError:
                    break

                if(len(arr) == 0):  # if only one value, datadf is a pd.Series
                    break

                newtrip += arr[0] + 1
                datadf.iloc[newtrip:, 0] += 1


class MultiAnalysis:

    # ...
    # istantiate the array of Analysis self.sim

    # For each element of the array execute trip_manager()
    def aggregate(self):

        num_cores = multiprocessing.cpu_count()
        njobs = num_cores
        inputs = range(len(self.msim))

        def wrapper(sim):
            sim.trip_manager()
            return sim

        if __name__ == '__main__':
            self.sim = Parallel(n_jobs=njobs, verbose=100, backend='loky')(
                                delayed(wrapper)(self.sim[i]) for i in inputs)