Python 并行程序设计方法在熊猫问题求解中的应用

我有一个以下格式的数据帧。 df

我使用pd.DataFramedf.corr.iloc[：-1，-1]查找除Target之外的每个列与目标列的相关性。 但问题是-我的实际数据帧的大小是21672391，在我的系统上处理它至少需要30分钟。有没有办法用gpu并行化它？我需要多次找到类似类型的值，因此无法等待每次30分钟的正常处理时间。

您应该查看。它应该能够做你想做的事情，还有更多。

---

它并行化了大多数数据帧函数。

在这里，我尝试使用numba实现您的操作

---

链接到。

所以您想查找2620192245个值，即72391选择2？不，不是72391选择2，而是72391个值。对于72391列，如A、Target、B、Target等可能会有所帮助。我尝试运行它，在读取csv文件时，得到以下错误值error:Sample不够大，无法包含至少一行数据。请增加read_csv/read_表调用中sample`中的字节数

A   B  Target
5   4   3
1   3   4

import numpy as np
import pandas as pd
from numba import jit, int64, float64

# 
#------------You can ignore the code starting from here---------
#
# Create a random DF with cols_size = 72391 and row_size =300
df_dict = {}
for i in range(0, 72391):
  df_dict[i] = np.random.randint(100, size=300)
target_array = np.random.randint(100, size=300)

df = pd.DataFrame(df_dict)
# ----------Ignore code till here. This is just to generate dummy data-------

# Assume df is your original DataFrame
target_array = df['target'].values

# You can choose to restore this column later
# But for now we will remove it, since we will 
# call the df.values and find correlation of each 
# column with target
df.drop(['target'], inplace=True, axis=1)

# This function takes in a numpy 2D array and a target array as input
# The numpy 2D array has the data of all the columns
# We find correlation of each column with target array
# numba's Jit required that both should have same columns
# Hence the first 2d array is transposed, i.e. it's shape is (72391,300)
# while target array's shape is (300,) 
def do_stuff(df_values, target_arr):
  # Just create a random array to store result
  # df_values.shape[0] = 72391, equal to no. of columns in df
  result = np.random.random(df_values.shape[0])

  # Iterator over each column
  for i in range(0, df_values.shape[0]):

    # Find correlation of a column with target column
    # In order to find correlation we must transpose array to make them compatible
    result[i] = np.corrcoef(np.transpose(df_values[i]), target_arr.reshape(300,))[0][1]
  return result

# Decorate the function do_stuff
do_stuff_numba = jit(nopython=True, parallel=True)(do_stuff)

# This contains all the correlation
result_array = do_stuff_numba(np.transpose(df.T.values), target_array)