Python Pypark动态平均值计算CV目标平均值编码

使用-Python3.6、Spark 2.3

原始DF-

key     a_fold_0    b_fold_0    a_fold_1    b_fold_1    a_fold_2    b_fold_2
1   1   2   3   4   5   6
2   7   5   3   5   2   1

我想从下面的数据框计算平均值，如下所示（对于所有列和所有折叠，如下所示）-

过程-

对于fold_0，我的平均值应该是fold_1+fold_2/2

对于fold_1，我的平均值应为fold_0+fold_2/2

对于折叠2，我的平均值应为折叠0+折叠1/2

对于每列

我的列数，折叠数，一切都是动态的

如何在pyspark数据帧上解决此问题

尝试通过交叉验证目标平均编码技术创建新功能。

自己解决了这个问题

以防任何人需要重用代码-

orig_list = ['Married-spouse-absent', 'Married-AF-spouse', 'Separated', 'Married-civ-spouse', 'Widowed', 'Divorced', 'Never-married']
k_folds = 3

cols = df.columns  # ['fnlwgt_bucketed', 'Married-spouse-absent_fold_0', 'Married-AF-spouse_fold_0', 'Separated_fold_0', 'Married-civ-spouse_fold_0', 'Widowed_fold_0', 'Divorced_fold_0', 'Never-married_fold_0', 'Married-spouse-absent_fold_1', 'Married-AF-spouse_fold_1', 'Separated_fold_1', 'Married-civ-spouse_fold_1', 'Widowed_fold_1', 'Divorced_fold_1', 'Never-married_fold_1', 'Married-spouse-absent_fold_2', 'Married-AF-spouse_fold_2', 'Separated_fold_2', 'Married-civ-spouse_fold_2', 'Widowed_fold_2', 'Divorced_fold_2', 'Never-married_fold_2']

for folds in range(k_folds):
    for column in orig_list:
        col_namer = []
        for fold in range(k_folds):
            if fold != folds:
                col_namer.append(column+'_fold_'+str(fold))
        df = df.withColumn(column+'_fold_'+str(folds)+'_mean', (sum(df[col] for col in col_namer)/(k_folds-1)))
        print(col_namer)
df.show(1)

orig_list = ['Married-spouse-absent', 'Married-AF-spouse', 'Separated', 'Married-civ-spouse', 'Widowed', 'Divorced', 'Never-married']
k_folds = 3

cols = df.columns  # ['fnlwgt_bucketed', 'Married-spouse-absent_fold_0', 'Married-AF-spouse_fold_0', 'Separated_fold_0', 'Married-civ-spouse_fold_0', 'Widowed_fold_0', 'Divorced_fold_0', 'Never-married_fold_0', 'Married-spouse-absent_fold_1', 'Married-AF-spouse_fold_1', 'Separated_fold_1', 'Married-civ-spouse_fold_1', 'Widowed_fold_1', 'Divorced_fold_1', 'Never-married_fold_1', 'Married-spouse-absent_fold_2', 'Married-AF-spouse_fold_2', 'Separated_fold_2', 'Married-civ-spouse_fold_2', 'Widowed_fold_2', 'Divorced_fold_2', 'Never-married_fold_2']

for folds in range(k_folds):
    for column in orig_list:
        col_namer = []
        for fold in range(k_folds):
            if fold != folds:
                col_namer.append(column+'_fold_'+str(fold))
        df = df.withColumn(column+'_fold_'+str(folds)+'_mean', (sum(df[col] for col in col_namer)/(k_folds-1)))
        print(col_namer)
df.show(1)