Python 带元组的dask延迟循环

如何正确使用task delayed进行多列的分组商计算

一些样本数据

raw_data = {
        'subject_id': ['1', '2', '3', '4', '5'],
        'name': ['A', 'B', 'C', 'D', 'E'],
        'nationality': ['DE', 'AUT', 'US', 'US', 'US'],
        'alotdifferent': ['x', 'y', 'z', 'x', 'a'],
        'target': [0,0,0,1,1],
        'age_group' : [1, 2, 1, 3, 1]}
df_a = pd.DataFrame(raw_data, columns = ['subject_id', 'name', 'nationality', 'alotdifferent','target','age_group'])
df_a.nationality = df_a.nationality.astype('category')
df_a.alotdifferent = df_a.alotdifferent.astype('category')
df_a.name = df_a.name.astype('category')

确定字符串/分类列的一些设置代码

FACTOR_FIELDS = df_a.select_dtypes(include=['category']).columns
columnsToDrop = ['alotdifferent']
columnsToBias_keep = FACTOR_FIELDS[~FACTOR_FIELDS.isin(columnsToDrop)]
target = 'target'

主体部分：群商的计算

def compute_weights(da, colname):
    # group only a single time
    grouped = da.groupby([colname, target]).size() 
    # calculate first ratio
    df = grouped / da[target].sum() 
    nameCol = "pre_" + colname 
    grouped_res = df.reset_index(name=nameCol) 
    grouped_res = grouped_res[grouped_res[target] == 1] 
    grouped_res = grouped_res.drop(target, 1) 
    # todo persist the result in dict for transformer
    result_1 = grouped_res
    return result_1, nameCol

现在在多个栏目中实际调用它

original = df_a.copy()
output_df = original
ratio_weights = {}

for colname in columnsToBias_keep.union(columnsToDrop):
    result_1, result_2, nameCol, nameCol_2 = compute_weights(original, colname)

    # persist the result in dict for transformer
    # this is required to separate fit and transform stage (later on in a sklearn transformer)
    ratio_weights[nameCol] = result_1
    ratio_weights[nameCol_2] = result_2

当尝试使用dask延迟时，我需要调用compute，这会破坏DAG。为了创建一个单独的并行计算的大型计算图，我该如何对此进行校正

compute_weights = delayed(compute_weights)
a,b = delayed_res_name.compute()
ratio_weights = {}
ratio_weights[b] = a

---

我怀疑，如果你能把你的问题简化成一个问题，你会更快地得到答案。一般来说，这意味着剥离所有特定于您的领域的细节，只关注API问题，也许使用一个具有最少代码行数的玩具示例。我怀疑，如果您能够将问题减少到一个最小值，您将更快地得到答案。一般来说，这意味着剥离所有特定于您的域的细节，只关注API问题，也许可以使用一个具有最少代码行数的玩具示例。