为PySpark中表中的分类数据分配标签

我想使用pysparksql将标签分配给下面数据框中的分类号

在婚姻栏中，1=已婚，2=未婚。在教育栏中，1=毕业生，2=本科生

Current Dataframe: +--------+---------+-----+ |MARRIAGE|EDUCATION|Total| +--------+---------+-----+ | 1| 2| 87| | 1| 1| 123| | 2| 2| 3| | 2| 1| 8| +--------+---------+-----+

---

一个UDF只能产生一列。但这可以是结构化的列，UDF可以在婚姻和教育上应用标签。见下面的代码：

from pyspark.sql.types import *
from pyspark.sql import Row

udf_result = StructType([StructField('MARRIAGE', StringType()), StructField('EDUCATION', StringType())]) 

marriage_dict = {1: 'Married', 2: 'UnMarried'}
education_dict = {1: 'Grad', 2: 'UnderGrad'}
def assign_labels(marriage, education):
    return Row(marriage_dict[marriage], education_dict[education])

assign_labels_udf = F.udf(assign_labels, udf_result)
df.withColumn('labels', assign_labels_udf('MARRIAGE', 'EDUCATION')).printSchema()
root
 |-- MARRIAGE: long (nullable = true)
 |-- EDUCATION: long (nullable = true)
 |-- Total: long (nullable = true)
 |-- labels: struct (nullable = true)
 |    |-- MARRIAGE: string (nullable = true)
 |    |-- EDUCATION: string (nullable = true)

---

但正如您所看到的，它并没有替换原来的列，它只是添加了一个新列。要替换它们，您需要使用

with column

两次，然后删除

标签

一个UDF只能生成一列。但这可以是结构化的列，UDF可以在婚姻和教育上应用标签。见下面的代码：

from pyspark.sql.types import *
from pyspark.sql import Row

udf_result = StructType([StructField('MARRIAGE', StringType()), StructField('EDUCATION', StringType())]) 

marriage_dict = {1: 'Married', 2: 'UnMarried'}
education_dict = {1: 'Grad', 2: 'UnderGrad'}
def assign_labels(marriage, education):
    return Row(marriage_dict[marriage], education_dict[education])

assign_labels_udf = F.udf(assign_labels, udf_result)
df.withColumn('labels', assign_labels_udf('MARRIAGE', 'EDUCATION')).printSchema()
root
 |-- MARRIAGE: long (nullable = true)
 |-- EDUCATION: long (nullable = true)
 |-- Total: long (nullable = true)
 |-- labels: struct (nullable = true)
 |    |-- MARRIAGE: string (nullable = true)
 |    |-- EDUCATION: string (nullable = true)

但正如您所看到的，它并没有替换原来的列，它只是添加了一个新列。要替换它们，您需要使用

with column

两次，然后放下

标签

from pyspark.sql.types import *
from pyspark.sql import Row

udf_result = StructType([StructField('MARRIAGE', StringType()), StructField('EDUCATION', StringType())]) 

marriage_dict = {1: 'Married', 2: 'UnMarried'}
education_dict = {1: 'Grad', 2: 'UnderGrad'}
def assign_labels(marriage, education):
    return Row(marriage_dict[marriage], education_dict[education])

assign_labels_udf = F.udf(assign_labels, udf_result)
df.withColumn('labels', assign_labels_udf('MARRIAGE', 'EDUCATION')).printSchema()
root
 |-- MARRIAGE: long (nullable = true)
 |-- EDUCATION: long (nullable = true)
 |-- Total: long (nullable = true)
 |-- labels: struct (nullable = true)
 |    |-- MARRIAGE: string (nullable = true)
 |    |-- EDUCATION: string (nullable = true)