具有多个参数的Python UDF
问题陈述是让员工的所有经理达到Spark中的给定级别 例如:在下面的数据集中具有多个参数的Python UDF,python,apache-spark,pyspark,apache-spark-sql,pyspark-sql,Python,Apache Spark,Pyspark,Apache Spark Sql,Pyspark Sql,问题陈述是让员工的所有经理达到Spark中的给定级别 例如:在下面的数据集中 EMPLOYEE_ID,FIRST_NAME,LAST_NAME,EMAIL,PHONE_NUMBER,HIRE_DATE,JOB_ID,SALARY,COMMISSION_PCT,MANAGER_ID,DEPARTMENT_ID 1,Donald,OConnell,DOCONNEL,650.507.9833,21/06/2007,SH_CLERK,2600,,2,500 2,Douglas,Grant,DGRANT,
EMPLOYEE_ID,FIRST_NAME,LAST_NAME,EMAIL,PHONE_NUMBER,HIRE_DATE,JOB_ID,SALARY,COMMISSION_PCT,MANAGER_ID,DEPARTMENT_ID
1,Donald,OConnell,DOCONNEL,650.507.9833,21/06/2007,SH_CLERK,2600,,2,500
2,Douglas,Grant,DGRANT,650.507.9844,13/01/2008,SH_CLERK,2600,,3,50
3,Jennifer,Whalen,JWHALEN,515.123.4444,17/09/2003,AD_ASST,4400,,4,10
4,Michael,Hartstein,MHARTSTE,515.123.5555,17/02/2004,MK_MAN,13000,,5,20
5,Pat,Fay,PFAY,603.123.6666,17/08/2005,MK_REP,6000,,6,20
6,Susan,Mavris,SMAVRIS,515.123.7777,07/06/2002,HR_REP,6500,,7,40
7,Hermann,Baer,HBAER,515.123.8888,07/06/2002,PR_REP,10000,,8,70
8,Shelley,Higgins,SHIGGINS,515.123.8080,07/06/2002,AC_MGR,12008,,9,110
9,William,Gietz,WGIETZ,515.123.8181,07/06/2002,AC_ACCOUNT,8300,,,110
给定员工“1”,预期结果为['3','4','5','6','7','8','9']
下面是我尝试编写的PySpark代码,但由于上述错误而失败
import sys
import os
from operator import add
import re
import pyspark.sql.functions as F
os.environ['SPARK_HOME'] = "path"
sys.path.append("path")
sys.path.append("path")
def recur_man(emp_id,lvl,list1):
with open("path\\employee_1.txt") as f:
for lines in f:
if lines.split(',')[0] == emp_id:
list1.append(lines.split(',')[9])
lvl-=1
recur_man(lines.split(',')[9],lvl,list1)
return list1
try:
from pyspark import SparkContext
from pyspark import SparkConf
from pyspark.sql import SQLContext
from pyspark.sql.functions import *
from pyspark.sql.types import *
config = SparkConf().setAll([('spark.num.executors','10'),('spark.ui.port','4050')])
sc = SparkContext(conf=config)
sqlContext = SQLContext(sc)
rdd = sc.textFile("path\\employee_1.txt")
header = rdd.first()
header_mod = [x.encode("utf-8") for x in header.split(',')]
rdd = rdd.filter(lambda line: line!=header)
rdd = rdd.map(lambda line: line.split(','))
df1 = rdd.toDF(header_mod)
spark_recur_man = udf(lambda x: recur_man,ArrayType(StringType()))
list1 = []
df1.select('EMPLOYEE_ID',spark_recur_man('EMPLOYEE_ID',3,list1).alias('heirarchy')).show(truncate=False)
sc.stop()
except ImportError as e:
print ("Error importing Spark Modules", e)
sys.exit(1)
错误如下:
df1.select('EMPLOYEE_ID',spark_recur_man('EMPLOYEE_ID',3,list1).alias('heirarchy')).show(truncate=False)
文件
“C:\opt\spark\spark-2.2.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\functions.py”,包装文件中的第1957行
“C:\opt\spark\spark-2.2.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\functions.py”,第1918行,在调用文件中
“C:\opt\spark\spark-2.2.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\column.py”,
第60行,在_至_顺序文件中
“C:\opt\spark\spark-2.2.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\column.py”,
第48行,在_to_java_列文件中
“C:\opt\spark\spark-2.2.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\column.py”,
第41行,从名称文件创建列
“C:\opt\spark\spark-2.2.0-bin-hadoop2.7\python\lib\py4j-0.10.4-src.zip\py4j\java_gateway.py”,
第1133行,在调用文件中
“C:\opt\spark\spark-2.2.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\utils.py”,
第63行,在deco文件中
“C:\opt\spark\spark-2.2.0-bin-hadoop2.7\python\lib\py4j-0.10.4-src.zip\py4j\protocol.py”,获取返回值py4j.protocol.Py4JError中的第323行:错误
调用z:org.apache.spark.sql.functions.col时发生。跟踪:
py4j.Py4JException:方法col([class java.lang.Integer])不存在
存在
有人能帮我解决上述问题吗
提前感谢Spark无法获取接受UDF数组最后一个参数的类型,即list1 以下是目前的解决方案,您可以尝试:
import sys
import os
from operator import add
import re
import numpy as np
import pyspark.sql.functions as F
import pyspark.sql.types as t
import pyspark.sql.column as c
def square(x):
return x**2
def recur_man(emp_id,lvl,list1):
if(type(list1)==int):
list1=[]
with open("D:\MOCK_DATA\employee_1.txt") as f:
for lines in f:
if lines.split(',')[0] == emp_id:
list1.append(lines.split(',')[9])
lvl-=1
recur_man(lines.split(',')[9],lvl,list1)
return list1
try:
from pyspark import SparkContext
from pyspark import SparkConf
from pyspark.sql import SQLContext
from pyspark.sql.functions import *
from pyspark.sql.types import *
sc = SparkContext.getOrCreate()
sqlContext = SQLContext(sc)
rdd = sc.textFile("D:\MOCK_DATA\employee_1.txt")
header = rdd.first()
header_mod = [x.encode("utf-8") for x in header.split(',')]
rdd = rdd.filter(lambda line: line!=header)
rdd = rdd.map(lambda line: line.split(','))
df1 = rdd.toDF(header_mod)
spark_recur_man = udf(lambda x,y,z: recur_man(x,y,z), t.ArrayType(t.StringType()))
list1 = lit(0)
df1.select('EMPLOYEE_ID',spark_recur_man('EMPLOYEE_ID',lit(3),list1).alias('heirarchy')).show(truncate=False)
sc.stop()
except ImportError as e:
print ("Error importing Spark Modules", e)
sys.exit(1)