Python Pyspark中对分组ML向量应用UDF余弦相似性的问题

对分组数据应用UDF（

dot\u group

）时出错。该UDF的目的是计算由

特征

列构成的每组ML

向量

之间的成对余弦相似性。根据输入数据（

cdf

）的

prediction

列进行分组。结果应该是一个

数组类型

，其中每个项目都是一个结果相似性，写入

余弦

列。这是我的尝试：

from pyspark.sql import SparkSession
from pyspark.sql.types import *
import pyspark.sql.functions as F
from pyspark.ml.linalg import Vectors
from itertools import combinations
from numpy import linalg as LA


def g_dot(M):
    combs = combinations(M, 2)
    return [i.dot(j) /(LA.norm(i) * LA.norm(j)) \
                                            for i, j in combs]
dot_group = F.udf(g_dot, ArrayType(DoubleType()))


cdf = spark.createDataFrame(
            [(1.0, Vectors.dense([0.0, 10.0, 0.5])), 
             (0.0, Vectors.dense([0.0, 1.0, 0.5])),
             (1.0, Vectors.dense([0.0, 10.0, 0.1])),
             (0.0, Vectors.dense([10.0, 10.0, 0.5])),
             (1.0, Vectors.dense([0.0, 0.0, 0.5])),],
            ["prediction", "features"])

dfs = cdf.groupBy(["prediction"]) \
         .agg(F.collect_list("features").alias("data")) \
         .withColumn("cosines", dot_group("data"))
dfs.show()

。。。这会产生以下错误。我不确定为什么会出现此错误，但序列化numpy操作似乎有问题：

19/02/19 16:21:39 ERROR Executor: Exception in task 0.0 in stage 2093.0 (TID 1185)
net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for numpy.dtype)
        at net.razorvine.pickle.objects.ClassDictConstructor.construct(ClassDictConstructor.java:23)
        at net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:707)
        at net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:175)
        at net.razorvine.pickle.Unpickler.load(Unpickler.java:99)
        at net.razorvine.pickle.Unpickler.loads(Unpickler.java:112)
        at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$doExecute$1$$anonfun$apply$6.apply(BatchEvalPythonExec.scala:156)
        at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$doExecute$1$$anonfun$apply$6.apply(BatchEvalPythonExec.scala:155)
        at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:108)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
19/02/19 16:21:39 WARN TaskSetManager: Lost task 0.0 in stage 2093.0 (TID 1185, localhost, executor driver): net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for numpy.dtype)

...
19/02/19 16:21:39 ERROR TaskSetManager: Task 0 in stage 2093.0 failed 1 times; aborting job
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Aplication/spark/spark-2.2.1-bin-hadoop2.7/python/pyspark/sql/dataframe.py", line 336, in show
    print(self._jdf.showString(n, 20))
  File "/Aplication/spark/spark-2.2.1-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/Aplication/spark/spark-2.2.1-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/Aplication/spark/spark-2.2.1-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o2000.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2093.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2093.0 (TID 1185, localhost, executor driver): net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for numpy.dtype)
        at net.razorvine.pickle.objects.ClassDictConstructor.construct(ClassDictConstructor.java:23)
        at net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:707)
        at net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:175)
        at net.razorvine.pickle.Unpickler.load(Unpickler.java:99)
        at net.razorvine.pickle.Unpickler.loads(Unpickler.java:112)
        at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$doExecute$1$$anonfun$apply$6.apply(BatchEvalPythonExec.scala:156)
        at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$doExecute$1$$anonfun$apply$6.apply(BatchEvalPythonExec.scala:155)
        at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:108)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

19/02/19 16:21:39错误执行者：2093.0阶段任务0.0中出现异常（TID 1185）
net.razorvine.pickle.PickleException:构造ClassDict（对于numpy.dtype）应为零参数
位于net.razorvine.pickle.objects.ClassDictConstructor.construct（ClassDictConstructor.java:23）
net.razorvine.pickle.Unpickler.load\u reduce（Unpickler.java:707）
位于net.razorvine.pickle.Unpickler.dispatch（Unpickler.java:175）
位于net.razorvine.pickle.Unpickler.load（Unpickler.java:99）
加载net.razorvine.pickle.Unpickler.load（Unpickler.java:112）
位于org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$doExecute$1$$anonfun$apply$6.apply（BatchEvalPythonExec.scala:156）
位于org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$doExecute$1$$anonfun$apply$6.apply（BatchEvalPythonExec.scala:155）
位于scala.collection.Iterator$$anon$12.nextCur（Iterator.scala:434）
位于scala.collection.Iterator$$anon$12.hasNext（Iterator.scala:440）
位于scala.collection.Iterator$$anon$11.hasNext（Iterator.scala:408）
位于org.apache.spark.sql.catalyst.expressions.GeneratedClass$GenerateEditor.processNext（未知源）
位于org.apache.spark.sql.execution.BufferedRowIterator.hasNext（BufferedRowIterator.java:43）
位于org.apache.spark.sql.execution.whisttagecodegenexec$$anonfun$8$$anon$1.hasNext（whisttagecodegenexec.scala:395）
位于org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply（SparkPlan.scala:234）
位于org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply（SparkPlan.scala:228）
位于org.apache.spark.rdd.rdd$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply（rdd.scala:827）
位于org.apache.spark.rdd.rdd$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply（rdd.scala:827）
在org.apache.spark.rdd.MapPartitionsRDD.compute上（MapPartitionsRDD.scala:38）
在org.apache.spark.rdd.rdd.computeOrReadCheckpoint（rdd.scala:323）上
位于org.apache.spark.rdd.rdd.iterator（rdd.scala:287）
位于org.apache.spark.scheduler.ResultTask.runTask（ResultTask.scala:87）
位于org.apache.spark.scheduler.Task.run（Task.scala:108）
位于org.apache.spark.executor.executor$TaskRunner.run（executor.scala:338）
位于java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1149）
位于java.util.concurrent.ThreadPoolExecutor$Worker.run（ThreadPoolExecutor.java:624）
运行（Thread.java:748）
19/02/19 16:21:39警告TaskSetManager:在阶段2093.0（TID 1185，localhost，executor驱动程序）中丢失了任务0.0:net.razorvine.pickle.PickleException:构造ClassDict（对于numpy.dtype）需要零个参数
...
19/02/19 16:21:39错误TaskSetManager:阶段2093.0中的任务0失败1次；中止工作
回溯（最近一次呼叫最后一次）：
文件“”，第1行，在
文件“/Aplication/spark/spark-2.2.1-bin-hadoop2.7/python/pyspark/sql/dataframe.py”，第336行，显示
打印（self.\u jdf.showString（n，20））
文件“/Aplication/spark/spark-2.2.1-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py”，第1133行，在__
文件“/Aplication/spark/spark-2.2.1-bin-hadoop2.7/python/pyspark/sql/utils.py”，第63行，deco格式
返回f（*a，**kw）
文件“/Aplication/spark/spark-2.2.1-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py”，第319行，在get\u返回值中
py4j.protocol.Py4JJavaError:调用o2000.showString时出错。
：org.apache.spark.sparkeexception:作业因阶段失败而中止：阶段2093.0中的任务0失败1次，最近的失败：阶段2093.0中的任务0.0丢失（TID 1185，localhost，executor driver）：net.razorvine.pickle.PickleException:构造ClassDict（对于numpy.dtype）的参数应为零
位于net.razorvine.pickle.objects.ClassDictConstructor.construct（ClassDictConstructor.java:23）
net.razorvine.pickle.Unpickler.load\u reduce（Unpickler.java:707）
位于net.razorvine.pickle.Unpickler.dispatch（Unpickler.java:175）
位于net.razorvine.pickle.Unpickler.load（Unpickler.java:99）
加载net.razorvine.pickle.Unpickler.load（Unpickler.java:112）
位于org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$doExecute$1$$anonfun$apply$6.apply（BatchEvalPythonExec.scala:156）
位于org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$doExecute$1$$anonfun$apply$6.apply（BatchEvalPythonExec.scala:155）
位于scala.collection.Iterator$$anon$12.nextCur（Iterator.scala:434）
位于scala.collection.Iterator$$anon$12.hasNext（Iterator.scala:440）
位于scala.collection.Iterator$$anon$11.hasNext（Iterator.scala:408）
位于org.apache.spark.sql.catalyst.expressions.GeneratedClass$GenerateEditor.processNext（未知源）
位于org.apache.spark.sql.execution.BufferedRowIterator.hasNext（BufferedRowIterator.java:43）
位于org.apache.spark.sql.execution.whisttagecodegenexec$$anonfun$8$$anon$1.hasNext（whisttagecodegenexec.scala:395）
位于org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply（SparkPlan.scala:234）
位于org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply（SparkPlan.scala:228）
在org.apache.spark.rdd.rdd$$anonfun$mapPartitionsInternal$1$$anonfun$appl
@F.udf(ArrayType(DoubleType()))
def dot_group(M):
    combs = combinations(M, 2)
    return [
        # or float(i.dot(j) / (LA.norm(i) * LA.norm(j)))
        (i.dot(j) / (LA.norm(i) * LA.norm(j))).tolist()
        for i, j in combs
    ]