Python 如何pickle从PySpark DataFrame继承的对象实例
因此,我有一个自定义类继承自Python 如何pickle从PySpark DataFrame继承的对象实例,python,apache-spark,pyspark,pickle,Python,Apache Spark,Pyspark,Pickle,因此,我有一个自定义类继承自pyspark.sql.dataframe.dataframe: from pyspark.sql.dataframe import DataFrame class MyCustomClass(DataFrame): def __init__(self, sdf): super().__init__(sdf._jdf, sdf.sql_ctx) 我希望能够定制此类实例的酸洗。我试图做的是重写\uuuu getstate\uuuuuuuuu
pyspark.sql.dataframe.dataframe
:
from pyspark.sql.dataframe import DataFrame
class MyCustomClass(DataFrame):
def __init__(self, sdf):
super().__init__(sdf._jdf, sdf.sql_ctx)
我希望能够定制此类实例的酸洗。我试图做的是重写\uuuu getstate\uuuuuuuuuuu()
和\uuuuuuu setstate\uuuuuuuuuu()
方法:
def __getstate__(self):
state = super().__getstate__()
state['extra_key'] = self.one_property
return state
但我得到了以下错误:
> state = super().__getstate__()
E AttributeError: 'super' object has no attribute '__getstate__'
这让我觉得\uuu getstate\uuu
不是由父类(即PySpark数据帧)实现的。
你有什么想法吗