Serialization 如何在Flink中使用自定义序列化方法?
我在RichMapFunctin.Like的构造函数中使用了Serialization 如何在Flink中使用自定义序列化方法?,serialization,apache-flink,Serialization,Apache Flink,我在RichMapFunctin.Like的构造函数中使用了AtomicLongMap public class PathAnalysis extends RichMapFunction<ApiLog, ApiLog> { private final AtomicLongMap<Object> mObjectAtomicLongMap; public PathAnalysis() { mObjectAtomicLongMap = AtomicLongMap.c
AtomicLongMap
public class PathAnalysis extends RichMapFunction<ApiLog, ApiLog> {
private final AtomicLongMap<Object> mObjectAtomicLongMap;
public PathAnalysis()
{
mObjectAtomicLongMap = AtomicLongMap.create();
}
}
它导致:
org.apache.flink.api.common.InvalidProgramException: The implementation of the RichMapFunction is not serializable. The object probably contains or references non serializable fields.
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:99)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:1550)
at org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:184)
at org.apache.flink.streaming.api.datastream.DataStream.map(DataStream.java:528)
at com.ghzs.Topology.main(Topology.java:91)
Caused by: java.io.NotSerializableException: org.apache.flink.shaded.curator.org.apache.curator.shaded.com.google.common.util.concurrent.AtomicLongMap
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:315)
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:81)
... 4 more
AtomicLongMap
不是可序列化的实现。如何注册有效的自定义序列化方法?尝试实现此函数,如:
import org.apache.flink.api.common.functions.RichMapFunction;
import com.google.common.util.concurrent.AtomicLongMap;
public class PathAnalysis extends RichMapFunction<ApiLog, ApiLog> {
private final AtomicLongMap<Object> mObjectAtomicLongMap;
public PathAnalysis()
{
mObjectAtomicLongMap = AtomicLongMap.create();
}
@Override
public ApiLog map(ApiLog value) throws Exception {
// TODO Auto-generated method stub
return null;
}
}
import org.apache.flink.api.common.functions.RichMapFunction;
导入com.google.common.util.concurrent.AtomicLongMap;
公共类PathAnalysis扩展了RichMapFunction{
私有最终AtomicLongMap mObjectAtomicLongMap;
公共路径分析()
{
mObjectAtomicLongMap=AtomicLongMap.create();
}
@凌驾
公共ApiLog映射(ApiLog值)引发异常{
//TODO自动生成的方法存根
返回null;
}
}
尝试实现以下功能:
import org.apache.flink.api.common.functions.RichMapFunction;
import com.google.common.util.concurrent.AtomicLongMap;
public class PathAnalysis extends RichMapFunction<ApiLog, ApiLog> {
private final AtomicLongMap<Object> mObjectAtomicLongMap;
public PathAnalysis()
{
mObjectAtomicLongMap = AtomicLongMap.create();
}
@Override
public ApiLog map(ApiLog value) throws Exception {
// TODO Auto-generated method stub
return null;
}
}
import org.apache.flink.api.common.functions.RichMapFunction;
导入com.google.common.util.concurrent.AtomicLongMap;
公共类PathAnalysis扩展了RichMapFunction{
私有最终AtomicLongMap mObjectAtomicLongMap;
公共路径分析()
{
mObjectAtomicLongMap=AtomicLongMap.create();
}
@凌驾
公共ApiLog映射(ApiLog值)引发异常{
//TODO自动生成的方法存根
返回null;
}
}
将AtomicLongMap标记为transient,并在函数将接收的open()调用中分配它(因为它是RichMapFunction)。比如:
public class PathAnalysis extends RichMapFunction<ApiLog, ApiLog> {
private transient AtomicLongMap<Object> mObjectAtomicLongMap;
public PathAnalysis() { }
@Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
mObjectAtomicLongMap = AtomicLongMap.create();
}
}
公共类路径分析扩展了RichMapFunction{
私有瞬态AtomicLongMap-mObjectAtomicLongMap;
公共路径分析(){}
@凌驾
公共void open(配置参数)引发异常{
super.open(参数);
mObjectAtomicLongMap=AtomicLongMap.create();
}
}
将AtomicLongMap标记为transient,并在函数将接收的open()调用中分配它(因为它是RichMapFunction)。比如:
public class PathAnalysis extends RichMapFunction<ApiLog, ApiLog> {
private transient AtomicLongMap<Object> mObjectAtomicLongMap;
public PathAnalysis() { }
@Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
mObjectAtomicLongMap = AtomicLongMap.create();
}
}
公共类路径分析扩展了RichMapFunction{
私有瞬态AtomicLongMap-mObjectAtomicLongMap;
公共路径分析(){}
@凌驾
公共void open(配置参数)引发异常{
super.open(参数);
mObjectAtomicLongMap=AtomicLongMap.create();
}
}
是我在构造中使用对象时遇到的问题吗?是我在构造中使用对象时遇到的问题吗?