Apache flink Flink支持BroadcastState的状态TTL吗?
在Flink 1.8.1中,我尝试将状态TTL应用于BroadcastState(使用MapStateDescriptor),如下所示: (Holder是一个包含私有int变量“deger”的POJO)Apache flink Flink支持BroadcastState的状态TTL吗?,apache-flink,Apache Flink,在Flink 1.8.1中,我尝试将状态TTL应用于BroadcastState(使用MapStateDescriptor),如下所示: (Holder是一个包含私有int变量“deger”的POJO) 。。。 StreamExecutionEnvironment envStream=StreamExecutionEnvironment.getExecutionEnvironment(); StateBackend StateBackend=新的FsStateBackend(“file://..
。。。
StreamExecutionEnvironment envStream=StreamExecutionEnvironment.getExecutionEnvironment();
StateBackend StateBackend=新的FsStateBackend(“file://.....");
setStateBackend(stateBackend);
envStream.enableCheckpointing(1_000L,CheckpointingMode.恰好\u一次);
...
MapStateDescriptor clientMapStateDescriptor=新的MapStateDescriptor(
“客户端广播状态”,
基本类型信息.INT\u类型信息,
TypeInformation.of(新的TypeHint(){})
);
StateTtlConfig ttlConfig=StateTtlConfig
.新造船工(时间.秒(3))
//.cleanupfullshopshot()
//.cleanupInBackground()
.增量清理(100,false)
.setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
.setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
.build();
clientMapStateDescriptor.enableTimeToLive(ttlConfig);
DataStream clientDataStream=envStream.fromCollection(clientList);
//clientDataStream.print(“clientDataStream”);
BroadcastStream clientBroadcastStream=clientDataStream
.广播(clientMapStateDescriptor);
列表持有者列表=新数组列表(计数);
for(int i=0;i
holderList有足够的实例来测试状态中的条目是否被逐出
但是广播状态
中的条目不会过期
我尝试过的事情:
- 使用不同的状态后端(
)fsstatebend
- 启用检查点
- 显式访问映射状态值
一种解决方案是广播由接收者解释为命令的流记录,以使广播状态的早期记录过期 所以,
BroadcastState
不支持StateTTL
,对吗?在processBroadcastElement
方法中处理逐出的缺点是,除非BroadcastState
支持StateTL
,否则在收到新的广播元素之前,实际上没有机会逐出条目,所以,BroadcastState
不支持StateTL
,对吗?在processBroadcastElement
方法中处理逐出的缺点是,在接收到新的广播元素之前,实际上没有机会逐出条目,除非BroadcastState
支持StateTL
...
StreamExecutionEnvironment envStream = StreamExecutionEnvironment.getExecutionEnvironment();
StateBackend stateBackend = new FsStateBackend("file://.....");
envStream.setStateBackend(stateBackend);
envStream.enableCheckpointing(1_000L, CheckpointingMode.EXACTLY_ONCE);
...
MapStateDescriptor<Integer, Client> clientMapStateDescriptor = new MapStateDescriptor<>(
"ClientBroadcastState",
BasicTypeInfo.INT_TYPE_INFO,
TypeInformation.of(new TypeHint<Client>() {})
);
StateTtlConfig ttlConfig = StateTtlConfig
.newBuilder(Time.seconds(3))
// .cleanupFullSnapshot()
// .cleanupInBackground()
.cleanupIncrementally(100, false)
.setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
.setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
.build();
clientMapStateDescriptor.enableTimeToLive(ttlConfig);
DataStream<Client> clientDataStream = envStream.fromCollection(clientList);
// clientDataStream.print("clientDataStream");
BroadcastStream<Client> clientBroadcastStream = clientDataStream
.broadcast(clientMapStateDescriptor);
List<Holder> holderList = new ArrayList<>(count);
for(int i = 0; i < count; i++) {
holderList.add(new Holder(i));
}
DataStream<Holder> integerHolderDataStream = envStream.fromCollection(holderList);
BroadcastConnectedStream<Holder, Client> connectedStreams = integerHolderDataStream
.keyBy("deger")
.connect(clientBroadcastStream);
SingleOutputStreamOperator<Row> operator = connectedStreams.process(new KeyedBroadcastProcessFunction<Integer, Holder, Client, Row>() {
@Override
public void processElement(Holder value, ReadOnlyContext ctx, Collector<Row> out) throws Exception {
for (Map.Entry<Integer, Client> entry : ctx.getBroadcastState(clientMapStateDescriptor).immutableEntries()) {
Client c = ctx.getBroadcastState(clientMapStateDescriptor).get(entry.getKey());
System.out.println(value.getDeger() + " - " + c);
}
Thread.sleep(1000L);
}
@Override
public void processBroadcastElement(Client value, Context ctx, Collector<Row> out) throws Exception {
ctx.getBroadcastState(clientMapStateDescriptor).put(value.getId(), value);
}
});
...