Apache flink Flink可查询状态错误
我试图在Flink(版本1.4.2)上使用queryable状态,但不幸的是,我一直收到以下错误:Apache flink Flink可查询状态错误,apache-flink,flink-streaming,Apache Flink,Flink Streaming,我试图在Flink(版本1.4.2)上使用queryable状态,但不幸的是,我一直收到以下错误: INFO my.test.flink.QueryableState - Params are a96438fa12879b7598c9cf32684e2669, kafka-cluster_jobmanager_1, 6123 INFO my.test.flink.QueryableState - Before the call java.util.concurrent.Completab
INFO my.test.flink.QueryableState - Params are a96438fa12879b7598c9cf32684e2669, kafka-cluster_jobmanager_1, 6123
INFO my.test.flink.QueryableState - Before the call java.util.concurrent.CompletableFuture@26aa12dd[Not completed]
java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: readerIndex(0) + length(4) exceeds writerIndex(0): PooledUnsafeDirectByteBuf(ridx: 0, widx: 0, cap: 0)
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at my.test.flink.QueryableState.main(QueryableState.java:67)
Caused by: java.lang.IndexOutOfBoundsException: readerIndex(0) + length(4) exceeds writerIndex(0): PooledUnsafeDirectByteBuf(ridx: 0, widx: 0, cap: 0)
at org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBuf.checkReadableBytes(AbstractByteBuf.java:1166)
at org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBuf.readInt(AbstractByteBuf.java:619)
at org.apache.flink.queryablestate.network.messages.MessageSerializer.deserializeHeader(MessageSerializer.java:231)
at org.apache.flink.queryablestate.network.ClientHandler.channelRead(ClientHandler.java:76)
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
在客户端,我使用flink-queryable-state-client-java_2_11.jar,可查询客户端的相关代码部分如下
QueryableStateClient client = new QueryableStateClient(jobManagerHost, jobManagerPort);
TypeInformation<MyEvent> typeInformation = TypeInformation.of(new TypeHint<MyEvent>() {});
ListStateDescriptor<MyEvent> descriptor = new ListStateDescriptor<MyEvent>("myEvents",
typeInformation.createSerializer(new ExecutionConfig()));
CompletableFuture<ListState<MyEvent>> resultFuture =
client.getKvState(JobID.fromHexString(jobIdParam),"myEvents", "1",
BasicTypeInfo.STRING_TYPE_INFO , descriptor );
logger.info("Before the call " + resultFuture);
try {
logger.info("Finished"+ resultFuture.get());
} catch(Exception ex) {
ex.printStackTrace();
}
QueryableStateClient=newqueryablestateclient(jobManagerHost,jobManagerPort);
TypeInformation-TypeInformation=TypeInformation.of(new-TypeHint(){});
ListStateDescriptor描述符=新的ListStateDescriptor(“myEvents”,
createSerializer(新的ExecutionConfig());
完全未来结果未来=
client.getKvState(JobID.fromHexString(jobIdParam),“myEvents”,“1”,
BasicTypeInfo.STRING\u TYPE\u INFO,描述符);
logger.info(“调用前”+resultFuture);
试一试{
info(“Finished”+resultFuture.get());
}捕获(例外情况除外){
例如printStackTrace();
}
最后,在Flink上运行的作业配置了ListState,如下所示。请注意,数据是通过字符串键入ListState的
TypeInformation<MyEvent> typeInformation = TypeInformation.of(new TypeHint<MyEvent>() {});
ListStateDescriptor<MyEvent> eventState =
new ListStateDescriptor<MyEvent>("myEvents",typeInformation);
eventState.setQueryable("myEvents");
eventListState = getRuntimeContext().getListState(eventState);
TypeInformation-TypeInformation=TypeInformation.of(new-TypeHint(){});
ListStateDescriptor事件状态=
新的ListStateDescriptor(“myEvents”,类型信息);
eventState.setQueryable(“myEvents”);
eventListState=getRuntimeContext().getListState(eventState);
在我看来,这似乎是一个序列化错误,但我不知道我需要做什么来修复它。有人知道上面的代码可能有什么问题吗?我遗漏了什么吗?我在更新Flink 1.4时遇到了完全相同的问题。如果我没记错的话,重要的部分是正确处理可完成的未来——你不能直接调用get() 有关工作示例,请参见,其关键部分如下所示:
try {
CompletableFuture<FoldingState<BumpEvent, Long>> resultFuture =
client.getKvState(jobId, EventCountJob.ITEM_COUNTS, key,
BasicTypeInfo.STRING_TYPE_INFO, countingState);
resultFuture.thenAccept(response -> {
try {
Long count = response.get();
// now we could do something with the value
} catch (Exception e) {
e.printStackTrace();
}
});
resultFuture.get(5, TimeUnit.SECONDS);
} catch (Exception e) {
e.printStackTrace();
}
试试看{
完全未来结果未来=
client.getKvState(jobId,EventCountJob.ITEM_COUNTS,key,
基本类型信息。字符串类型信息,计数状态);
结果未来。然后接受(响应->{
试一试{
Long count=response.get();
//现在我们可以用这个值做点什么
}捕获(例外e){
e、 printStackTrace();
}
});
resultFuture.get(5,时间单位秒);
}捕获(例外e){
e、 printStackTrace();
}
实际上,问题是我的印象是我应该在jobmanager上使用RPC端口,而不是端口9069,后者是taskmanager上的默认查询代理端口。谢谢,不过关于CompletableFuture的信息也有助于获得最终结果。