Apache flink 失败消息:使用apache flink 1.11时,检查点在完成之前已过期
最近我在kubernetes集群中将ApacheFlink升级到1.11版,但今天我发现一个任务检查点总是失败。此任务从RabbitMQ读取数据并计算结果,然后调用HTTP请求将数据保存到MySQL。这是taskmanager错误日志输出:Apache flink 失败消息:使用apache flink 1.11时,检查点在完成之前已过期,apache-flink,Apache Flink,最近我在kubernetes集群中将ApacheFlink升级到1.11版,但今天我发现一个任务检查点总是失败。此任务从RabbitMQ读取数据并计算结果,然后调用HTTP请求将数据保存到MySQL。这是taskmanager错误日志输出: 2020-08-21 15:53:28 org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold. at org.apac
2020-08-21 15:53:28
org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold.
at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleJobLevelCheckpointException(CheckpointFailureManager.java:66)
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1626)
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1603)
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.access$600(CheckpointCoordinator.java:90)
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:1736)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
这是Apache Flink UI错误消息:
Failure Message: Checkpoint expired before completing.
作业总是在一段时间后重新启动。我有2个任务和另一个任务检查点保持成功。那么问题出在哪里?我应该怎么做才能解决这个问题
这是我的flink配置:
public static void initEnv(StreamExecutionEnvironment env) {
env.setParallelism(1);
env.enableCheckpointing(10000);
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(1000);
env.getCheckpointConfig().setCheckpointTimeout(10000);
env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
StateBackend stateBackend = new FsStateBackend("file:///opt/flink/data");
env.setStateBackend(stateBackend);
}
尝试增加检查点的超时时间:
public static void initEnv(StreamExecutionEnvironment env) {
env.setParallelism(1);
env.enableCheckpointing(120000);
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(1000);
env.getCheckpointConfig().setCheckpointTimeout(120000);
env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
StateBackend stateBackend = new FsStateBackend("file:///opt/flink/data");
env.setStateBackend(stateBackend);
}
尝试增加检查点的超时时间:
public static void initEnv(StreamExecutionEnvironment env) {
env.setParallelism(1);
env.enableCheckpointing(120000);
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(1000);
env.getCheckpointConfig().setCheckpointTimeout(120000);
env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
StateBackend stateBackend = new FsStateBackend("file:///opt/flink/data");
env.setStateBackend(stateBackend);
}
请指定应用程序中的检查点超时是多少?我可能错了,但从web ui看,它似乎是10秒-你能试着把它增加到30秒,看看它是否能够完成吗?谢谢。我已经在问题@Mikalailushcytski中添加了我的配置。你能尝试将检查点超时增加到30秒吗?似乎没有足够的时间让任务完成检查点。您可以将检查点超时设置为任何值。通常,这取决于操作员执行检查点所需的预期时间——例如,根据我的经验,我们在应用程序中有一个2分钟的检查点超时,写入外部存储器。这只是设置了检查点被取消时的上限-如果它在15秒内完成,那么它就完全正常了。我增加了超时和问题solved@MikalaiLushchytskiCould请指定应用程序中的检查点超时是多少?我可能错了,但从web ui看,它似乎是10秒-你能试着把它增加到30秒,看看它是否能够完成吗?谢谢。我已经在问题@Mikalailushcytski中添加了我的配置。你能尝试将检查点超时增加到30秒吗?似乎没有足够的时间让任务完成检查点。您可以将检查点超时设置为任何值。通常,这取决于操作员执行检查点所需的预期时间——例如,根据我的经验,我们在应用程序中有一个2分钟的检查点超时,写入外部存储器。这只是设置了检查点被取消时的上限-如果它在15秒内完成,那么它就完全正常了。我增加了超时和问题solved@MikalaiLushchytski