Apache flink 在停靠的flink集群中，除了一个胖罐子以外的所有东西都会失败

我有一个容器化的flink集群，有一个独立的JobManager和两个任务管理器。当我提交一个只包含我需要的连接器和代码的胖jar时，一切都会正常运行，但是一个只包含我的代码的jar将失败——即使我已经将连接器jar添加到/opt/flink/lib，并且容器进程显示它们将被添加到类路径。当我尝试从IDE进行远程连接时，也会发生同样的故障；不用说，每次都要打包一个肥罐子会让开发体验很痛苦

我需要做什么才能让flink明白我在/opt/flink/lib中放置的额外jar应该用于所有新作业

类加载器异常：

org.apache.flink.streaming.runtime.tasks.StreamTaskException:无法 加载用户类： org.apache.flink.streaming.connectors.kafka.flinkkafka

flink Dockerfile：

FROM flink:1.7.2

ADD http://central.maven.org/maven2/org/apache/flink/flink-connector-kafka_2.12/1.7.2/flink-connector-kafka_2.12-1.7.2.jar /opt/flink/lib/
ADD http://central.maven.org/maven2/org/apache/flink/flink-avro-confluent-registry/1.7.2/flink-avro-confluent-registry-1.7.2.jar /opt/flink/lib/
ADD http://central.maven.org/maven2/org/apache/flink/flink-connector-elasticsearch6_2.12/1.7.2/flink-connector-elasticsearch6_2.12-1.7.2.jar /opt/flink/lib/

作业管理器容器：

root@2406b722dae1:/tmp# ps ax | more
  PID TTY      STAT   TIME COMMAND
    1 ?        Ssl    1:01 /docker-java-home/jre/bin/java -Xms1024m -Xmx1024m -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties -Dlogback.configurationFil
e=file:/opt/flink/conf/logback-console.xml -classpath /opt/flink/lib/flink-avro-confluent-registry-1.7.2.jar:/opt/flink/lib/flink-connector-elasticsearch6_2.12-1.7.2.jar:/o
pt/flink/lib/flink-connector-kafka_2.12-1.7.2.jar:/opt/flink/lib/flink-python_2.12-1.7.2.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/fl
ink/lib/flink-dist_2.12-1.7.2.jar::: org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint --configDir /opt/flink/conf --executionMode cluster

root@bd1aa6e35b5a:/tmp# ps ax | more
  PID TTY      STAT   TIME COMMAND
    1 ?        Ssl    0:28 /docker-java-home/jre/bin/java -XX:+UseG1GC -Xms922M -Xmx922M -XX:MaxDirectMemorySize=8388607T -Dlog4j.configuration=file:/opt/flink/conf/log4j-c
onsole.properties -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml -classpath /opt/flink/lib/flink-avro-confluent-registry-1.7.2.jar:/opt/flink/lib/flin
k-connector-elasticsearch6_2.12-1.7.2.jar:/opt/flink/lib/flink-connector-kafka_2.12-1.7.2.jar:/opt/flink/lib/flink-python_2.12-1.7.2.jar:/opt/flink/lib/log4j-1.2.17.jar:/op
t/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/flink/lib/flink-dist_2.12-1.7.2.jar::: org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir /opt/flink/conf

任务管理器容器：

root@2406b722dae1:/tmp# ps ax | more
  PID TTY      STAT   TIME COMMAND
    1 ?        Ssl    1:01 /docker-java-home/jre/bin/java -Xms1024m -Xmx1024m -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties -Dlogback.configurationFil
e=file:/opt/flink/conf/logback-console.xml -classpath /opt/flink/lib/flink-avro-confluent-registry-1.7.2.jar:/opt/flink/lib/flink-connector-elasticsearch6_2.12-1.7.2.jar:/o
pt/flink/lib/flink-connector-kafka_2.12-1.7.2.jar:/opt/flink/lib/flink-python_2.12-1.7.2.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/fl
ink/lib/flink-dist_2.12-1.7.2.jar::: org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint --configDir /opt/flink/conf --executionMode cluster

root@bd1aa6e35b5a:/tmp# ps ax | more
  PID TTY      STAT   TIME COMMAND
    1 ?        Ssl    0:28 /docker-java-home/jre/bin/java -XX:+UseG1GC -Xms922M -Xmx922M -XX:MaxDirectMemorySize=8388607T -Dlog4j.configuration=file:/opt/flink/conf/log4j-c
onsole.properties -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml -classpath /opt/flink/lib/flink-avro-confluent-registry-1.7.2.jar:/opt/flink/lib/flin
k-connector-elasticsearch6_2.12-1.7.2.jar:/opt/flink/lib/flink-connector-kafka_2.12-1.7.2.jar:/opt/flink/lib/flink-python_2.12-1.7.2.jar:/opt/flink/lib/log4j-1.2.17.jar:/op
t/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/flink/lib/flink-dist_2.12-1.7.2.jar::: org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir /opt/flink/conf

/在运行的JobManager和TaskManager上选择/flink/lib：

root@bd1aa6e35b5a:/opt/flink/lib# ls -lh
total 84M
-rw------- 1 root  root  2.7M Feb 11 16:25 flink-avro-confluent-registry-1.7.2.jar
-rw------- 1 root  root   30K Feb 11 16:21 flink-connector-elasticsearch6_2.12-1.7.2.jar
-rw------- 1 root  root   67K Feb 11 16:24 flink-connector-kafka_2.12-1.7.2.jar
-rw-r--r-- 1 flink flink  81M Feb 11 14:50 flink-dist_2.12-1.7.2.jar
-rw-r--r-- 1 flink flink 139K Feb 11 14:49 flink-python_2.12-1.7.2.jar
-rw-rw-r-- 1 flink flink 479K Feb 11 14:32 log4j-1.2.17.jar
-rw-rw-r-- 1 flink flink 9.7K Feb 11 14:32 slf4j-log4j12-1.7.15.jar

以防有任何疑问--该类位于预期的jar中：

root@bd1aa6e35b5a:/opt/flink/lib# unzip -l flink-connector-kafka_2.12-1.7.2.jar | grep FlinkKafkaConsumer
    14272  2019-02-12 00:24   org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer.class

---

我想明白了。我的第一个问题是flink是在flink用户下运行的，从清单中可以看出，我在Dockerfile中添加的文件归root所有。在那之后我遇到的问题是，sbt程序集为您绑定了所有可传递的依赖项，您必须在Dockerfile中手动执行这些操作（fat jar文件大小与我添加的文件之间的差异应该会泄露这一点）

我修复了Dockerfile的所有权问题和Kafka连接器的可传递依赖项（elasticsearch可能仍然有许多缺少的依赖项）：