Java Apache Beam Spark runner侧输入导致信号项

我希望使用侧输入将一些配置传递到我的管道，但是在本地spark群集（spark版本2.4.7，1个主机，1个工作机，在本地主机上运行）上创建

PCollectionView

后，驱动程序命令关闭。但是，这在DirectRunner上非常有效

我试图将代码剥离到最基本的部分（见下文）。在spark cluster上运行时，问题仍然存在。DirectRunner仍然可以正常工作

spark集群确实接受作业，我成功地运行了一个“hello world”管道，该管道顺利完成

这里发生了什么

贴在下面的日志

Environment:
------------
Beam: 2.25
SparkRunner: 2.25
Java version: 11.0.9-ea
Maven Compiler Source: 1.8
Maven Compiler Target: 1.8
Spark version: 2.4.7 

//管道
专用静态PipelineResult运行管道（PipelineOptions选项）{
Pipeline p=Pipeline.create（选项）；
PCollectionView模式=p
.apply（“虚拟表格模式生成器”，Create.of（“这是一个字符串”））
.apply（“Collect”，View.asSingleton（））；
P
.apply（“Hello world”，Create.of（“Hello world”））
.apply（“Side input test”，ParDo.of（DummyFn.builder（）.setSchemaView（schema.build（））。with Side input（“schema”，schema））
.apply（ConsoleIO.create（））；
返回p.run（）；
}

打印侧面输入的简单FN @自动值 公共抽象类DummyFn扩展了DoFn{ 私有最终静态记录器LOG=LoggerFactory.getLogger（DummyFn.class）； 公共静态生成器（）{ 返回新的org.odp.beam.io.fn.AutoValue_DummyFn.Builder（）； } 公共摘要PCollectionView getSchemaView（）； @过程元素 public void processElement（@Element字符串元素， 输出接收器输出， ProcessContext（上下文）{ 字符串schema=context.sideInput（getSchemaView（））； LOG.warn（schema.toString（））； output（element.toUpperCase（））； } @AutoValue.Builder 公共抽象静态类生成器{ 公共摘要生成器setSchemaView（PCollectionView值）； 公共抽象DummyFn build（）； } }

//打印toString方法输出的简单PTransform
公共类控制台扩展了PTTransform{
公共静态控制台创建（）{
返回新控制台（）；
}
@凌驾
公共PDone扩展（PCollection输入）{
输入
.apply（“打印元素”，第页，共页（新打印元素fn（））；
返回PDone.in（input.getPipeline（））；
}
静态类PrintElementFn扩展了DoFn{
@DoFn.ProcessElement
public void processElement（@Element T Element，ProcessContext context）引发异常{
System.out.println（element.toString（））；
}
}
}

$spark提交\
--类org.odp.beam.extractors.csvtocdrawdextractor\
--冗长的\
--驱动程序内存4G\
--执行器存储器4G\
--执行器核心总数4\
--部署模式客户端\
--监督\
--conf spark.dynamicAllocation.enabled=false\
--conf spark.network.timeout=420000\
--主人spark://192.168.10.172:7077 \
target/beam-poc-0.1-shaded.jar\
--跑者=斯巴克跑者
使用属性文件：null
20/11/10 15:46:44警告Utils：您的主机名localhost.localdomain解析为环回地址：127.0.0.1；改用192.168.10.172（在接口enp7s0上）
20/11/10 15:46:44警告Utils：如果需要绑定到其他地址，请设置SPARK_LOCAL_IP
警告：发生了非法的反射访问操作
警告：org.apache.spark.unsafe.Platform（文件：/home/tom/app/spark/spark-2.4.7-bin-hadoop2.7/jars/spark-unsafe_2.11-2.4.7.jar）对java.nio.Bits.unaligned（）方法的非法反射访问
警告：请考虑将此报告给Or.ApHEC.SPARK.UNSAFE平台的维护者。
警告：使用--invalize access=warn以启用对进一步非法访问操作的警告
警告：所有非法访问操作将在未来版本中被拒绝
解析的参数：
主人spark://192.168.10.172:7077
部署模式客户端
执行器存储器4G
执行器核心为空
共4个核心
属性文件为空
驾驶员记忆4G
驱动核心无效
driverExtraClassPath null
driverExtraLibraryPath为空
driverExtraJavaOptions为空
监督真实
队列空
numExecutors null
文件为空
pyFiles null
档案无效
mainClass org.odp.beam.extractors.csvtocdrawdextractor
主资源文件：/home/tom/project/odf/beam-poc/target/beam-poc-0.1-shaded.jar
名称org.odp.beam.extractors.csvtocdrawdextractor
childArgs[--runner=SparkRunner]
空罐子
包为空
PackagesExclutions空
存储库空
冗长真实
使用的火花特性，包括通过
--conf和属性文件null中的值：
（spark.network.timeout，420000）
（spark.driver.memory，4G）
（spark.DynamicLocation.enabled，false）
20/11/10 15:46:45警告NativeCodeLoader:无法为您的平台加载本机hadoop库。。。在适用的情况下使用内置java类
主要类别：
org.odp.beam.extractors.csvtocdrawdextractor
论据：
--跑者=斯巴克跑者
Spark配置：
（spark.jars，文件：/home/tom/project/odf/beam-poc/target/beam-poc-0.1-shaded.jar）
（spark.app.name，org.odp.beam.extractors.csvtocdrawdextractor）
（火花芯最大值4）
（spark.network.timeout，420000）
（spark.driver.memory，4G）
（spark.submit.deployMode，客户端）
（斯帕克·马斯特，spark://192.168.10.172:7077)
（spark.executor.memory，4G）
（spark.DynamicLocation.enabled，false）
类路径元素：
文件：/home/tom/project/odf/beam-poc/target/beam-poc-0.1-shaded.jar
log4j:警告：找不到记录器（o）的附加器
Spark Executor Command: "/usr/lib/jvm/java-11-openjdk-11.0.9.6-0.0.ea.fc33.x86_64/bin/java" "-cp" "/home/tom/app/spark/spark/conf/:/home/tom/app/spark/spark/jars/*" "-Xmx4096M" "-Dspark.driver.port=35103" "-Dspark.network.timeout=420000" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@fedora:35103" "--executor-id" "0" "--hostname" "192.168.10.172" "--cores" "4" "--app-id" "app-20201110154647-0020" "--worker-url" "spark://Worker@192.168.10.172:45535"
========================================

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/11/10 15:46:48 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 360353@localhost.localdomain
20/11/10 15:46:48 INFO SignalUtils: Registered signal handler for TERM
20/11/10 15:46:48 INFO SignalUtils: Registered signal handler for HUP
20/11/10 15:46:48 INFO SignalUtils: Registered signal handler for INT
20/11/10 15:46:48 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 192.168.10.172 instead (on interface enp7s0)
20/11/10 15:46:48 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/home/tom/app/spark/spark-2.4.7-bin-hadoop2.7/jars/spark-unsafe_2.11-2.4.7.jar) to method java.nio.Bits.unaligned()
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
20/11/10 15:46:48 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/10 15:46:48 INFO SecurityManager: Changing view acls to: tom
20/11/10 15:46:48 INFO SecurityManager: Changing modify acls to: tom
20/11/10 15:46:48 INFO SecurityManager: Changing view acls groups to: 
20/11/10 15:46:48 INFO SecurityManager: Changing modify acls groups to: 
20/11/10 15:46:48 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(tom); groups with view permissions: Set(); users  with modify permissions: Set(tom); groups with modify permissions: Set()
20/11/10 15:46:49 INFO TransportClientFactory: Successfully created connection to fedora/192.168.10.172:35103 after 54 ms (0 ms spent in bootstraps)
20/11/10 15:46:49 INFO SecurityManager: Changing view acls to: tom
20/11/10 15:46:49 INFO SecurityManager: Changing modify acls to: tom
20/11/10 15:46:49 INFO SecurityManager: Changing view acls groups to: 
20/11/10 15:46:49 INFO SecurityManager: Changing modify acls groups to: 
20/11/10 15:46:49 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(tom); groups with view permissions: Set(); users  with modify permissions: Set(tom); groups with modify permissions: Set()
20/11/10 15:46:49 INFO TransportClientFactory: Successfully created connection to fedora/192.168.10.172:35103 after 4 ms (0 ms spent in bootstraps)
20/11/10 15:46:49 INFO DiskBlockManager: Created local directory at /tmp/spark-0e47fa97-8714-4e8e-950e-b1032fe36995/executor-e7667d04-198d-4144-8897-ddada0bfd1de/blockmgr-019262b3-4d3e-4158-b984-ff85c0846191
20/11/10 15:46:49 INFO MemoryStore: MemoryStore started with capacity 2.2 GB
20/11/10 15:46:49 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@fedora:35103
20/11/10 15:46:49 INFO WorkerWatcher: Connecting to worker spark://Worker@192.168.10.172:45535
20/11/10 15:46:49 INFO TransportClientFactory: Successfully created connection to /192.168.10.172:45535 after 2 ms (0 ms spent in bootstraps)
20/11/10 15:46:49 INFO WorkerWatcher: Successfully connected to spark://Worker@192.168.10.172:45535
20/11/10 15:46:49 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
20/11/10 15:46:49 INFO Executor: Starting executor ID 0 on host 192.168.10.172
20/11/10 15:46:49 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43781.
20/11/10 15:46:49 INFO NettyBlockTransferService: Server created on 192.168.10.172:43781
20/11/10 15:46:49 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/11/10 15:46:49 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(0, 192.168.10.172, 43781, None)
20/11/10 15:46:49 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(0, 192.168.10.172, 43781, None)
20/11/10 15:46:49 INFO BlockManager: Initialized BlockManager: BlockManagerId(0, 192.168.10.172, 43781, None)
20/11/10 15:46:49 INFO CoarseGrainedExecutorBackend: Got assigned task 0
20/11/10 15:46:49 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
20/11/10 15:46:49 INFO Executor: Fetching spark://fedora:35103/jars/beam-poc-0.1-shaded.jar with timestamp 1605019607514
20/11/10 15:46:49 INFO TransportClientFactory: Successfully created connection to fedora/192.168.10.172:35103 after 2 ms (0 ms spent in bootstraps)
20/11/10 15:46:49 INFO Utils: Fetching spark://fedora:35103/jars/beam-poc-0.1-shaded.jar to /tmp/spark-0e47fa97-8714-4e8e-950e-b1032fe36995/executor-e7667d04-198d-4144-8897-ddada0bfd1de/spark-62556d02-a044-4c2c-8f97-c7f25ef3e337/fetchFileTemp6325880319900581024.tmp
20/11/10 15:46:49 INFO Utils: Copying /tmp/spark-0e47fa97-8714-4e8e-950e-b1032fe36995/executor-e7667d04-198d-4144-8897-ddada0bfd1de/spark-62556d02-a044-4c2c-8f97-c7f25ef3e337/2058038551605019607514_cache to /home/tom/app/spark/spark-2.4.7-bin-hadoop2.7/work/app-20201110154647-0020/0/./beam-poc-0.1-shaded.jar
20/11/10 15:46:50 INFO Executor: Adding file:/home/tom/app/spark/spark-2.4.7-bin-hadoop2.7/work/app-20201110154647-0020/0/./beam-poc-0.1-shaded.jar to class loader
20/11/10 15:46:50 INFO TorrentBroadcast: Started reading broadcast variable 0
20/11/10 15:46:50 INFO TransportClientFactory: Successfully created connection to fedora/192.168.10.172:33169 after 2 ms (0 ms spent in bootstraps)
20/11/10 15:46:50 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 6.8 KB, free 2.2 GB)
20/11/10 15:46:50 INFO TorrentBroadcast: Reading broadcast variable 0 took 112 ms
20/11/10 15:46:50 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 16.2 KB, free 2.2 GB)
20/11/10 15:46:51 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 6312 bytes result sent to driver
20/11/10 15:46:51 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
20/11/10 15:46:51 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
tdown