Apache spark Pyspark应用程序无法在群集模式下运行

当我运行以下spark代码时，很少有运行会因为TimeoutException而失败

测试火花应用程序py

print("importing libraries")
from pyspark import SparkContext
from datetime import datetime
print("finish importing libraries")
def printMessage(msg):
    print (datetime.now().strftime("%Y/%m/%d %H:%M:%S") +" - " + msg)
if __name__ == "__main__":
    sc = SparkContext()
    printMessage("Application start")
    printMessage("Application end")

TimeoutException

20/10/21 11:10:59 INFO ApplicationMaster: Starting the user application in a separate Thread
20/10/21 11:10:59 INFO ApplicationMaster: Waiting for spark context initialization...
20/10/21 11:11:29 DEBUG Client: IPC Client (989889899) connection to host1.mysite.com/182.25.26.29:8020 from streamf09w: closed
20/10/21 11:11:29 DEBUG Client: IPC Client (989889899) connection to host1.mysite.com/182.25.26.29:8020 from streamf09w: stopped, remaining connections 0
20/10/21 11:12:39 ERROR ApplicationMaster: Uncaught exception: 
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
    at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
    at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:201)
    at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:345)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:815)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
    at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:814)
    at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:259)
    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:839)
    at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
20/10/21 11:12:39 INFO ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: Uncaught exception: java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds])
20/10/21 11:12:39 DEBUG ApplicationMaster: shutting down user thread

容器日志：

importing libraries
finish importing libraries
ERROR:root:Exception while sending command.
Traceback (most recent call last):
  File "/opt/hadoop/data/08/hadoop/yarn/local/usercache/streamf10w/appcache/application_1603777923068_25093/container_e72_1603777923068_25184_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1159, in send_command
    raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/hadoop/data/08/hadoop/yarn/local/usercache/streamf09w/appcache/application_1603777923068_25093/container_e72_1603777923068_25184_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 985, in send_command
    response = connection.send_command(command)
  File "/opt/hadoop/data/08/hadoop/yarn/local/usercache/streamf10w/appcache/application_1603777923068_25093/container_e72_1603777923068_25184_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1164, in send_command
    "Error while receiving", e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving
Traceback (most recent call last):
  File "test_spark_app.py", line 9, in <module>
    sc = SparkContext()
  File "/opt/hadoop/data/08/hadoop/yarn/local/usercache/streamf09w/appcache/application_1603777923068_25093/container_e72_1603777923068_25184_01_000001/pyspark.zip/pyspark/context.py", line 119, in __init__

导入库
完成导入库
错误：root:发送命令时发生异常。
回溯（最近一次呼叫最后一次）：
文件“/opt/hadoop/data/08/hadoop/thread/local/usercache/streamf10w/appcache/application_1603777923068_25093/container_e72_1603777923068_25184_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py”，在send_命令的第1159行
raise Py4JNetworkError（“来自Java端的答案为空”）
py4j.protocol.Py4JNetworkError:来自Java端的答案为空
在处理上述异常期间，发生了另一个异常：
回溯（最近一次呼叫最后一次）：
文件“/opt/hadoop/data/08/hadoop/thread/local/usercache/streamf09w/appcache/application_1603777923068_25093/container_e72_1603777923068_25184_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py”，第985行，在send_命令中
响应=连接。发送命令（命令）
文件“/opt/hadoop/data/08/hadoop/thread/local/usercache/streamf10w/appcache/application_1603777923068_25093/container_e72_1603777923068_25184_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py”，第1164行，在send_命令中
“接收时出错”，e，接收时出现协议错误）
py4j.protocol.Py4JNetworkError:接收时出错
回溯（最近一次呼叫最后一次）：
文件“test_spark_app.py”，第9行，在
sc=SparkContext（）
文件“/opt/hadoop/data/08/hadoop/thread/local/usercache/streamf09w/appcache/application\u 1603777923068\u 25093/container\u e72\u 1603777923068\u 25184\u 01\u000001/pyspark.zip/pyspark/context.py”，第119行，在__

资源管理器和节点管理器可以相互交互吗？你确定没有网络问题吗？是的，没有网络问题。这是在中间发生的。已使用IPV6启用群集。我怀疑这会引起问题。