无法连接在docker中运行的Apache Spark

我正在尝试通过主机系统连接docker中运行的Spark群集。我尝试了python脚本和spark shell都得到了相同的结果：

在Docker内部

park-master_1  | 20/07/24 10:13:26 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
spark-master_1  | java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible: stream classdesc serialVersionUID = 1574364215946805297, local class serialVersionUID = 6543101073799644159
spark-master_1  |   at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
spark-master_1  |   at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
spark-master_1  |   at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
spark-master_1  |   at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
spark-master_1  |   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
spark-master_1  |   at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
spark-master_1  |   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
spark-master_1  |   at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
spark-master_1  |   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
spark-master_1  |   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
spark-master_1  |   at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
spark-master_1  |   at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:108)
spark-master_1  |   at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$deserialize$1$$anonfun$apply$1.apply(Nett

在主机系统的命令行上运行

spark shell

，会出现以下错误：

➜

火花壳命令

git:(master) ✗ docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
dfe3d47790ee        spydernaz/spark-worker:latest   "/bin/bash /start-wo…"   42 hours ago        Up 23 minutes       0.0.0.0:32769->8081/tcp                                    docker-spark-cluster_spark-worker_2
c5e36b94efdd        spydernaz/spark-worker:latest   "/bin/bash /start-wo…"   42 hours ago        Up 23 minutes       0.0.0.0:32768->8081/tcp                                    docker-spark-cluster_spark-worker_3
60f3d29e9059        spydernaz/spark-worker:latest   "/bin/bash /start-wo…"   42 hours ago        Up 23 minutes       0.0.0.0:32770->8081/tcp                                    docker-spark-cluster_spark-worker_1
d11c67d462fb        spydernaz/spark-master:latest   "/bin/bash /start-ma…"   42 hours ago        Up 23 minutes       6066/tcp, 0.0.0.0:7077->7077/tcp, 0.0.0.0:9090->8080/tcp   docker-spark-cluster_spark-master_1
➜  docker-spark-cluster git:(master) ✗ 

---

火花壳-主spark://localhost:7077

正如@koiralo在评论中已经提到的，这是由于本地和服务器上运行的pySpark版本不同造成的

---

出现了相同的错误，并且在两个位置的版本匹配后修复了该错误。

正如@koiralo在评论中已经提到的，这是由于本地和服务器上运行的pySpark版本不同造成的

---

有相同的错误，在两个地方的版本匹配后就被修复了。

docker中运行的spark和您在本地使用的spark的版本是什么？@koiralo在docker pySpark下说它是2.4.3，主机是3.0.0。我确实改变了Spark的版本。我正在使用此群集设置

https://github.com/mvillarrealb/docker-spark-cluster.git

您是否尝试在这两个容器中使用相同的版本？@koiralo是的，我跨容器修复了它，现在可以使用了。谢谢docker中运行的spark版本是什么，你在本地使用的spark版本是什么？@docker pySpark下的koiralo说它是2.4.3，主机是3.0.0。我确实改变了Spark的版本。我正在使用此群集设置

https://github.com/mvillarrealb/docker-spark-cluster.git

您是否尝试在这两个容器中使用相同的版本？@koiralo是的，我跨容器修复了它，现在可以使用了。谢谢

git:(master) ✗ docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
dfe3d47790ee        spydernaz/spark-worker:latest   "/bin/bash /start-wo…"   42 hours ago        Up 23 minutes       0.0.0.0:32769->8081/tcp                                    docker-spark-cluster_spark-worker_2
c5e36b94efdd        spydernaz/spark-worker:latest   "/bin/bash /start-wo…"   42 hours ago        Up 23 minutes       0.0.0.0:32768->8081/tcp                                    docker-spark-cluster_spark-worker_3
60f3d29e9059        spydernaz/spark-worker:latest   "/bin/bash /start-wo…"   42 hours ago        Up 23 minutes       0.0.0.0:32770->8081/tcp                                    docker-spark-cluster_spark-worker_1
d11c67d462fb        spydernaz/spark-master:latest   "/bin/bash /start-ma…"   42 hours ago        Up 23 minutes       6066/tcp, 0.0.0.0:7077->7077/tcp, 0.0.0.0:9090->8080/tcp   docker-spark-cluster_spark-master_1
➜  docker-spark-cluster git:(master) ✗