Hadoop HA中的Spark:java.lang.NoSuchMethodError:org.apache.curator.utils.PathUtils.validatePath(Ljava/lang/String;)Ljava/lang/String;
我正在尝试在独立模式下为Spark 1.5.2设置HA(通过zookeeper)。启动spark时,我遇到了这个异常。看起来是不兼容的问题。然而,我不知道如何继续Hadoop HA中的Spark:java.lang.NoSuchMethodError:org.apache.curator.utils.PathUtils.validatePath(Ljava/lang/String;)Ljava/lang/String;,hadoop,apache-spark,nosuchmethoderror,Hadoop,Apache Spark,Nosuchmethoderror,我正在尝试在独立模式下为Spark 1.5.2设置HA(通过zookeeper)。启动spark时,我遇到了这个异常。看起来是不兼容的问题。然而,我不知道如何继续 15/12/14 10:42:27 ERROR ErrorMonitor: Uncaught fatal error from thread [sparkMaster-akka.actor.default-dispatcher-3] shutting down ActorSystem [sparkMaster] java.lang.N
15/12/14 10:42:27 ERROR ErrorMonitor: Uncaught fatal error from thread [sparkMaster-akka.actor.default-dispatcher-3] shutting down ActorSystem [sparkMaster]
java.lang.NoSuchMethodError: org.apache.curator.utils.PathUtils.validatePath(Ljava/lang/String;)Ljava/lang/String;
at org.apache.curator.framework.recipes.leader.LeaderLatch.<init>(LeaderLatch.java:147)
at org.apache.curator.framework.recipes.leader.LeaderLatch.<init>(LeaderLatch.java:125)
at org.apache.spark.deploy.master.ZooKeeperLeaderElectionAgent.start(ZooKeeperLeaderElectionAgent.scala:39)
at org.apache.spark.deploy.master.ZooKeeperLeaderElectionAgent.<init>(ZooKeeperLeaderElectionAgent.scala:34)
at org.apache.spark.deploy.master.ZooKeeperRecoveryModeFactory.createLeaderElectionAgent(RecoveryModeFactory.scala:74)
at org.apache.spark.deploy.master.Master.onStart(Master.scala:165)
at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$preStart$1.apply$mcV$sp(AkkaRpcEnv.scala:100)
at org.apache.spark.rpc.akka.AkkaRpcEnv.org$apache$spark$rpc$akka$AkkaRpcEnv$$safelyCall(AkkaRpcEnv.scala:197)
at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.preStart(AkkaRpcEnv.scala:99)
at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.aroundPreStart(AkkaRpcEnv.scala:92)
at akka.actor.ActorCell.create(ActorCell.scala:580)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
15/12/14 10:42:27 ERROR ActorSystemImpl: Uncaught fatal error from thread [sparkMaster-akka.actor.default-dispatcher-3] shutting down ActorSystem [sparkMaster]
java.lang.NoSuchMethodError: org.apache.curator.utils.PathUtils.validatePath(Ljava/lang/String;)Ljava/lang/String;
at org.apache.curator.framework.recipes.leader.LeaderLatch.<init>(LeaderLatch.java:147)
at org.apache.curator.framework.recipes.leader.LeaderLatch.<init>(LeaderLatch.java:125)
at org.apache.spark.deploy.master.ZooKeeperLeaderElectionAgent.start(ZooKeeperLeaderElectionAgent.scala:39)
at org.apache.spark.deploy.master.ZooKeeperLeaderElectionAgent.<init>(ZooKeeperLeaderElectionAgent.scala:34)
at org.apache.spark.deploy.master.ZooKeeperRecoveryModeFactory.createLeaderElectionAgent(RecoveryModeFactory.scala:74)
at org.apache.spark.deploy.master.Master.onStart(Master.scala:165)
at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$preStart$1.apply$mcV$sp(AkkaRpcEnv.scala:100)
at org.apache.spark.rpc.akka.AkkaRpcEnv.org$apache$spark$rpc$akka$AkkaRpcEnv$$safelyCall(AkkaRpcEnv.scala:197)
at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.preStart(AkkaRpcEnv.scala:99)
at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.aroundPreStart(AkkaRpcEnv.scala:92)
at akka.actor.ActorCell.create(ActorCell.scala:580)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
15/12/14 10:42:27错误错误监视器:线程[sparkMaster akka.actor.default-dispatcher-3]未捕获的致命错误正在关闭ActorSystem[sparkMaster]
java.lang.NoSuchMethodError:org.apache.curator.utils.PathUtils.validatePath(Ljava/lang/String;)Ljava/lang/String;
位于org.apache.curator.framework.recipes.leader.LeaderLatch.(LeaderLatch.java:147)
位于org.apache.curator.framework.recipes.leader.LeaderLatch.(LeaderLatch.java:125)
位于org.apache.spark.deploy.master.ZookePerLeaderElectionAgent.start(ZookePerLeaderElectionAgent.scala:39)
位于org.apache.spark.deploy.master.ZookePerLeaderElectionAgent。(ZookePerLeaderElectionAgent.scala:34)
位于org.apache.spark.deploy.master.ZooKeeperRecoveryModeFactory.createLeaderElectionAgent(RecoveryModeFactory.scala:74)
位于org.apache.spark.deploy.master.master.onStart(master.scala:165)
在org.apache.spark.rpc.akka.akkarpcev$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$preStart$1.apply$mcV$sp(akkarpcev.scala:100)
在org.apache.spark.rpc.akka.akkarpcev.org$apache$spark$rpc$akka$akkarpcev$$safelyCall(akkarpcev.scala:197)
在org.apache.spark.rpc.akka.akkarpcev$$anonfun$actorRef$lzycompute$1$1$$anon$1.preStart(akkarpcev.scala:99)
在akka.actor.actor$class.aroundPreStart(actor.scala:472)
在org.apache.spark.rpc.akka.akkarpcev$$anonfun$actorRef$lzycompute$1$1$$anon$1.aroundPreStart(akkarpcev.scala:92)
在akka.actor.ActorCell.create(ActorCell.scala:580)
在akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
在akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
在akka.dispatch.Mailbox.processAllSystemMessages上(Mailbox.scala:263)
在akka.dispatch.Mailbox.run(Mailbox.scala:219)
在akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
位于scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
位于scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
位于scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
在scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)中
15/12/14 10:42:27错误ActorSystemImpl:线程[sparkMaster akka.actor.default-dispatcher-3]未捕获的致命错误正在关闭ActorSystem[sparkMaster]
java.lang.NoSuchMethodError:org.apache.curator.utils.PathUtils.validatePath(Ljava/lang/String;)Ljava/lang/String;
位于org.apache.curator.framework.recipes.leader.LeaderLatch.(LeaderLatch.java:147)
位于org.apache.curator.framework.recipes.leader.LeaderLatch.(LeaderLatch.java:125)
位于org.apache.spark.deploy.master.ZookePerLeaderElectionAgent.start(ZookePerLeaderElectionAgent.scala:39)
位于org.apache.spark.deploy.master.ZookePerLeaderElectionAgent。(ZookePerLeaderElectionAgent.scala:34)
位于org.apache.spark.deploy.master.ZooKeeperRecoveryModeFactory.createLeaderElectionAgent(RecoveryModeFactory.scala:74)
位于org.apache.spark.deploy.master.master.onStart(master.scala:165)
在org.apache.spark.rpc.akka.akkarpcev$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$preStart$1.apply$mcV$sp(akkarpcev.scala:100)
在org.apache.spark.rpc.akka.akkarpcev.org$apache$spark$rpc$akka$akkarpcev$$safelyCall(akkarpcev.scala:197)
在org.apache.spark.rpc.akka.akkarpcev$$anonfun$actorRef$lzycompute$1$1$$anon$1.preStart(akkarpcev.scala:99)
在akka.actor.actor$class.aroundPreStart(actor.scala:472)
在org.apache.spark.rpc.akka.akkarpcev$$anonfun$actorRef$lzycompute$1$1$$anon$1.aroundPreStart(akkarpcev.scala:92)
在akka.actor.ActorCell.create(ActorCell.scala:580)
在akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
在akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
在akka.dispatch.Mailbox.processAllSystemMessages上(Mailbox.scala:263)
在akka.dispatch.Mailbox.run(Mailbox.scala:219)
在akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
位于scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
位于scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
位于scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
在scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)中
使用与spark相同的策展人版本
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-client</artifactId>
<version>1.5.2</version>
</dependency>
org.apache.curator
馆长客户
1.5.2
使用与spark相同的策展人版本
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-client</artifactId>
<version>1.5.2</version>
</dependency>
org.apache.curator
馆长客户
1.5.2
我在尝试用Hadoop 3.3.0设置高可用性Spark 2.4.7时遇到了这个错误。将Hadoop版本降级到3.2.2解决了这个问题。我在尝试用Hadoop 3.3.0设置高可用性Spark 2.4.7时遇到了这个错误。将Hadoop版本降级到3.2.2解决了问题。检查集群的版本和正在使用的库!火花1.5.2。Hadoop 2.6.0。Zookeeper 3.4.5检查集群的版本和正在使用的库!火花1.5.2。Hadoop 2.6.0。动物园管理员3.4.5