Scala 与akka actor在spark群集之间通信时遇到死信
因为spark构建在Akka之上,所以我想使用Akka在spark集群之间发送和接收消息 根据本教程,我可以在本地运行StreamingApp,并将消息发送到actorStream本身 然后,我尝试将发送方部分连接到我的另一个spark master,并将消息从spark master发送到StreamingApp中的远程参与者。代码如下Scala 与akka actor在spark群集之间通信时遇到死信,scala,apache-spark,akka,Scala,Apache Spark,Akka,因为spark构建在Akka之上,所以我想使用Akka在spark集群之间发送和接收消息 根据本教程,我可以在本地运行StreamingApp,并将消息发送到actorStream本身 然后,我尝试将发送方部分连接到我的另一个spark master,并将消息从spark master发送到StreamingApp中的远程参与者。代码如下 object SenderApp extends Serializable { def main(args: Array[String]) {
object SenderApp extends Serializable {
def main(args: Array[String]) {
val driverPort = 12345
val driverHost = "xxxx"
val conf = new SparkConf(false)
.setMaster("spark://localhost:8888") // Connecting to my spark master
.setAppName("Spark Akka Streaming Sender")
.set("spark.logConf", "true")
.set("spark.akka.logLifecycleEvents", "true")
val actorName = "helloer"
val sc = new SparkContext(conf)
val actorSystem = SparkEnv.get.actorSystem
val url = s"akka.tcp://sparkDriver@$driverHost:$driverPort/user/Supervisor0/$actorName"
val helloer = actorSystem.actorSelection(url)
helloer ! "Hello"
helloer ! "from"
helloer ! "Spark Streaming"
helloer ! "with"
helloer ! "Scala"
helloer ! "and"
helloer ! "Akka"
}
}
然后我从StreamingApp收到消息说它遇到了死信。
详情如下:
INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkDriver/deadLetters] to Actor[akka://sparkDriver/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkDriver%40111.22.33.444%3A56840-4#-2094758237] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
根据这篇文章:
我换了helloer,现在可以用了
val timeout = 100 seconds
val helloer = Await.result(actorSystem.actorSelection(url).resolveOne(timeout),
timeout)