Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/6.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Scala 与akka actor在spark群集之间通信时遇到死信_Scala_Apache Spark_Akka - Fatal编程技术网

Scala 与akka actor在spark群集之间通信时遇到死信

Scala 与akka actor在spark群集之间通信时遇到死信,scala,apache-spark,akka,Scala,Apache Spark,Akka,因为spark构建在Akka之上,所以我想使用Akka在spark集群之间发送和接收消息 根据本教程,我可以在本地运行StreamingApp,并将消息发送到actorStream本身 然后,我尝试将发送方部分连接到我的另一个spark master,并将消息从spark master发送到StreamingApp中的远程参与者。代码如下 object SenderApp extends Serializable { def main(args: Array[String]) {

因为spark构建在Akka之上,所以我想使用Akka在spark集群之间发送和接收消息

根据本教程,我可以在本地运行StreamingApp,并将消息发送到actorStream本身

然后,我尝试将发送方部分连接到我的另一个spark master,并将消息从spark master发送到StreamingApp中的远程参与者。代码如下

object SenderApp extends Serializable {

    def main(args: Array[String]) {

        val driverPort = 12345
        val driverHost = "xxxx"
        val conf = new SparkConf(false) 
            .setMaster("spark://localhost:8888") // Connecting to my spark master
            .setAppName("Spark Akka Streaming Sender")
            .set("spark.logConf", "true")
            .set("spark.akka.logLifecycleEvents", "true")
        val actorName = "helloer"

        val sc = new SparkContext(conf)

        val actorSystem = SparkEnv.get.actorSystem

        val url = s"akka.tcp://sparkDriver@$driverHost:$driverPort/user/Supervisor0/$actorName"

        val helloer = actorSystem.actorSelection(url)

        helloer ! "Hello"
        helloer ! "from"
        helloer ! "Spark Streaming"
        helloer ! "with"
        helloer ! "Scala"
        helloer ! "and"
        helloer ! "Akka"
    }
}
然后我从StreamingApp收到消息说它遇到了死信。 详情如下:

INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkDriver/deadLetters] to Actor[akka://sparkDriver/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkDriver%40111.22.33.444%3A56840-4#-2094758237] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
根据这篇文章:

我换了helloer,现在可以用了

val timeout = 100 seconds

val helloer = Await.result(actorSystem.actorSelection(url).resolveOne(timeout), 
                           timeout)