Java 使用apachespark进行UDP

我是Apache Spark的新手

我在我的UDP端口8060接收数据，我想捕获它并实时执行一些操作，为此我使用Spark Streaming。 这是我的主要课程：

import java.util.Arrays;
import java.util.Comparator;
import java.util.Map;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaReceiverInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;


public class Main {

    public static void main(String[] args) throws Exception {
        SparkConf conf = new SparkConf().setMaster("local").setAppName("NetworkWordCount");
        JavaStreamingContext jssc = new JavaStreamingContext(conf, new Duration(1000));
        JavaReceiverInputDStream<String> lines = jssc.receiverStream(new CustomReceiver("localhost",8060));
        JavaDStream<String> hash = lines.flatMap(
                  new FlatMapFunction<String, String>() {
                        @Override public Iterable<String> call(String x) {
                      //    System.out.println(x);
                            String[] s=x.split("\\}\\{");
                          return Arrays.asList(s);

                      }
            });
        hash.print();
        jssc.start();
    }
}

和客户接收器类别：

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.ConnectException;
import java.net.DatagramPacket;
import java.net.DatagramSocket;

import org.apache.spark.storage.StorageLevel;
import org.apache.spark.streaming.receiver.Receiver;

public class CustomReceiver extends Receiver<String> {

      String host = null;
      int port = -1;


      public CustomReceiver(String host_ , int port_) {
        super(StorageLevel.MEMORY_AND_DISK_2());
        host = host_;
        port = port_;
      }

      public void onStart() {
        // Start the thread that receives data over a connection

        new Thread()  {
          @Override public void run() {
            receive();
          }
        }.start();
      }

      public void onStop() {
        // There is nothing much to do as the thread calling receive()
        // is designed to stop by itself isStopped() returns false
      }

      /** Create a socket connection and receive data until receiver is stopped */
      private void receive() {
        DatagramSocket socket = null;
        String userInput = null;

        try {
          // connect to the server
          socket = new DatagramSocket(port);
          byte[] receiveData = new byte[1024];
          DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length);
          socket.receive(receivePacket);
          userInput = new String( receivePacket.getData());
      //    System.out.println(userInput);
          // Until stopped or connection broken continue reading


    //        System.out.println(userInput);
              store(userInput);

          socket.close();

          // Restart in an attempt to connect again when server is active again
          restart("Trying to connect again");
        } catch(ConnectException ce) {
          // restart if could not connect to server
          restart("Could not connect", ce);
        } catch(Throwable t) {
          // restart if there is any other error
          restart("Error receiving data", t);
        }
      }
    }

代码循环，但一旦我发送数据，就会出现一个奇怪的错误，它停止工作：

15/01/20 17:44:53 INFO spark.SecurityManager: Changing view acls to: gautambajaj
15/01/20 17:44:53 INFO spark.SecurityManager: Changing modify acls to: gautambajaj
15/01/20 17:44:53 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(gautambajaj); users with modify permissions: Set(gautambajaj)
15/01/20 17:44:54 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/01/20 17:44:55 INFO Remoting: Starting remoting
15/01/20 17:44:55 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.34.71.121:63919]
15/01/20 17:44:55 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@10.34.71.121:63919]
15/01/20 17:44:55 INFO util.Utils: Successfully started service 'sparkDriver' on port 63919.
15/01/20 17:44:55 INFO spark.SparkEnv: Registering MapOutputTracker
15/01/20 17:44:55 INFO spark.SparkEnv: Registering BlockManagerMaster
15/01/20 17:44:55 INFO storage.DiskBlockManager: Created local directory at /var/folders/z3/347_jn996097tyhzl905tm7m0000gn/T/spark-local-20150120174455-e4cf
15/01/20 17:44:56 INFO util.Utils: Successfully started service 'Connection manager for block manager' on port 63920.
15/01/20 17:44:56 INFO network.ConnectionManager: Bound socket to port 63920 with id = ConnectionManagerId(10.34.71.121,63920)
15/01/20 17:44:56 INFO storage.MemoryStore: MemoryStore started with capacity 818.5 MB
15/01/20 17:44:56 INFO storage.BlockManagerMaster: Trying to register BlockManager
15/01/20 17:44:56 INFO storage.BlockManagerMasterActor: Registering block manager 10.34.71.121:63920 with 818.5 MB RAM, BlockManagerId(<driver>, 10.34.71.121, 63920, 0)
15/01/20 17:44:56 INFO storage.BlockManagerMaster: Registered BlockManager
15/01/20 17:44:56 INFO spark.HttpFileServer: HTTP File server directory is /var/folders/z3/347_jn996097tyhzl905tm7m0000gn/T/spark-9737478e-492e-44c6-accc-e8a7c91b2ab3
15/01/20 17:44:56 INFO spark.HttpServer: Starting HTTP Server
15/01/20 17:44:56 INFO server.Server: jetty-8.1.14.v20131031
15/01/20 17:44:56 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:63921
15/01/20 17:44:56 INFO util.Utils: Successfully started service 'HTTP file server' on port 63921.
15/01/20 17:44:57 INFO server.Server: jetty-8.1.14.v20131031
15/01/20 17:44:57 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
15/01/20 17:44:57 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
15/01/20 17:44:57 INFO ui.SparkUI: Started SparkUI at http://10.34.71.121:4040
2015-01-20 17:44:57.460 java[17010:1404608] Unable to load realm info from SCDynamicStore
15/01/20 17:44:57 INFO util.AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@10.34.71.121:63919/user/HeartbeatReceiver
15/01/20 17:44:58 INFO scheduler.ReceiverTracker: ReceiverTracker started
15/01/20 17:44:58 INFO dstream.ForEachDStream: metadataCleanupDelay = -1
15/01/20 17:44:58 INFO dstream.FlatMappedDStream: metadataCleanupDelay = -1
15/01/20 17:44:58 INFO dstream.PluggableInputDStream: metadataCleanupDelay = -1
15/01/20 17:44:58 INFO dstream.PluggableInputDStream: Slide time = 1000 ms
15/01/20 17:44:58 INFO dstream.PluggableInputDStream: Storage level = StorageLevel(false, false, false, false, 1)
15/01/20 17:44:58 INFO dstream.PluggableInputDStream: Checkpoint interval = null
15/01/20 17:44:58 INFO dstream.PluggableInputDStream: Remember duration = 1000 ms
15/01/20 17:44:58 INFO dstream.PluggableInputDStream: Initialized and validated org.apache.spark.streaming.dstream.PluggableInputDStream@e039859
15/01/20 17:44:58 INFO dstream.FlatMappedDStream: Slide time = 1000 ms
15/01/20 17:44:58 INFO dstream.FlatMappedDStream: Storage level = StorageLevel(false, false, false, false, 1)
15/01/20 17:44:58 INFO dstream.FlatMappedDStream: Checkpoint interval = null
15/01/20 17:44:58 INFO dstream.FlatMappedDStream: Remember duration = 1000 ms
15/01/20 17:44:58 INFO dstream.FlatMappedDStream: Initialized and validated org.apache.spark.streaming.dstream.FlatMappedDStream@6e247d4a
15/01/20 17:44:58 INFO dstream.ForEachDStream: Slide time = 1000 ms
15/01/20 17:44:58 INFO dstream.ForEachDStream: Storage level = StorageLevel(false, false, false, false, 1)
15/01/20 17:44:58 INFO dstream.ForEachDStream: Checkpoint interval = null
15/01/20 17:44:58 INFO dstream.ForEachDStream: Remember duration = 1000 ms
15/01/20 17:44:58 INFO dstream.ForEachDStream: Initialized and validated org.apache.spark.streaming.dstream.ForEachDStream@5f159e0c
15/01/20 17:44:58 INFO scheduler.ReceiverTracker: Starting 1 receivers
15/01/20 17:44:58 INFO spark.SparkContext: Starting job: start at Main.java:76
15/01/20 17:44:58 INFO scheduler.DAGScheduler: Got job 0 (start at Main.java:76) with 1 output partitions (allowLocal=false)
15/01/20 17:44:58 INFO scheduler.DAGScheduler: Final stage: Stage 0(start at Main.java:76)
15/01/20 17:44:58 INFO scheduler.DAGScheduler: Parents of final stage: List()
15/01/20 17:44:58 INFO util.RecurringTimer: Started timer for JobGenerator at time 1421743499000
15/01/20 17:44:58 INFO scheduler.JobGenerator: Started JobGenerator at 1421743499000 ms
15/01/20 17:44:58 INFO scheduler.JobScheduler: Started JobScheduler
15/01/20 17:44:58 INFO scheduler.DAGScheduler: Missing parents: List()
15/01/20 17:44:58 INFO scheduler.DAGScheduler: Submitting Stage 0 (ParallelCollectionRDD[0] at start at Main.java:76), which has no missing parents
15/01/20 17:44:59 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:44:59 INFO scheduler.JobScheduler: Added jobs for time 1421743499000 ms
15/01/20 17:44:59 INFO scheduler.JobScheduler: Starting job streaming job 1421743499000 ms.0 from job set of time 1421743499000 ms
-------------------------------------------
Time: 1421743499000 ms
-------------------------------------------

15/01/20 17:44:59 INFO scheduler.JobScheduler: Finished job streaming job 1421743499000 ms.0 from job set of time 1421743499000 ms
15/01/20 17:44:59 INFO scheduler.JobScheduler: Total delay: 0.302 s for time 1421743499000 ms (execution: 0.069 s)
15/01/20 17:45:00 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:00 INFO scheduler.JobScheduler: Added jobs for time 1421743500000 ms
15/01/20 17:45:00 INFO scheduler.JobScheduler: Starting job streaming job 1421743500000 ms.0 from job set of time 1421743500000 ms
-------------------------------------------
Time: 1421743500000 ms
-------------------------------------------

15/01/20 17:45:00 INFO scheduler.JobScheduler: Finished job streaming job 1421743500000 ms.0 from job set of time 1421743500000 ms
15/01/20 17:45:00 INFO scheduler.JobScheduler: Total delay: 0.005 s for time 1421743500000 ms (execution: 0.001 s)
15/01/20 17:45:00 INFO rdd.FlatMappedRDD: Removing RDD 2 from persistence list
15/01/20 17:45:00 INFO storage.BlockManager: Removing RDD 2
15/01/20 17:45:00 INFO rdd.BlockRDD: Removing RDD 1 from persistence list
15/01/20 17:45:00 INFO storage.BlockManager: Removing RDD 1
15/01/20 17:45:00 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[1] at receiverStream at Main.java:30 of time 1421743500000 ms
15/01/20 17:45:01 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:01 INFO scheduler.JobScheduler: Added jobs for time 1421743501000 ms
-------------------------------------------15/01/20 17:45:01 INFO scheduler.JobScheduler: Starting job streaming job 1421743501000 ms.0 from job set of time 1421743501000 ms
15/01/20 17:45:01 INFO scheduler.JobScheduler: Finished job streaming job 1421743501000 ms.0 from job set of time 1421743501000 ms
15/01/20 17:45:01 INFO rdd.FlatMappedRDD: Removing RDD 4 from persistence list
15/01/20 17:45:01 INFO scheduler.JobScheduler: Total delay: 0.004 s for time 1421743501000 ms (execution: 0.000 s)

Time: 1421743501000 ms
-------------------------------------------

15/01/20 17:45:01 INFO storage.BlockManager: Removing RDD 4
15/01/20 17:45:01 INFO rdd.BlockRDD: Removing RDD 3 from persistence list
15/01/20 17:45:01 INFO storage.BlockManager: Removing RDD 3
15/01/20 17:45:01 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[3] at receiverStream at Main.java:30 of time 1421743501000 ms
15/01/20 17:45:01 INFO storage.MemoryStore: ensureFreeSpace(1216) called with curMem=0, maxMem=858229309
15/01/20 17:45:01 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1216.0 B, free 818.5 MB)
15/01/20 17:45:01 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 0 (ParallelCollectionRDD[0] at start at Main.java:76)
15/01/20 17:45:01 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
15/01/20 17:45:01 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, PROCESS_LOCAL, 1558 bytes)
15/01/20 17:45:01 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
15/01/20 17:45:01 INFO receiver.ReceiverSupervisorImpl: Registered receiver 0
15/01/20 17:45:01 INFO util.RecurringTimer: Started timer for BlockGenerator at time 1421743502000
15/01/20 17:45:01 INFO receiver.BlockGenerator: Started block pushing thread
15/01/20 17:45:01 INFO receiver.BlockGenerator: Started BlockGenerator
15/01/20 17:45:01 INFO receiver.ReceiverSupervisorImpl: Starting receiver
15/01/20 17:45:01 INFO receiver.ReceiverSupervisorImpl: Called receiver onStart
15/01/20 17:45:01 INFO scheduler.ReceiverTracker: Registered receiver for stream 0 from akka://sparkDriver
15/01/20 17:45:01 INFO scheduler.ReceiverTracker: Registered receiver for stream 0 from akka://sparkDriver
15/01/20 17:45:02 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:02 INFO scheduler.JobScheduler: Added jobs for time 1421743502000 ms
-------------------------------------------
Time: 1421743502000 ms
-------------------------------------------

15/01/20 17:45:02 INFO scheduler.JobScheduler: Starting job streaming job 1421743502000 ms.0 from job set of time 1421743502000 ms
15/01/20 17:45:02 INFO scheduler.JobScheduler: Finished job streaming job 1421743502000 ms.0 from job set of time 1421743502000 ms
15/01/20 17:45:02 INFO rdd.FlatMappedRDD: Removing RDD 6 from persistence list
15/01/20 17:45:02 INFO scheduler.JobScheduler: Total delay: 0.004 s for time 1421743502000 ms (execution: 0.000 s)
15/01/20 17:45:02 INFO storage.BlockManager: Removing RDD 6
15/01/20 17:45:02 INFO rdd.BlockRDD: Removing RDD 5 from persistence list
15/01/20 17:45:02 INFO storage.BlockManager: Removing RDD 5
15/01/20 17:45:02 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[5] at receiverStream at Main.java:30 of time 1421743502000 ms
15/01/20 17:45:03 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:03 INFO scheduler.JobScheduler: Added jobs for time 1421743503000 ms
-------------------------------------------
15/01/20 17:45:03 INFO scheduler.JobScheduler: Starting job streaming job 1421743503000 ms.0 from job set of time 1421743503000 ms
15/01/20 17:45:03 INFO scheduler.JobScheduler: Finished job streaming job 1421743503000 ms.0 from job set of time 1421743503000 ms
15/01/20 17:45:03 INFO rdd.FlatMappedRDD: Removing RDD 8 from persistence list
15/01/20 17:45:03 INFO scheduler.JobScheduler: Total delay: 0.006 s for time 1421743503000 ms (execution: 0.000 s)
15/01/20 17:45:03 INFO storage.BlockManager: Removing RDD 8
15/01/20 17:45:03 INFO rdd.BlockRDD: Removing RDD 7 from persistence list
Time: 1421743503000 ms
-------------------------------------------

15/01/20 17:45:03 INFO storage.BlockManager: Removing RDD 7
15/01/20 17:45:03 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[7] at receiverStream at Main.java:30 of time 1421743503000 ms
15/01/20 17:45:04 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:04 INFO scheduler.JobScheduler: Added jobs for time 1421743504000 ms
15/01/20 17:45:04 INFO scheduler.JobScheduler: Starting job streaming job 1421743504000 ms.0 from job set of time 1421743504000 ms
-------------------------------------------
Time: 1421743504000 ms
-------------------------------------------

15/01/20 17:45:04 INFO scheduler.JobScheduler: Finished job streaming job 1421743504000 ms.0 from job set of time 1421743504000 ms
15/01/20 17:45:04 INFO rdd.FlatMappedRDD: Removing RDD 10 from persistence list
15/01/20 17:45:04 INFO scheduler.JobScheduler: Total delay: 0.007 s for time 1421743504000 ms (execution: 0.000 s)
15/01/20 17:45:04 INFO storage.BlockManager: Removing RDD 10
15/01/20 17:45:04 INFO rdd.BlockRDD: Removing RDD 9 from persistence list
15/01/20 17:45:04 INFO storage.BlockManager: Removing RDD 9
15/01/20 17:45:04 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[9] at receiverStream at Main.java:30 of time 1421743504000 ms
15/01/20 17:45:05 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:05 INFO scheduler.JobScheduler: Added jobs for time 1421743505000 ms
15/01/20 17:45:05 INFO scheduler.JobScheduler: Starting job streaming job 1421743505000 ms.0 from job set of time 1421743505000 ms
-------------------------------------------
Time: 1421743505000 ms
-------------------------------------------

15/01/20 17:45:05 INFO scheduler.JobScheduler: Finished job streaming job 1421743505000 ms.0 from job set of time 1421743505000 ms
15/01/20 17:45:05 INFO rdd.FlatMappedRDD: Removing RDD 12 from persistence list
15/01/20 17:45:05 INFO scheduler.JobScheduler: Total delay: 0.006 s for time 1421743505000 ms (execution: 0.000 s)
15/01/20 17:45:05 INFO storage.BlockManager: Removing RDD 12
15/01/20 17:45:05 INFO rdd.BlockRDD: Removing RDD 11 from persistence list
15/01/20 17:45:05 INFO storage.BlockManager: Removing RDD 11
15/01/20 17:45:05 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[11] at receiverStream at Main.java:30 of time 1421743505000 ms
15/01/20 17:45:06 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:06 INFO scheduler.JobScheduler: Added jobs for time 1421743506000 ms
15/01/20 17:45:06 INFO scheduler.JobScheduler: Starting job streaming job 1421743506000 ms.0 from job set of time 1421743506000 ms
-------------------------------------------
Time: 1421743506000 ms
-------------------------------------------

15/01/20 17:45:06 INFO scheduler.JobScheduler: Finished job streaming job 1421743506000 ms.0 from job set of time 1421743506000 ms
15/01/20 17:45:06 INFO rdd.FlatMappedRDD: Removing RDD 14 from persistence list
15/01/20 17:45:06 INFO scheduler.JobScheduler: Total delay: 0.004 s for time 1421743506000 ms (execution: 0.000 s)
15/01/20 17:45:06 INFO storage.BlockManager: Removing RDD 14
15/01/20 17:45:06 INFO rdd.BlockRDD: Removing RDD 13 from persistence list
15/01/20 17:45:06 INFO storage.BlockManager: Removing RDD 13
15/01/20 17:45:06 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[13] at receiverStream at Main.java:30 of time 1421743506000 ms
15/01/20 17:45:07 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:07 INFO scheduler.JobScheduler: Added jobs for time 1421743507000 ms
-------------------------------------------
15/01/20 17:45:07 INFO scheduler.JobScheduler: Starting job streaming job 1421743507000 ms.0 from job set of time 1421743507000 ms
Time: 1421743507000 ms
-------------------------------------------

15/01/20 17:45:07 INFO scheduler.JobScheduler: Finished job streaming job 1421743507000 ms.0 from job set of time 1421743507000 ms
15/01/20 17:45:07 INFO rdd.FlatMappedRDD: Removing RDD 16 from persistence list
15/01/20 17:45:07 INFO scheduler.JobScheduler: Total delay: 0.004 s for time 1421743507000 ms (execution: 0.001 s)
15/01/20 17:45:07 INFO storage.BlockManager: Removing RDD 16
15/01/20 17:45:07 INFO rdd.BlockRDD: Removing RDD 15 from persistence list
15/01/20 17:45:07 INFO storage.BlockManager: Removing RDD 15
15/01/20 17:45:07 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[15] at receiverStream at Main.java:30 of time 1421743507000 ms
15/01/20 17:45:08 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:08 INFO scheduler.JobScheduler: Added jobs for time 1421743508000 ms
15/01/20 17:45:08 INFO scheduler.JobScheduler: Starting job streaming job 1421743508000 ms.0 from job set of time 1421743508000 ms
-------------------------------------------
Time: 1421743508000 ms
-------------------------------------------
15/01/20 17:45:08 INFO scheduler.JobScheduler: Finished job streaming job 1421743508000 ms.0 from job set of time 1421743508000 ms
15/01/20 17:45:08 INFO scheduler.JobScheduler: Total delay: 0.004 s for time 1421743508000 ms (execution: 0.000 s)
15/01/20 17:45:08 INFO rdd.FlatMappedRDD: Removing RDD 18 from persistence list
15/01/20 17:45:08 INFO storage.BlockManager: Removing RDD 18

15/01/20 17:45:08 INFO rdd.BlockRDD: Removing RDD 17 from persistence list
15/01/20 17:45:08 INFO storage.BlockManager: Removing RDD 17
15/01/20 17:45:08 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[17] at receiverStream at Main.java:30 of time 1421743508000 ms
15/01/20 17:45:09 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:09 INFO scheduler.JobScheduler: Added jobs for time 1421743509000 ms
15/01/20 17:45:09 INFO scheduler.JobScheduler: Starting job streaming job 1421743509000 ms.0 from job set of time 1421743509000 ms
15/01/20 17:45:09 INFO scheduler.JobScheduler: Finished job streaming job 1421743509000 ms.0 from job set of time 1421743509000 ms
-------------------------------------------
Time: 1421743509000 ms
-------------------------------------------

15/01/20 17:45:09 INFO rdd.FlatMappedRDD: Removing RDD 20 from persistence list
15/01/20 17:45:09 INFO scheduler.JobScheduler: Total delay: 0.011 s for time 1421743509000 ms (execution: 0.000 s)
15/01/20 17:45:09 INFO storage.BlockManager: Removing RDD 20
15/01/20 17:45:09 INFO rdd.BlockRDD: Removing RDD 19 from persistence list
15/01/20 17:45:09 INFO storage.BlockManager: Removing RDD 19
15/01/20 17:45:09 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[19] at receiverStream at Main.java:30 of time 1421743509000 ms
15/01/20 17:45:09 WARN receiver.ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Trying to connect again
15/01/20 17:45:09 INFO receiver.ReceiverSupervisorImpl: Stopping receiver with message: Restarting receiver with delay 2000ms: Trying to connect again: 
15/01/20 17:45:09 INFO receiver.ReceiverSupervisorImpl: Called receiver onStop
15/01/20 17:45:09 INFO receiver.ReceiverSupervisorImpl: Deregistering receiver 0
15/01/20 17:45:09 ERROR scheduler.ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Trying to connect again
15/01/20 17:45:09 INFO receiver.ReceiverSupervisorImpl: Stopped receiver 0
15/01/20 17:45:09 INFO storage.MemoryStore: ensureFreeSpace(2120) called with curMem=1216, maxMem=858229309
15/01/20 17:45:09 INFO storage.MemoryStore: Block input-0-1421743509200 stored as values in memory (estimated size 2.1 KB, free 818.5 MB)
15/01/20 17:45:09 INFO storage.BlockManagerInfo: Added input-0-1421743509200 in memory on 10.34.71.121:63920 (size: 2.1 KB, free: 818.5 MB)
15/01/20 17:45:09 INFO storage.BlockManagerMaster: Updated info of block input-0-1421743509200
15/01/20 17:45:09 WARN storage.BlockManager: Block input-0-1421743509200 replicated to only 0 peer(s) instead of 1 peers
15/01/20 17:45:09 INFO receiver.BlockGenerator: Pushed block input-0-1421743509200
15/01/20 17:45:10 INFO scheduler.ReceiverTracker: Stream 0 received 1 blocks
15/01/20 17:45:10 INFO scheduler.JobScheduler: Added jobs for time 1421743510000 ms
15/01/20 17:45:10 INFO scheduler.JobScheduler: Starting job streaming job 1421743510000 ms.0 from job set of time 1421743510000 ms
15/01/20 17:45:10 INFO spark.SparkContext: Starting job: getCallSite at DStream.scala:294
15/01/20 17:45:10 INFO scheduler.DAGScheduler: Got job 1 (getCallSite at DStream.scala:294) with 1 output partitions (allowLocal=true)
15/01/20 17:45:10 INFO scheduler.DAGScheduler: Final stage: Stage 1(getCallSite at DStream.scala:294)
15/01/20 17:45:10 INFO scheduler.DAGScheduler: Parents of final stage: List()
15/01/20 17:45:10 INFO scheduler.DAGScheduler: Missing parents: List()
15/01/20 17:45:10 INFO scheduler.DAGScheduler: Submitting Stage 1 (FlatMappedRDD[24] at flatMap at Main.java:32), which has no missing parents
15/01/20 17:45:10 INFO storage.MemoryStore: ensureFreeSpace(1800) called with curMem=3336, maxMem=858229309
15/01/20 17:45:10 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 1800.0 B, free 818.5 MB)
15/01/20 17:45:10 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 1 (FlatMappedRDD[24] at flatMap at Main.java:32)
15/01/20 17:45:10 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
15/01/20 17:45:11 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:11 INFO scheduler.JobScheduler: Added jobs for time 1421743511000 ms
15/01/20 17:45:11 INFO receiver.ReceiverSupervisorImpl: Starting receiver again
15/01/20 17:45:11 INFO receiver.ReceiverSupervisorImpl: Starting receiver
15/01/20 17:45:11 INFO receiver.ReceiverSupervisorImpl: Called receiver onStart
15/01/20 17:45:11 INFO scheduler.ReceiverTracker: Registered receiver for stream 0 from akka://sparkDriver
15/01/20 17:45:11 INFO receiver.ReceiverSupervisorImpl: Receiver started again
15/01/20 17:45:12 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:12 INFO scheduler.JobScheduler: Added jobs for time 1421743512000 ms
15/01/20 17:45:13 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:13 INFO scheduler.JobScheduler: Added jobs for time 1421743513000 ms
15/01/20 17:45:14 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:14 INFO scheduler.JobScheduler: Added jobs for time 1421743514000 ms

---

我到底做错了什么？

socket.receive函数上没有while循环或其他重复构造。它接收一个1024位数据包，存储它，然后关闭套接字。此接收器不能接收多个数据包。Spark尝试重新启动您的接收器两次，但失败。

我在添加While后尝试了！在创建套接字之前和创建套接字之后都有一个循环，但这也不起作用。在同一地址签出我的更新代码。这很正常。您需要循环检查数据源是否仍有数据包要提供。从SocketInputStream中的代码中获得灵感！你能说得更准确些吗。假设您的流有2048位要在UDP套接字上发送。通过步骤了解接收者对此的反应。2.与位于此处的SocketInputStream中的TCP套接字进行比较：注意while循环。请注意，这不仅仅是条件反射。想一想你会如何在你的案例中添加一个与其他条件相当的条件。3.结合1+2，总结出需要循环的内容。4.对您提出的问题的答案进行投票。谢谢！但实际上这解决了我的问题：。我的循环是正确的，但我没有足够的插槽。