Apache spark 启动Apache Spark应用程序一次，然后等待数据处理_Apache Spark_Yarn

Apache spark 启动Apache Spark应用程序一次，然后等待数据处理

apache-spark

Apache spark 启动Apache Spark应用程序一次，然后等待数据处理,apache-spark,yarn,Apache Spark,Yarn,我正在纱线上启动ApacheSpark应用程序（Hadoop）。此应用程序工作正常，但等待验收和运行的过程太长。例如：我要数一数小文件中的单词（~100个单词）。我正在启动应用程序： /opt/spark/bin/spark-submit --class org.apache.spark.examples.JavaWordCount --deploy-mode cluster --master yarn --driver-memory 2g --executor-memory 2g /opt/s

我正在纱线上启动ApacheSpark应用程序（Hadoop）。此应用程序工作正常，但等待验收和运行的过程太长。例如：我要数一数小文件中的单词（~100个单词）。我正在启动应用程序：

/opt/spark/bin/spark-submit --class org.apache.spark.examples.JavaWordCount --deploy-mode cluster --master yarn --driver-memory 2g --executor-memory 2g /opt/spark/examples/jars/spark-examples_2.11-2.0.0.jar hdfs://hadoop-master:9000/input/file.txt

我在等待：
-已接受-11s，
-正在运行-25s
除验收前和运行后的几秒钟外：

16/08/26 15:18:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/08/26 15:18:27 INFO client.RMProxy: Connecting to ResourceManager at hadoop-master/172.29.74.68:8032
16/08/26 15:18:27 INFO yarn.Client: Requesting a new application from cluster with 2 NodeManagers
16/08/26 15:18:27 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (4096 MB per container)
16/08/26 15:18:27 INFO yarn.Client: Will allocate AM container, with 2432 MB memory including 384 MB overhead
16/08/26 15:18:27 INFO yarn.Client: Setting up container launch context for our AM
16/08/26 15:18:27 INFO yarn.Client: Setting up the launch environment for our AM container
16/08/26 15:18:27 INFO yarn.Client: Preparing resources for our AM container
16/08/26 15:18:27 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
16/08/26 15:18:32 INFO yarn.Client: Uploading resource file:/tmp/spark-b8aa8874-9747-4c1f-8390-d0abbad019ee/__spark_libs__3386575858123884242.zip -> hdfs://hadoop-master:9000/user/root/.sparkStaging/application_1472201718061_0015/__spark_libs__3386575858123884242.zip
16/08/26 15:18:37 INFO yarn.Client: Uploading resource file:/opt/spark/examples/jars/spark-examples_2.11-2.0.0.jar -> hdfs://hadoop-master:9000/user/root/.sparkStaging/application_1472201718061_0015/spark-examples_2.11-2.0.0.jar
16/08/26 15:18:37 INFO yarn.Client: Uploading resource file:/tmp/spark-b8aa8874-9747-4c1f-8390-d0abbad019ee/__spark_conf__1130150930664135048.zip -> hdfs://hadoop-master:9000/user/root/.sparkStaging/application_1472201718061_0015/__spark_conf__.zip
16/08/26 15:18:37 INFO spark.SecurityManager: Changing view acls to: root
16/08/26 15:18:37 INFO spark.SecurityManager: Changing modify acls to: root
16/08/26 15:18:37 INFO spark.SecurityManager: Changing view acls groups to: 
16/08/26 15:18:37 INFO spark.SecurityManager: Changing modify acls groups to: 
16/08/26 15:18:37 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
16/08/26 15:18:37 INFO yarn.Client: Submitting application application_1472201718061_0015 to ResourceManager
16/08/26 15:18:37 INFO impl.YarnClientImpl: Submitted application application_1472201718061_0015
16/08/26 15:18:38 INFO yarn.Client: Application report for application_1472201718061_0015 (state: ACCEPTED)
16/08/26 15:18:38 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1472217517552
     final status: UNDEFINED
     tracking URL: http://hadoop-master:8088/proxy/application_1472201718061_0015/
     user: root
16/08/26 15:18:39 INFO yarn.Client: Application report for application_1472201718061_0015 (state: ACCEPTED)
16/08/26 15:18:40 INFO yarn.Client: Application report for application_1472201718061_0015 (state: ACCEPTED)
16/08/26 15:18:41 INFO yarn.Client: Application report for application_1472201718061_0015 (state: ACCEPTED)
16/08/26 15:18:42 INFO yarn.Client: Application report for application_1472201718061_0015 (state: ACCEPTED)
16/08/26 15:18:43 INFO yarn.Client: Application report for application_1472201718061_0015 (state: ACCEPTED)
16/08/26 15:18:44 INFO yarn.Client: Application report for application_1472201718061_0015 (state: ACCEPTED)
16/08/26 15:18:45 INFO yarn.Client: Application report for application_1472201718061_0015 (state: ACCEPTED)
16/08/26 15:18:46 INFO yarn.Client: Application report for application_1472201718061_0015 (state: ACCEPTED)
16/08/26 15:18:47 INFO yarn.Client: Application report for application_1472201718061_0015 (state: ACCEPTED)
16/08/26 15:18:48 INFO yarn.Client: Application report for application_1472201718061_0015 (state: ACCEPTED)
16/08/26 15:18:49 INFO yarn.Client: Application report for application_1472201718061_0015 (state: ACCEPTED)
16/08/26 15:18:50 INFO yarn.Client: Application report for application_1472201718061_0015 (state: ACCEPTED)
16/08/26 15:18:51 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:18:51 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 172.29.77.40
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1472217517552
     final status: UNDEFINED
     tracking URL: http://hadoop-master:8088/proxy/application_1472201718061_0015/
     user: root
16/08/26 15:18:52 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:18:53 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:18:54 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:18:55 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:18:56 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:18:57 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:18:58 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:18:59 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:00 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:01 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:02 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:03 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:04 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:05 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:06 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:07 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:08 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:09 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:10 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:11 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:12 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:13 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:14 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:15 INFO yarn.Client: Application report for application_1472201718061_0015 (state: RUNNING)
16/08/26 15:19:16 INFO yarn.Client: Application report for application_1472201718061_0015 (state: FINISHED)
16/08/26 15:19:16 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 172.29.77.40
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1472217517552
     final status: SUCCEEDED
     tracking URL: http://hadoop-master:8088/proxy/application_1472201718061_0015/
     user: root
16/08/26 15:19:16 INFO util.ShutdownHookManager: Shutdown hook called
16/08/26 15:19:16 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-b8aa8874-9747-4c1f-8390-d0abbad019ee

对我来说太长了我想启动一次，它应该可以工作并等待数据。在我给它一个文件后，它应该处理数据，给我一个结果，然后回到等待下一个文件的状态。

Apache Spark在纱线上运行时，这可能吗？

是的，这是可能的，并且被称为允许以连续方式进行批处理