在spark应用程序中连接到mongodb时无法退出，为什么？
我的build.sbt文件是：在spark应用程序中连接到mongodb时无法退出，为什么？,mongodb,scala,apache-spark,sbt,Mongodb,Scala,Apache Spark,Sbt,我的build.sbt文件是： import Dependencies._ lazy val root = (project in file(".")). settings( inThisBuild(List( organization := "org.dean", scalaVersion := "2.11.8", version := "0.1.0" )), name := "Hello", libraryD
import Dependencies._
lazy val root = (project in file(".")).
settings(
inThisBuild(List(
organization := "org.dean",
scalaVersion := "2.11.8",
version := "0.1.0"
)),
name := "Hello",
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1" % "provided",
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.1" % "provided",
libraryDependencies += "org.mongodb.spark" % "mongo-spark-connector_2.10" % "2.0.0"
)
package org.dean
import org.apache.spark.sql.SparkSession;
import org.apache.log4j.{Level, Logger}
object Hello {
def main(args: Array[String]) {
if(args.length != 3) {
Logger.getRootLogger().error("Need three arguments for application, quit now")
System.exit(1)
}
val spark = SparkSession
.builder()
.appName("Hello Spark exampl with MongoDB connector")
.config("spark.some.config.option", "some-value")
.getOrCreate()
val sc1 = spark.sparkContext
//e.g. arg0: mongodb://192.168.72.19:27017/device_detail_tags
val weekday = args(1)
val mongoUrl = args(0)+".js_watchVideo_weekday_" + weekday
val hour = args(2)
Logger.getRootLogger().info("MongoURL: " + mongoUrl)
Logger.getRootLogger().info("Weekday: " + weekday)
Logger.getRootLogger().info("Hour: " + hour)
// cal user type profile
val cos_ps = spark.read.format("com.mongodb.spark.sql.DefaultSource").option("uri", "mongodb://192.168.72.19:27017/cos.programset").load().select("id","allprogram")
spark.stop()
}
}
#!/bin/bash
sbt clean assembly
/home/dean/work/software/spark-2.1.1-bin-hadoop2.7/bin/spark-submit \
--class org.dean.Hello \
--master spark://192.168.72.35:7077,192.168.72.36:7077,192.168.72.37:7077\
--executor-memory 4G \
--total-executor-cores 10 \
--executor-cores 1 \
./target/scala-2.11/Hello-assembly-0.1.0.jar \
mongodb://192.168.72.19:27017/device_detail_tags 3 9
[info] downloading http://maven.aliyun.com/nexus/content/groups/public/org/mongodb/spark/mongo-spark-connector_2.10/2.0.0/mongo-spark-connector_2.10-2.0.0.jar ...
[info] [SUCCESSFUL ] org.mongodb.spark#mongo-spark-connector_2.10;2.0.0!mongo-spark-connector_2.10.jar (998ms)
[info] downloading http://maven.aliyun.com/nexus/content/groups/public/org/mongodb/mongo-java-driver/3.2.2/mongo-java-driver-3.2.2.jar ...
[info] [SUCCESSFUL ] org.mongodb#mongo-java-driver;3.2.2!mongo-java-driver.jar (1622ms)
[info] Done updating.
[info] Compiling 1 Scala source to /home/dean/work/gitlab/gitlab.com/mystudy/spark/code/scala/sbt-projects/hellospark/target/scala-2.11/classes...
[info] Including: mongo-spark-connector_2.10-2.0.0.jar
[info] Including: mongo-java-driver-3.2.2.jar
[info] Including: scala-library-2.11.8.jar
[info] Including: tools.jar
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[warn] Merging 'META-INF/MANIFEST.MF' with strategy 'discard'
[warn] Strategy 'discard' was applied to a file
[info] SHA-1: 1a26c6902bce3e530649a0bb7c5978712334dafc
[info] Packaging /home/dean/work/gitlab/gitlab.com/mystudy/spark/code/scala/sbt-projects/hellospark/target/scala-2.11/Hello-assembly-0.1.0.jar ...
[info] Done packaging.
[success] Total time: 18 s, completed 2017-7-18 22:05:32
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/07/18 22:05:33 INFO SparkContext: Running Spark version 2.1.1
17/07/18 22:05:33 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/07/18 22:05:33 WARN Utils: Your hostname, dean-mi resolves to a loopback address: 127.0.1.1; using 192.168.31.161 instead (on interface wlp2s0)
17/07/18 22:05:33 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
17/07/18 22:05:33 INFO SecurityManager: Changing view acls to: dean
17/07/18 22:05:33 INFO SecurityManager: Changing modify acls to: dean
17/07/18 22:05:33 INFO SecurityManager: Changing view acls groups to:
17/07/18 22:05:33 INFO SecurityManager: Changing modify acls groups to:
17/07/18 22:05:33 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(dean); groups with view permissions: Set(); users with modify permissions: Set(dean); groups with modify permissions: Set()
17/07/18 22:05:33 INFO Utils: Successfully started service 'sparkDriver' on port 33633.
17/07/18 22:05:33 INFO SparkEnv: Registering MapOutputTracker
17/07/18 22:05:33 INFO SparkEnv: Registering BlockManagerMaster
17/07/18 22:05:33 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
17/07/18 22:05:33 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
17/07/18 22:05:33 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-28bd3b13-68cb-4ea3-8f24-a7d6e1768f28
17/07/18 22:05:33 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
17/07/18 22:05:34 INFO SparkEnv: Registering OutputCommitCoordinator
17/07/18 22:05:34 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/07/18 22:05:34 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.31.161:4040
17/07/18 22:05:34 INFO SparkContext: Added JAR file:/home/dean/work/gitlab/gitlab.com/mystudy/spark/code/scala/sbt-projects/hellospark/./target/scala-2.11/Hello-assembly-0.1.0.jar at spark://192.168.31.161:33633/jars/Hello-assembly-0.1.0.jar with timestamp 1500386734339
17/07/18 22:05:34 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://192.168.72.35:7077...
17/07/18 22:05:34 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://192.168.72.36:7077...
17/07/18 22:05:34 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://192.168.72.37:7077...
17/07/18 22:05:34 INFO TransportClientFactory: Successfully created connection to /192.168.72.36:7077 after 83 ms (0 ms spent in bootstraps)
17/07/18 22:05:34 INFO TransportClientFactory: Successfully created connection to /192.168.72.37:7077 after 80 ms (0 ms spent in bootstraps)
17/07/18 22:05:34 INFO TransportClientFactory: Successfully created connection to /192.168.72.35:7077 after 160 ms (0 ms spent in bootstraps)
17/07/18 22:05:35 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20170718220535-0379
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/0 on worker-20170601143814-192.168.72.44-12395 (192.168.72.44:12395) with 1 cores
17/07/18 22:05:35 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/0 on hostPort 192.168.72.44:12395 with 1 cores, 4.0 GB RAM
17/07/18 22:05:35 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40689.
17/07/18 22:05:35 INFO NettyBlockTransferService: Server created on 192.168.31.161:40689
17/07/18 22:05:35 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
17/07/18 22:05:35 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.31.161, 40689, None)
17/07/18 22:05:35 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.31.161:40689 with 366.3 MB RAM, BlockManagerId(driver, 192.168.31.161, 40689, None)
17/07/18 22:05:35 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.31.161, 40689, None)
17/07/18 22:05:35 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.31.161, 40689, None)
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/1 on worker-20170711142458-192.168.72.18-25899 (192.168.72.18:25899) with 1 cores
17/07/18 22:05:35 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/1 on hostPort 192.168.72.18:25899 with 1 cores, 4.0 GB RAM
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/2 on worker-20170711142201-192.168.72.53-21310 (192.168.72.53:21310) with 1 cores
17/07/18 22:05:35 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/2 on hostPort 192.168.72.53:21310 with 1 cores, 4.0 GB RAM
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/3 on worker-20170711142123-192.168.72.54-31587 (192.168.72.54:31587) with 1 cores
17/07/18 22:05:35 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/3 on hostPort 192.168.72.54:31587 with 1 cores, 4.0 GB RAM
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/4 on worker-20170601143944-192.168.72.36-27580 (192.168.72.36:27580) with 1 cores
17/07/18 22:05:35 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/4 on hostPort 192.168.72.36:27580 with 1 cores, 4.0 GB RAM
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/5 on worker-20170601144000-192.168.72.37-23651 (192.168.72.37:23651) with 1 cores
17/07/18 22:05:35 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/5 on hostPort 192.168.72.37:23651 with 1 cores, 4.0 GB RAM
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/6 on worker-20170601175454-192.168.72.28-15845 (192.168.72.28:15845) with 1 cores
17/07/18 22:05:35 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/6 on hostPort 192.168.72.28:15845 with 1 cores, 4.0 GB RAM
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/7 on worker-20170601144214-192.168.72.52-21432 (192.168.72.52:21432) with 1 cores
17/07/18 22:05:35 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/7 on hostPort 192.168.72.52:21432 with 1 cores, 4.0 GB RAM
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/8 on worker-20170601175523-192.168.72.45-21668 (192.168.72.45:21668) with 1 cores
17/07/18 22:05:35 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/8 on hostPort 192.168.72.45:21668 with 1 cores, 4.0 GB RAM
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/9 on worker-20170601154456-192.168.72.35-4759 (192.168.72.35:4759) with 1 cores
17/07/18 22:05:35 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/9 on hostPort 192.168.72.35:4759 with 1 cores, 4.0 GB RAM
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/1 is now RUNNING
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/4 is now RUNNING
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/2 is now RUNNING
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/6 is now RUNNING
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/5 is now RUNNING
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/7 is now RUNNING
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/0 is now RUNNING
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/3 is now RUNNING
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/8 is now RUNNING
17/07/18 22:05:35 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/9 is now RUNNING
17/07/18 22:05:35 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
17/07/18 22:05:35 INFO SharedState: Warehouse path is 'file:/home/dean/work/gitlab/gitlab.com/mystudy/spark/code/scala/sbt-projects/hellospark/spark-warehouse/'.
17/07/18 22:05:35 INFO root: MongoURL: mongodb://192.168.72.19:27017/device_detail_tags.js_watchVideo_weekday_3
17/07/18 22:05:35 INFO root: Weekday: 3
17/07/18 22:05:35 INFO root: Hour: 9
17/07/18 22:05:35 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 232.0 B, free 366.3 MB)
17/07/18 22:05:35 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 414.0 B, free 366.3 MB)
17/07/18 22:05:35 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.31.161:40689 (size: 414.0 B, free: 366.3 MB)
17/07/18 22:05:35 INFO SparkContext: Created broadcast 0 from broadcast at MongoSpark.scala:499
17/07/18 22:05:35 INFO cluster: Cluster created with settings {hosts=[192.168.72.19:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
17/07/18 22:05:35 INFO cluster: Cluster description not yet available. Waiting for 30000 ms before timing out
17/07/18 22:05:36 INFO connection: Opened connection [connectionId{localValue:1, serverValue:136103}] to 192.168.72.19:27017
17/07/18 22:05:36 INFO cluster: Monitor thread successfully connected to server with description ServerDescription{address=192.168.72.19:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 4, 4]}, minWireVersion=0, maxWireVersion=5, maxDocumentSize=16777216, roundTripTimeNanos=34882275}
17/07/18 22:05:36 INFO MongoClientCache: Creating MongoClient: [192.168.72.19:27017]
17/07/18 22:05:36 INFO connection: Opened connection [connectionId{localValue:2, serverValue:136104}] to 192.168.72.19:27017
17/07/18 22:05:37 INFO SparkContext: Starting job: treeAggregate at MongoInferSchema.scala:76
17/07/18 22:05:37 INFO DAGScheduler: Got job 0 (treeAggregate at MongoInferSchema.scala:76) with 1 output partitions
17/07/18 22:05:37 INFO DAGScheduler: Final stage: ResultStage 0 (treeAggregate at MongoInferSchema.scala:76)
17/07/18 22:05:37 INFO DAGScheduler: Parents of final stage: List()
17/07/18 22:05:37 INFO DAGScheduler: Missing parents: List()
17/07/18 22:05:37 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[4] at treeAggregate at MongoInferSchema.scala:76), which has no missing parents
17/07/18 22:05:37 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.8 KB, free 366.3 MB)
17/07/18 22:05:37 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.6 KB, free 366.3 MB)
17/07/18 22:05:37 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.31.161:40689 (size: 2.6 KB, free: 366.3 MB)
17/07/18 22:05:37 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:996
17/07/18 22:05:37 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[4] at treeAggregate at MongoInferSchema.scala:76)
17/07/18 22:05:37 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/07/18 22:05:42 INFO MongoClientCache: Closing MongoClient: [192.168.72.19:27017]
17/07/18 22:05:42 INFO connection: Closed connection [connectionId{localValue:2, serverValue:136104}] to 192.168.72.19:27017 because the pool has been closed.
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/4 is now EXITED (Command exited with code 1)
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Executor app-20170718220535-0379/4 removed: Command exited with code 1
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/10 on worker-20170601143944-192.168.72.36-27580 (192.168.72.36:27580) with 1 cores
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/10 on hostPort 192.168.72.36:27580 with 1 cores, 4.0 GB RAM
17/07/18 22:05:51 INFO BlockManagerMaster: Removal of executor 4 requested
17/07/18 22:05:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 4
17/07/18 22:05:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 4 from BlockManagerMaster.
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/10 is now RUNNING
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/3 is now EXITED (Command exited with code 1)
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Executor app-20170718220535-0379/3 removed: Command exited with code 1
17/07/18 22:05:51 INFO BlockManagerMaster: Removal of executor 3 requested
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/11 on worker-20170711142123-192.168.72.54-31587 (192.168.72.54:31587) with 1 cores
17/07/18 22:05:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 3
17/07/18 22:05:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 3 from BlockManagerMaster.
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/11 on hostPort 192.168.72.54:31587 with 1 cores, 4.0 GB RAM
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/11 is now RUNNING
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/1 is now EXITED (Command exited with code 1)
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Executor app-20170718220535-0379/1 removed: Command exited with code 1
17/07/18 22:05:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster.
17/07/18 22:05:51 INFO BlockManagerMaster: Removal of executor 1 requested
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/12 on worker-20170711142458-192.168.72.18-25899 (192.168.72.18:25899) with 1 cores
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/12 on hostPort 192.168.72.18:25899 with 1 cores, 4.0 GB RAM
17/07/18 22:05:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 1
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/12 is now RUNNING
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/0 is now EXITED (Command exited with code 1)
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Executor app-20170718220535-0379/0 removed: Command exited with code 1
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/13 on worker-20170601143814-192.168.72.44-12395 (192.168.72.44:12395) with 1 cores
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/13 on hostPort 192.168.72.44:12395 with 1 cores, 4.0 GB RAM
17/07/18 22:05:51 INFO BlockManagerMaster: Removal of executor 0 requested
17/07/18 22:05:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 0
17/07/18 22:05:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 0 from BlockManagerMaster.
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/7 is now EXITED (Command exited with code 1)
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Executor app-20170718220535-0379/7 removed: Command exited with code 1
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/14 on worker-20170601144214-192.168.72.52-21432 (192.168.72.52:21432) with 1 cores
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/14 on hostPort 192.168.72.52:21432 with 1 cores, 4.0 GB RAM
17/07/18 22:05:51 INFO BlockManagerMaster: Removal of executor 7 requested
17/07/18 22:05:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 7 from BlockManagerMaster.
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/13 is now RUNNING
17/07/18 22:05:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 7
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/14 is now RUNNING
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/2 is now EXITED (Command exited with code 1)
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Executor app-20170718220535-0379/2 removed: Command exited with code 1
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/15 on worker-20170711142201-192.168.72.53-21310 (192.168.72.53:21310) with 1 cores
17/07/18 22:05:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 2 from BlockManagerMaster.
17/07/18 22:05:51 INFO BlockManagerMaster: Removal of executor 2 requested
17/07/18 22:05:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 2
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/15 on hostPort 192.168.72.53:21310 with 1 cores, 4.0 GB RAM
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/15 is now RUNNING
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/9 is now EXITED (Command exited with code 1)
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Executor app-20170718220535-0379/9 removed: Command exited with code 1
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/16 on worker-20170601154456-192.168.72.35-4759 (192.168.72.35:4759) with 1 cores
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/16 on hostPort 192.168.72.35:4759 with 1 cores, 4.0 GB RAM
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/16 is now RUNNING
17/07/18 22:05:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 9 from BlockManagerMaster.
17/07/18 22:05:51 INFO BlockManagerMaster: Removal of executor 9 requested
17/07/18 22:05:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 9
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/5 is now EXITED (Command exited with code 1)
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Executor app-20170718220535-0379/5 removed: Command exited with code 1
17/07/18 22:05:51 INFO BlockManagerMaster: Removal of executor 5 requested
17/07/18 22:05:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 5 from BlockManagerMaster.
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/17 on worker-20170601144000-192.168.72.37-23651 (192.168.72.37:23651) with 1 cores
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/17 on hostPort 192.168.72.37:23651 with 1 cores, 4.0 GB RAM
17/07/18 22:05:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 5
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/17 is now RUNNING
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/6 is now EXITED (Command exited with code 1)
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Executor app-20170718220535-0379/6 removed: Command exited with code 1
17/07/18 22:05:51 INFO BlockManagerMaster: Removal of executor 6 requested
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/18 on worker-20170601175454-192.168.72.28-15845 (192.168.72.28:15845) with 1 cores
17/07/18 22:05:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 6 from BlockManagerMaster.
17/07/18 22:05:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 6
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/18 on hostPort 192.168.72.28:15845 with 1 cores, 4.0 GB RAM
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/18 is now RUNNING
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/8 is now EXITED (Command exited with code 1)
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Executor app-20170718220535-0379/8 removed: Command exited with code 1
17/07/18 22:05:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster.
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/19 on worker-20170601175523-192.168.72.45-21668 (192.168.72.45:21668) with 1 cores
17/07/18 22:05:51 INFO BlockManagerMaster: Removal of executor 8 requested
17/07/18 22:05:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 8
17/07/18 22:05:51 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/19 on hostPort 192.168.72.45:21668 with 1 cores, 4.0 GB RAM
17/07/18 22:05:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/19 is now RUNNING
17/07/18 22:05:52 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
17/07/18 22:06:07 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
17/07/18 22:06:08 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/10 is now EXITED (Command exited with code 1)
17/07/18 22:06:08 INFO StandaloneSchedulerBackend: Executor app-20170718220535-0379/10 removed: Command exited with code 1
17/07/18 22:06:08 INFO BlockManagerMaster: Removal of executor 10 requested
17/07/18 22:06:08 INFO BlockManagerMasterEndpoint: Trying to remove executor 10 from BlockManagerMaster.
17/07/18 22:06:08 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 10
17/07/18 22:06:08 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/20 on worker-20170601143944-192.168.72.36-27580 (192.168.72.36:27580) with 1 cores
17/07/18 22:06:08 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/20 on hostPort 192.168.72.36:27580 with 1 cores, 4.0 GB RAM
17/07/18 22:06:08 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/20 is now RUNNING
17/07/18 22:06:08 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/13 is now EXITED (Command exited with code 1)
17/07/18 22:06:08 INFO StandaloneSchedulerBackend: Executor app-20170718220535-0379/13 removed: Command exited with code 1
17/07/18 22:06:08 INFO BlockManagerMaster: Removal of executor 13 requested
17/07/18 22:06:08 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 13
17/07/18 22:06:08 INFO BlockManagerMasterEndpoint: Trying to remove executor 13 from BlockManagerMaster.
17/07/18 22:06:08 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170718220535-0379/21 on worker-20170601143814-192.168.72.44-12395 (192.168.72.44:12395) with 1 cores
17/07/18 22:06:08 INFO StandaloneSchedulerBackend: Granted executor ID app-20170718220535-0379/21 on hostPort 192.168.72.44:12395 with 1 cores, 4.0 GB RAM
17/07/18 22:06:08 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/21 is now RUNNING
17/07/18 22:06:08 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170718220535-0379/11 is now EXITED (Command exited with code 1)
17/07/18 22:06:08 INFO StandaloneSchedulerBackend: Executor app-20170718220535-0379/11 removed: Command exited with code 1
17/07/18 22:06:08 INFO BlockManagerMasterEndpoint: Trying to remove executor 11 from BlockManagerMaster.
17/07/18 22:06:08 INFO BlockManagerMaster: Removal of executor 11 requested
您是否尝试过使用此libraryDependencies+=“org.mongodb.spark”%%“mongo spark connector”%%“2.1.1”？我尝试过在spark帐户下的服务器上提交，它很快就会工作并退出。上面我运行int的问题是在我的笔记本电脑上的dean帐户下。这可能是个线索。@Shankar，是的，我刚开始试过最新版本的连接器