Spark streaming 在spark提交时识别live spark master
我有5个节点的spark集群,其中2个节点运行master。在HA(由动物园管理员)场景中,任何人都将被选为主人Spark streaming 在spark提交时识别live spark master,spark-streaming,Spark Streaming,我有5个节点的spark集群,其中2个节点运行master。在HA(由动物园管理员)场景中,任何人都将被选为主人 at the time of submitting application using command /bin/spark-submit --class SparkAggregator.java --deploy-mode cluster --supervise --master spark://host1:7077 getting error Can only accep
at the time of submitting application using command
/bin/spark-submit --class SparkAggregator.java --deploy-mode cluster --supervise --master spark://host1:7077
getting error
Can only accept driver submissions in ALIVE state. Current state: STANDBY.
spark-submit doe not allow multiple master name in --master.
Question:
How to identify the elected master at the time of spark-submit.
Thanks
Pankaj
master选项可以接受多个spark master,因此如果有多个spark master,请使用逗号列出它们。e、 g /bin/spark submit——类SparkAggregator.java——部署模式集群——监控——masterspark://host1:7077,host2:7077,host3:7077
如果将尝试连接到所有主机,则使用第一个响应的主机,这允许您在群集中使用多个主机,其中只有一个主机处于活动状态,其余主机处于备用状态。Spark有一个隐藏的API,告诉您Spark群集的状态 API请求- 输出-
{
"url" : "spark://10.204.216.233:7077",
"workers" : [ {
"id" : "worker-20170606104140-10.204.217.96-40047",
"host" : "10.204.217.96",
"port" : 40047,
"webuiaddress" : "http://10.204.217.96:8081",
"cores" : 4,
"coresused" : 0,
"coresfree" : 4,
"memory" : 29713,
"memoryused" : 0,
"memoryfree" : 29713,
"state" : "ALIVE",
"lastheartbeat" : 1496760671542
}, {
"id" : "worker-20170606104144-10.204.219.15-42749",
"host" : "10.204.219.15",
"port" : 42749,
"webuiaddress" : "http://10.204.219.15:8081",
"cores" : 4,
"coresused" : 0,
"coresfree" : 4,
"memory" : 29713,
"memoryused" : 0,
"memoryfree" : 29713,
"state" : "ALIVE",
"lastheartbeat" : 1496760675649
}, {
"id" : "worker-20170606104151-10.204.217.249-35869",
"host" : "10.204.217.249",
"port" : 35869,
"webuiaddress" : "http://10.204.217.249:8081",
"cores" : 4,
"coresused" : 0,
"coresfree" : 4,
"memory" : 29713,
"memoryused" : 0,
"memoryfree" : 29713,
"state" : "ALIVE",
"lastheartbeat" : 1496760682270
} ],
"cores" : 12,
"coresused" : 0,
"memory" : 89139,
"memoryused" : 0,
"activeapps" : [ ],
"completedapps" : [ ],
"activedrivers" : [ ],
"status" : "ALIVE"
}
它在1.3.0版本上不起作用。您在其他版本上试用过吗?