Amazon web services 如何优化纱线的火花驱动性能

我正在尝试在纱线中设置火花驱动程序内存的各种选项

用例：

spark.submit.deployMode client
spark.driver.cores  7
spark.driver.memory 24G
spark.driver.memoryOverhead 3072M
spark.executor.cores    1
spark.executor.memory   3G
spark.executor.memoryOverhead   512M
spark.yarn.am.cores 1
spark.yarn.am.memory    3G
spark.yarn.am.memoryOverhead    512M

spark.submit.deployMode client
spark.driver.cores  7
spark.driver.memory 12G
spark.driver.memoryOverhead 3072M
spark.executor.cores    1
spark.executor.memory   3G
spark.executor.memoryOverhead   512M
spark.yarn.am.cores 1
spark.yarn.am.memory    3G
spark.yarn.am.memoryOverhead    512M

spark.driver.cores  2
spark.driver.memory 7G
spark.driver.memoryOverhead 1024M  

我有一个火花簇，有一个主火花簇和两个从火花簇

master : r5d xlarge - 8 vcore, 32GB
slave  : r5d xlarge - 8 vcore, 32GB

我正在使用ApacheZeppelin在spark集群上运行查询。Spark解释器配置有齐柏林飞艇提供的属性。我正在使用spark2.3.1在纱线上运行。我想创建4个解释器，这样4个用户可以并行使用这个集群

配置1:

spark.submit.deployMode client
spark.driver.cores  7
spark.driver.memory 24G
spark.driver.memoryOverhead 3072M
spark.executor.cores    1
spark.executor.memory   3G
spark.executor.memoryOverhead   512M
spark.yarn.am.cores 1
spark.yarn.am.memory    3G
spark.yarn.am.memoryOverhead    512M

spark.submit.deployMode client
spark.driver.cores  7
spark.driver.memory 12G
spark.driver.memoryOverhead 3072M
spark.executor.cores    1
spark.executor.memory   3G
spark.executor.memoryOverhead   512M
spark.yarn.am.cores 1
spark.yarn.am.memory    3G
spark.yarn.am.memoryOverhead    512M

spark.driver.cores  2
spark.driver.memory 7G
spark.driver.memoryOverhead 1024M  

以下是spark executor用户界面：

配置2:

spark.submit.deployMode client
spark.driver.cores  7
spark.driver.memory 24G
spark.driver.memoryOverhead 3072M
spark.executor.cores    1
spark.executor.memory   3G
spark.executor.memoryOverhead   512M
spark.yarn.am.cores 1
spark.yarn.am.memory    3G
spark.yarn.am.memoryOverhead    512M

spark.submit.deployMode client
spark.driver.cores  7
spark.driver.memory 12G
spark.driver.memoryOverhead 3072M
spark.executor.cores    1
spark.executor.memory   3G
spark.executor.memoryOverhead   512M
spark.yarn.am.cores 1
spark.yarn.am.memory    3G
spark.yarn.am.memoryOverhead    512M

spark.driver.cores  2
spark.driver.memory 7G
spark.driver.memoryOverhead 1024M  

以下是spark executor用户界面：

问题：

spark.submit.deployMode client
spark.driver.cores  7
spark.driver.memory 24G
spark.driver.memoryOverhead 3072M
spark.executor.cores    1
spark.executor.memory   3G
spark.executor.memoryOverhead   512M
spark.yarn.am.cores 1
spark.yarn.am.memory    3G
spark.yarn.am.memoryOverhead    512M

spark.submit.deployMode client
spark.driver.cores  7
spark.driver.memory 12G
spark.driver.memoryOverhead 3072M
spark.executor.cores    1
spark.executor.memory   3G
spark.executor.memoryOverhead   512M
spark.yarn.am.cores 1
spark.yarn.am.memory    3G
spark.yarn.am.memoryOverhead    512M

spark.driver.cores  2
spark.driver.memory 7G
spark.driver.memoryOverhead 1024M  

为什么驱动程序0的容器大小为

spark.memory.fraction是否计算为（spark.driver.memory-300）*0.6？如果是，为什么不确切？（分别为14.22、7.02）

为什么executor的容器大小为3.8 GB？根据我的配置，它应该是3G+512M=3.5 GB。spark 2.1不存在此问题

纱线可用的V孔数为每个节点8个。既然AWS提供了vCPU及其实例，这怎么可能呢？因此，根据AWS，我应该只获得4个vCore。

如果我想使用4个口译员，我是否应该将32 GB的master平均分配给所有口译员 驱动程序：

spark.submit.deployMode client
spark.driver.cores  7
spark.driver.memory 24G
spark.driver.memoryOverhead 3072M
spark.executor.cores    1
spark.executor.memory   3G
spark.executor.memoryOverhead   512M
spark.yarn.am.cores 1
spark.yarn.am.memory    3G
spark.yarn.am.memoryOverhead    512M

spark.submit.deployMode client
spark.driver.cores  7
spark.driver.memory 12G
spark.driver.memoryOverhead 3072M
spark.executor.cores    1
spark.executor.memory   3G
spark.executor.memoryOverhead   512M
spark.yarn.am.cores 1
spark.yarn.am.memory    3G
spark.yarn.am.memoryOverhead    512M

spark.driver.cores  2
spark.driver.memory 7G
spark.driver.memoryOverhead 1024M