Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Apache spark 火花驱动器存储器配置_Apache Spark_Amazon Ec2_Spark Dataframe - Fatal编程技术网

Apache spark 火花驱动器存储器配置

Apache spark 火花驱动器存储器配置,apache-spark,amazon-ec2,spark-dataframe,Apache Spark,Amazon Ec2,Spark Dataframe,我的Spark应用程序中出现以下错误:- Container [pid=7115,containerID=container_1516699670283_0001_02_000001] is running beyond physical memory limits. Current usage: 1.4 GB of 1.4 GB physical memory used; 3.4 GB of 6.9 GB virtual memory used. Killing container. 我目前

我的Spark应用程序中出现以下错误:-

Container [pid=7115,containerID=container_1516699670283_0001_02_000001] is running beyond physical memory limits. Current usage: 1.4 GB of 1.4 GB physical memory used; 3.4 GB of 6.9 GB virtual memory used. Killing container.
我目前正在使用
m3.2x大型EC2
实例()作为我的驱动程序资源

我这里有几个问题:

1) 如果我没有指定
spark.driver.memory
,是否使用任何默认值

2) 在这种情况下,我应该指定驱动程序的哪个内存?

1)spark.driver.memory的默认值是1g。
(见附件)

2) 您需要指定spark.driver.memory、spark.driver.cores、spark.driver.maxResultSize(在链接后面阅读此参数)