apachespark耗尽了Java堆空间：它发生在哪里？

我的Spark有Java内存问题。在我的8GB Mac上运行的同一个应用程序在我的72GB Ubuntu服务器上崩溃

我已经更改了conf文件中的内容，但是看起来Spark并不在意，所以我想知道我的问题是与驱动程序还是执行程序有关

我设定：

spark.driver.memory             20g
spark.executor.memory           20g

而且，无论我做什么，车祸总是在应用程序中的同一个位置，这让我认为这是一个司机的问题

我得到的例外是：

16/07/13 20:36:30 WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID 208, micha.nc.rr.com): java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
    at java.nio.CharBuffer.allocate(CharBuffer.java:335)
    at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:810)
    at org.apache.hadoop.io.Text.decode(Text.java:412)
    at org.apache.hadoop.io.Text.decode(Text.java:389)
    at org.apache.hadoop.io.Text.toString(Text.java:280)
    at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
    at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:144)
    at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1157)
    at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:201)
    at scala.collection.AbstractIterator.aggregate(Iterator.scala:1157)
    at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
    at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
    at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
    at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

就在坠机前，上面写着：

**  Free ......... 1,413,566
**  Allocated .... 1,705,984
**  Max .......... 16,495,104
**> Total free ... 16,202,686

**  Free ......... 1,461,633
**  Allocated .... 1,786,880
**  Max .......... 16,495,104
**> Total free ... 16,169,857

---

因此，由于某些原因，我无法让Spark在服务器端读取配置文件，但将代码修改为：

SparkConf conf = new SparkConf()
    .setAppName("app")
    .set("spark.executor.memory", "4g")
    .setMaster("spark://10.0.100.120:7077");

---

（感谢所有投票否决该问题的人，回到这里发布解决方案真的很有动力）。

每台机器中有什么JVM？从堆栈跟踪来看，这是executor中的崩溃。@Leo:他们都运行Java 8，Spark 1.6。2@Dikei：这也是我的结论，但是我不明白为什么执行器不从配置文件中获取内存参数。它在我动态设置它时起作用（参见我的答案）。