Java 火花塞的清洁和安装

在过去的几天里，我一直在解决这个问题：

我试图从MLLIB运行随机林，它可以通过大部分，但在执行mapPartition操作时会中断。显示以下堆栈跟踪：

: An error occurred while calling o94.trainRandomForestModel. 
: java.lang.OutOfMemoryError 
        at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123) 
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117) 
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) 
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) 
        at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877) 
        at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786) 
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189) 
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) 
        at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44) 
        at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:84) 
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301) 
        at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294) 
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122) 
        at org.apache.spark.SparkContext.clean(SparkContext.scala:2021) 
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:703) 
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:702) 
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) 
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) 
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:306) 
        at org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:702) 
        at org.apache.spark.mllib.tree.DecisionTree$.findBestSplits(DecisionTree.scala:625) 
        at org.apache.spark.mllib.tree.RandomForest.run(RandomForest.scala:235) 
        at org.apache.spark.mllib.tree.RandomForest$.trainClassifier(RandomForest.scala:291) 
        at org.apache.spark.mllib.api.python.PythonMLLibAPI.trainRandomForestModel(PythonMLLibAPI.scala:742) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:497) 
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) 
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) 
        at py4j.Gateway.invoke(Gateway.java:259) 
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) 
        at py4j.commands.CallCommand.execute(CallCommand.java:79) 
        at py4j.GatewayConnection.run(GatewayConnection.java:207) 
        at java.lang.Thread.run(Thread.java:745) 

在我看来，它试图序列化mapPartitions闭包，但这样做会耗尽空间。然而，我不明白当我给驱动程序~190GB的45MB文件时，它怎么会耗尽空间

我在AWS上有一个集群设置，这样我的主节点是一个r3.8XL以及两个r3.4XL工作节点。我有以下配置：

spark version: 1.5.0 
----------------------------------- 
spark.executor.memory 32000m 
spark.driver.memory 230000m 
spark.driver.cores 10 
spark.executor.cores 5 
spark.executor.instances 17 
spark.driver.maxResultSize 0 
spark.storage.safetyFraction 1 
spark.storage.memoryFraction 0.9 
spark.storage.shuffleFraction 0.05 
spark.default.parallelism 128 

主机大约有240 GB的ram，每个工人大约有120 GB的ram

---

我加载了一个相对较小的MLLIB LabeledPoint对象RDD，其中每个对象都包含稀疏向量。此RDD的总大小约为45MB。我的稀疏向量的总长度约为1500万，而只有大约3000个非零

你能解决这个问题吗？我也有同样的问题。