Multithreading Pyspark:OOM无法创建新线程

我在pyspark shell上运行一个Spark任务并获得

Exception in thread "stdout writer for python2.7" java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:714)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:92)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
    at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
...
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
    at org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:248)
    at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
    at org.apache.spark.api.python.PythonRDD$WriterThread.run(PythonRDD.scala:208)

作业是从拼花文件中拾取数据帧，转换为RDD，执行转换并将结果存储到其他拼花文件

在1节点独立模式下运行以及在小型2节点纱线集群上运行时，也会出现相同的问题。 当我增加默认pyspark shell参数时，同样的问题仍然存在：

pyspark --driver-memory 8G --executor-memory 8G

在抛出异常（处理停止）之前，控制台中有两个警告：

15/09/13 00:12:17 WARN TaskSetManager: Stage 186 contains a task of very large size (4722 KB). The maximum recommended task size is 100 KB.
15/09/13 00:12:19 ERROR ShuffleBlockFetcherIterator: Error occurred while fetching local blocks
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
    at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
    at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.fetchLocalBlocks(ShuffleBlockFetcherIterator.scala:239)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.initialize(ShuffleBlockFetcherIterator.scala:269)
    at org.apache.spark.storage.ShuffleBlockFetcherIterator.<init>(ShuffleBlockFetcherIterator.scala:115)
    at org.apache.spark.shuffle.hash.BlockStoreShuffleFetcher$.fetch(BlockStoreShuffleFetcher.scala:76)
    at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:40)
    at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:90)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
    at org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:248)
    at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
    at org.apache.spark.api.python.PythonRDD$WriterThread.run(PythonRDD.scala:208)

15/09/13 00:12:17警告任务集管理器：阶段186包含一个非常大的任务（4722 KB）。建议的最大任务大小为100 KB。
15/09/13 00:12:19错误ShuffleBlockFetcherInterator:获取本地块时出错
java.lang.InterruptedException
位于java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly（AbstractQueuedSynchronizer.java:1220）
在java.util.concurrent.locks.ReentrantLock.lockInterruptablely（ReentrantLock.java:335）处
位于java.util.concurrent.LinkedBlockingQueue.put（LinkedBlockingQueue.java:339）
位于org.apache.spark.storage.ShuffleBlockFetcherIterator.fetchLocalBlocks（ShuffleBlockFetcherIterator.scala:239）
位于org.apache.spark.storage.ShuffleBlockFetcherIterator.initialize（ShuffleBlockFetcherIterator.scala:269）
位于org.apache.spark.storage.ShuffleBlockFetcherIterator。（ShuffleBlockFetcherIterator.scala:115）
位于org.apache.spark.shuffle.hash.blockstoreshuffleetcher$.fetch（blockstoreshuffleetcher.scala:76）
在org.apache.spark.shuffle.hash.hashuffleReader.read上（hashuffleReader.scala:40）
位于org.apache.spark.rdd.shuffleddd.compute（shuffleddd.scala:90）
在org.apache.spark.rdd.rdd.computeOrReadCheckpoint（rdd.scala:277）
位于org.apache.spark.rdd.rdd.iterator（rdd.scala:244）
在org.apache.spark.rdd.MapPartitionsRDD.compute上（MapPartitionsRDD.scala:35）
在org.apache.spark.rdd.rdd.computeOrReadCheckpoint（rdd.scala:277）
位于org.apache.spark.rdd.rdd.iterator（rdd.scala:244）
位于org.apache.spark.api.PythonRDD$WriterThread$$anonfun$run$3.apply（PythonRDD.scala:248）
位于org.apache.spark.util.Utils$.logUncaughtExceptions（Utils.scala:1772）
位于org.apache.spark.api.python.PythonRDD$WriterThread.run（PythonRDD.scala:208）

及

15/09/13 00:12:21警告任务集管理器：阶段369包含一个非常大的任务（4722 KB）。建议的最大任务大小为100 KB。
15/09/13 00:12:21错误ShuffleBlockFetcherInterator:获取本地块时出错
java.lang.InterruptedException
位于java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly（AbstractQueuedSynchronizer.java:1220）
在java.util.concurrent.locks.ReentrantLock.lockInterruptablely（ReentrantLock.java:335）处
位于java.util.concurrent.LinkedBlockingQueue.put（LinkedBlockingQueue.java:339）
位于org.apache.spark.storage.ShuffleBlockFetcherIterator.fetchLocalBlocks（ShuffleBlockFetcherIterator.scala:239）
位于org.apache.spark.storage.ShuffleBlockFetcherIterator.initialize（ShuffleBlockFetcherIterator.scala:269）
位于org.apache.spark.storage.ShuffleBlockFetcherIterator。（ShuffleBlockFetcherIterator.scala:115）
位于org.apache.spark.shuffle.hash.blockstoreshuffleetcher$.fetch（blockstoreshuffleetcher.scala:76）
在org.apache.spark.shuffle.hash.hashuffleReader.read上（hashuffleReader.scala:40）
位于org.apache.spark.rdd.shuffleddd.compute（shuffleddd.scala:90）
在org.apache.spark.rdd.rdd.computeOrReadCheckpoint（rdd.scala:277）
位于org.apache.spark.rdd.rdd.iterator（rdd.scala:244）
在org.apache.spark.rdd.MapPartitionsRDD.compute上（MapPartitionsRDD.scala:35）
在org.apache.spark.rdd.rdd.computeOrReadCheckpoint（rdd.scala:277）
位于org.apache.spark.rdd.rdd.iterator（rdd.scala:244）
位于org.apache.spark.api.PythonRDD$WriterThread$$anonfun$run$3.apply（PythonRDD.scala:248）
位于org.apache.spark.util.Utils$.logUncaughtExceptions（Utils.scala:1772）
位于org.apache.spark.api.python.PythonRDD$WriterThread.run（PythonRDD.scala:208）

对类似问题的一些回答提到，这种情况的发生是因为火花的记忆没有得到适当的调整。 如何调整Spark内存？哪些设置会影响此行为

第一个堆栈跟踪是巨大的，由相同的几个被递归调用的方法组成。如果我在崩溃时也监视Spark JVM，那么当前正在运行的线程将出现一个巨大的峰值，这些线程“突破了顶部”。 所有这些看起来都像是某种基于Java

ThreadPoolExector

的糟糕的内部实现，它没有限制线程的数量，以防任务开始处理得比新任务来得慢

哪些配置旋钮可以帮助我防止突然的线程创建高峰

15/09/13 00:12:21 WARN TaskSetManager: Stage 369 contains a task of very large size (4722 KB). The maximum recommended task size is 100 KB.
15/09/13 00:12:21 ERROR ShuffleBlockFetcherIterator: Error occurred while fetching local blocks
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.fetchLocalBlocks(ShuffleBlockFetcherIterator.scala:239)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.initialize(ShuffleBlockFetcherIterator.scala:269)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.<init>(ShuffleBlockFetcherIterator.scala:115)
at org.apache.spark.shuffle.hash.BlockStoreShuffleFetcher$.fetch(BlockStoreShuffleFetcher.scala:76)
at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:40)
at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:90)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:248)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
at org.apache.spark.api.python.PythonRDD$WriterThread.run(PythonRDD.scala:208)