Python 如何处理；溢出错误：大小不适合整型“；错误？

我正在运行一个Spark作业，如果我对样本数据执行计算（想想~1000行），一切都会正常工作。但当我尝试在更大的数据集上执行相同的计算时，我得到

19/07/20 14:21:53 WARN TaskSetManager: Lost task 198.0 in stage 150.0 (TID 21928, 10.46.225.176, executor 17): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/databricks/spark/python/pyspark/worker.py", line 403, in main
    process()
  File "/databricks/spark/python/pyspark/worker.py", line 398, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/databricks/spark/python/pyspark/rdd.py", line 2516, in pipeline_func
    return func(split, prev_func(split, iterator))
  File "/databricks/spark/python/pyspark/rdd.py", line 2516, in pipeline_func
    return func(split, prev_func(split, iterator))
  File "/databricks/spark/python/pyspark/rdd.py", line 352, in func
    return f(iterator)
  File "/databricks/spark/python/pyspark/rdd.py", line 1886, in _mergeCombiners
    merger.mergeCombiners(iterator)
  File "/databricks/spark/python/pyspark/shuffle.py", line 289, in mergeCombiners
    self._spill()
  File "/databricks/spark/python/pyspark/shuffle.py", line 317, in _spill
    self.serializer.dump_stream([(k, v)], streams[h])
  File "/databricks/spark/python/pyspark/serializers.py", line 417, in dump_stream
    bytes = self.serializer.dumps(vs)
  File "/databricks/spark/python/pyspark/serializers.py", line 679, in dumps
    return zlib.compress(self.serializer.dumps(obj), 1)
OverflowError: size does not fit in an int

    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:490)
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:626)
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:609)
    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:444)
    at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1187)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1161)
    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1096)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1161)
    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:883)
    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:351)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:302)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:75)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:340)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:304)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.doRunTask(Task.scala:139)
    at org.apache.spark.scheduler.Task.run(Task.scala:112)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1481)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

什么触发了它？我想在收款操作的最后阶段

rdd.take（500）

到目前为止，我已经尝试：

重新分区到4000个分区。没有帮助/如果有，我无法理解

使用一个大型集群-m5.xlarge+r4.4xlagle（16个工人）。使用较小的集群会对此有所帮助。大型集群是否可能导致某些序列化问题

使用Python2.7，因为我使用的库是用2.7编写的。我看到一篇帖子说zlib可能有问题，但我不知道如何解决它

---

我觉得我对这个问题的有限理解已经用尽了。非常感谢任何可能有帮助的指导或事情。请不要标记它是重复的，因为我已经检查了它周围的几篇文章，没有发现任何有用的东西。

最有可能的是，对于单个缓冲区，您达到了zlib的2G限制。如果可能，请更新您的python版本

尝试下面的方法，看看是否适合你。第二行应该失败

> python2 -c "import zlib; zlib.compress(b'a' * (2**31 - 1))"
> python2 -c "import zlib; zlib.compress(b'a' * (2**31))"

---

更多信息：

对python2 for spark的支持即将结束。这听起来可能很奇怪，但当我将集群的大小增加一倍时，我就不再有问题了。您可能知道是什么原因导致的吗？将集群的大小增加一倍基本上就是减少了每个节点的分区大小，所以我敢打赌它会起作用。尝试相反的方法，查看数据集的大小是否超过2g限制导致了问题。在任何情况下，尝试为每个节点提供更小但大小最佳的分区。