Python 在ProcessPoolExecutor中打开数据帧时,永远不会释放内存

Python 在ProcessPoolExecutor中打开数据帧时,永远不会释放内存,python,pandas,multiprocessing,concurrent.futures,Python,Pandas,Multiprocessing,Concurrent.futures,下面是一个非常简单的代码示例 from concurrent.futures import ProcessPoolExecutor import pandas if __name__ == "__main__": def i_use_lots_of_memory(): print 'doing something that uses a lot of memory' data = pandas.read_csv('large_txt_file.txt

下面是一个非常简单的代码示例

from concurrent.futures import ProcessPoolExecutor
import pandas


if __name__ == "__main__":

    def i_use_lots_of_memory():
        print 'doing something that uses a lot of memory'
        data = pandas.read_csv('large_txt_file.txt')
        del data
        # do other things here as soon as I've solved mem usage issues
        print 'ha ha  I used up a ton of memory.'

    def simplest_callback_ever(future):
        _ = future.result()
        print 'callback was run'


    class ManagesFileReading(object):
        def __init__(self):
            self.pool = ProcessPoolExecutor(max_workers=24)

        def add_job(self, callback=None):
            future = self.pool.submit(i_use_lots_of_memory)
            if callback:
                future.add_done_callback(callback)


    mfr = ManagesFileReading()
    mfr.add_job(simplest_callback_ever)
在这个例子中,我打开了一个800MB的文本文件,它占用了大约2GB的内存。输出是

doing something that uses a lot of memory
ha ha  I used up a ton of memory.
callback function was run. Task is complete.
因此,任务完成后,问题是内存从未释放。即使未来已经完成,它也不会释放记忆。我可以释放它的唯一方法是通过运行self关闭进程池。_pool.shutdown()

除非我误解了ProcessPoolExecutor的工作原理,否则当回调函数完成时,意味着任务完成了,对吗?为什么不删除未来,释放内存?有什么想法吗