Python RandomForestClassifier.fit在ec2上因内存错误而失败，但在本地运行时没有错误

我在熊猫数据帧上安装了一个随机森林分类器，该数据帧包含tfidf和其他维度为（42238155085）的特征。大小是26GB。在本地训练模型时，代码运行时不会出错（尽管速度很慢），但是在内存为4倍的ec2实例上，当内存利用率达到37%时，训练过程会因内存错误而终止。这在6次运行时执行中是一致的。在代码和数据都与Anaconda2.7相同的情况下，为什么这个过程会在ec2上失败

我正在16GB macbook pro上本地运行该型号。ec2实例有64GB内存

我试过的

n_jobs=1

n_估计量=10

最大深度=10

增加ec2上的交换空间

将最大锁定内存设置为无限

ulimit-a在ec2实例上。 代码 堆栈跟踪 以此为指导：

调整了以下参数，模型能够完成训练

vm.swappiness=70

鼓励使用交换空间

vm.vfs\u cache\u pressure=50

鼓励操作系统更多地使用缓存

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 251728
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 251728
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

model = RandomForestClassifier(max_depth=None, 
                       max_features=0.2, 
                       min_samples_leaf=1, 
                       min_samples_split=2,
                       n_estimators=10, 
                       n_jobs=-1,
                       verbose=1)
model.fit(X_train,y_train)

model.fit(X_train,y_train)
  File "/opt/anaconda2/lib/python2.7/site-packages/sklearn/ensemble/forest.py", line 247, in fit
    X = check_array(X, accept_sparse="csc", dtype=DTYPE)
  File "/opt/anaconda2/lib/python2.7/site-packages/sklearn/utils/validation.py", line 433, in check_array
    array = np.array(array, dtype=dtype, order=order, copy=copy)
  File "/opt/anaconda2/lib/python2.7/site-packages/pandas/core/generic.py", line 1603, in __array__
    return com._values_from_object(self)
  File "pandas/_libs/lib.pyx", line 47, in pandas._libs.lib.values_from_object
  File "/opt/anaconda2/lib/python2.7/site-packages/pandas/core/generic.py", line 4684, in get_values
    return self.values
  File "/opt/anaconda2/lib/python2.7/site-packages/pandas/core/generic.py", line 4629, in values
    return self._data.as_array(transpose=self._AXIS_REVERSED)
  File "/opt/anaconda2/lib/python2.7/site-packages/pandas/core/internals.py", line 3949, in as_array
    arr = mgr._interleave()
  File "/opt/anaconda2/lib/python2.7/site-packages/pandas/core/internals.py", line 3960, in _interleave
    result = np.empty(self.shape, dtype=dtype)
MemoryError