Scikit learn 随机林使用太多内存

我正在尝试使用随机林（scikit learn 0.18.1安装了Anaconda；Python 3）进行研究。我的数据集包含大约325000个样本，每个样本由11个特征组成（所有非零值）。 我使用以下调用创建一个随机林（

max_depth

已设置为10以限制使用的内存量）：

不幸的是，随机林构建需要大量内存（我有128 GB的内存可供使用，100%的内存都已使用（使用

top

检索信息）。Python然后会引发一个MemoryError

我通过以下方式创建矩阵：

np.array(Xl, dtype=np.float32)

我怎么可能需要超过128 GB的RAM来完成这个相当轻松的任务呢？（即使当

n_jobs=1

时，我仍然有内存问题，但对于预测，使用的内存超过了可用内存…）

为了进行调试，我启动了以下命令：

dmesg | grep -E -i -B30 'killed process'

结果如下：

[1947333.164124] [ 1193]    81  1193     6137       52      16       81          -900 dbus-daemon
[1947333.164126] [ 1212]     0  1212    81781       73      80     5004             0 firewalld
[1947333.164127] [ 1213]     0  1213    31556       22      17      133             0 crond
[1947333.164129] [ 1215]     0  1215     6461        0      17       62             0 atd
[1947333.164131] [ 1228]     0  1228    27509        1      10       31             0 agetty
[1947333.164133] [ 1230]     0  1230   108909       60      65      487             0 NetworkManager
[1947333.164134] [ 1569]     0  1569    93416      153      91      181             0 rsyslogd
[1947333.164136] [ 1576]     0  1576   138290       63      87     2613             0 tuned
[1947333.164138] [ 1577]     0  1577    28335        1      11       37             0 rhsmcertd
[1947333.164140] [ 1582]     0  1582    20617       15      41      201         -1000 sshd
[1947333.164142] [ 1589]     0  1589    26978        8       7       28             0 rhnsd
[1947333.164143] [ 2221]     0  2221    22244        0      42      256             0 master
[1947333.164146] [ 2267]    89  2267    22287        0      42      251             0 qmgr
[1947333.164149] [19994]     0 19994    36365        2      73      326             0 sshd
[1947333.164151] [19996]  1002 19996    36365        0      68      329             0 sshd
[1947333.164153] [19997]  1002 19997    13175        0      29      142             0 sftp-server
[1947333.164155] [20826]     0 20826    36365       98      72      233             0 sshd
[1947333.164156] [20828]  1002 20828    36400      114      69      220             0 sshd
[1947333.164158] [20829]  1002 20829    28872       46      13       68             0 bash
[1947333.164160] [20862]     0 20862    36365        6      73      324             0 sshd
[1947333.164161] [20877]  1002 20877    36400       38      70      295             0 sshd
[1947333.164163] [20878]  1002 20878    28846        0      13      110             0 bash
[1947333.164164] [20899]  1002 20899    39521      198      30       72             0 top
[1947333.164166] [20929]     0 20929    36379      116      74      215             0 sshd
[1947333.164168] [20931]  1002 20931    36417      118      71      213             0 sshd
[1947333.164169] [20932]  1002 20932    28874       34      14       81             0 bash
[1947333.164171] [20972]  1002 20972    37348      229      27      468             0 vim
[1947333.164172] [20996]    89 20996    22270       83      44      164             0 pickup
[1947333.164174] [21075]  1002 21075 67384359 31985935   71321  4435535             0 python3
[1947333.164176] Out of memory: Kill process 21075 (python3) score 974 or sacrifice child
[1947333.164190] Killed process 21075 (python3) total-vm:269537436kB, anon-rss:127943740kB, file-rss:0kB, shmem-rss:0kB

---

好的，我找到了这个问题的解决方案

我在研究回归问题（而不是分类问题）

使用

randomForestRegressionor

而不是

RandomForestClassifier

解决了我的记忆问题

[1947333.164124] [ 1193]    81  1193     6137       52      16       81          -900 dbus-daemon
[1947333.164126] [ 1212]     0  1212    81781       73      80     5004             0 firewalld
[1947333.164127] [ 1213]     0  1213    31556       22      17      133             0 crond
[1947333.164129] [ 1215]     0  1215     6461        0      17       62             0 atd
[1947333.164131] [ 1228]     0  1228    27509        1      10       31             0 agetty
[1947333.164133] [ 1230]     0  1230   108909       60      65      487             0 NetworkManager
[1947333.164134] [ 1569]     0  1569    93416      153      91      181             0 rsyslogd
[1947333.164136] [ 1576]     0  1576   138290       63      87     2613             0 tuned
[1947333.164138] [ 1577]     0  1577    28335        1      11       37             0 rhsmcertd
[1947333.164140] [ 1582]     0  1582    20617       15      41      201         -1000 sshd
[1947333.164142] [ 1589]     0  1589    26978        8       7       28             0 rhnsd
[1947333.164143] [ 2221]     0  2221    22244        0      42      256             0 master
[1947333.164146] [ 2267]    89  2267    22287        0      42      251             0 qmgr
[1947333.164149] [19994]     0 19994    36365        2      73      326             0 sshd
[1947333.164151] [19996]  1002 19996    36365        0      68      329             0 sshd
[1947333.164153] [19997]  1002 19997    13175        0      29      142             0 sftp-server
[1947333.164155] [20826]     0 20826    36365       98      72      233             0 sshd
[1947333.164156] [20828]  1002 20828    36400      114      69      220             0 sshd
[1947333.164158] [20829]  1002 20829    28872       46      13       68             0 bash
[1947333.164160] [20862]     0 20862    36365        6      73      324             0 sshd
[1947333.164161] [20877]  1002 20877    36400       38      70      295             0 sshd
[1947333.164163] [20878]  1002 20878    28846        0      13      110             0 bash
[1947333.164164] [20899]  1002 20899    39521      198      30       72             0 top
[1947333.164166] [20929]     0 20929    36379      116      74      215             0 sshd
[1947333.164168] [20931]  1002 20931    36417      118      71      213             0 sshd
[1947333.164169] [20932]  1002 20932    28874       34      14       81             0 bash
[1947333.164171] [20972]  1002 20972    37348      229      27      468             0 vim
[1947333.164172] [20996]    89 20996    22270       83      44      164             0 pickup
[1947333.164174] [21075]  1002 21075 67384359 31985935   71321  4435535             0 python3
[1947333.164176] Out of memory: Kill process 21075 (python3) score 974 or sacrifice child
[1947333.164190] Killed process 21075 (python3) total-vm:269537436kB, anon-rss:127943740kB, file-rss:0kB, shmem-rss:0kB