Windows 10和Linux Ubuntu 16.04在运行时间上的巨大差异,用于ExtraTreesClassifier训练和预测(Python)
我使用Windows 10和Linux Ubuntu 16.04在运行时间上的巨大差异,用于ExtraTreesClassifier训练和预测(Python),python,numpy,time,scikit-learn,Python,Numpy,Time,Scikit Learn,我使用extratreeClassifier进行训练和预测。我在Windows10和LinuxUbuntu16.04的相同数据集上执行相同的源代码,令人惊讶的是,我在执行时间上得到了巨大的差异 结果是: +---------------+-----------+----------+----------+---------+ | Dataset in Mo | Win Train | Win Pred | Ub Train | Ub Pred | +---------------+-------
extratreeClassifier
进行训练和预测。我在Windows10和LinuxUbuntu16.04的相同数据集上执行相同的源代码,令人惊讶的是,我在执行时间上得到了巨大的差异
结果是:
+---------------+-----------+----------+----------+---------+
| Dataset in Mo | Win Train | Win Pred | Ub Train | Ub Pred |
+---------------+-----------+----------+----------+---------+
| 430 | 104 | 11 | 2420 | 2019 |
+---------------+-----------+----------+----------+---------+
| 530 | 122 | 14 | 2948 | 2162 |
+---------------+-----------+----------+----------+---------+
| 699 | 140 | 18 | 3672 | 2500 |
+---------------+-----------+----------+----------+---------+
所使用的操作系统信息和库版本如下表所示。我更新了所有使用过的库
+---------------------------------------+-------------------------------------------+
| Windows 10 | Ubuntu 16.04 |
| Intel i7-8550U CPU @ 1.80Ghz 1.99Ghz | Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz |
| cpu cores : 4 | cpu cores : 1 |
| 64 bit OS | 64 bit OS |
| RAM 16 Go | RAM 1007 Go |
+---------------------------------------+-------------------------------------------+
| Python 3.7.7 | Python 3.5.2 |
| ----------------- | ------------- |
| biopython==1.77 | biopython==1.73 |
| datatable==0.11.0a0+pr2536.12 | datatable==0.10.1 |
| numpy==1.19.0 | numpy==1.18.5 |
| pandas==1.0.5 | pandas==0.24.2 |
| pyahocorasick==1.4.0 | pyahocorasick==1.4.0 |
| scikit-learn==0.23.1 | scikit-learn==0.22.2.post1 |
| scipy==1.5.0 | scipy==1.4.1 |
| suffix-trees==0.3.0 | suffix-trees==0.3.0 |
+---------------------------------------+-------------------------------------------+
+---------------------------------------+-------------------------------------------+
|Windows 10 | Ubuntu 16.04|
|英特尔i7-8550U CPU@1.80Ghz 1.99Ghz |英特尔(R)至强(R)CPU E5-2660 v4@2.00GHz|
|cpu核数:4 | cpu核数:1|
|64位操作系统| 64位操作系统|
|RAM 16通电| RAM 1007通电|
+---------------------------------------+-------------------------------------------+
|Python 3.7.7 | Python 3.5.2|
| ----------------- | ------------- |
|生物皮顿==1.77 |生物皮顿==1.73|
|数据表==0.11.0a0+pr2536.12 |数据表==0.10.1|
|numpy==1.19.0 | numpy==1.18.5|
|熊猫==1.0.5 |熊猫==0.24.2|
|pyahocorasick==1.4.0 | pyahocorasick==1.4.0|
|scikit学习==0.23.1 | scikit学习==0.22.2.1|
|scipy==1.5.0 | scipy==1.4.1|
|后缀树==0.3.0 |后缀树==0.3.0|
+---------------------------------------+-------------------------------------------+
使用cprofile:
1619734 function calls (1589052 primitive calls) in 6495.451 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
4828 6248.349 1.294 6248.349 1.294 {built-in method numpy.core.multiarray.array}
100 130.458 1.305 130.458 1.305 {method 'build' of 'sklearn.tree._tree.DepthFirstTreeBuilder' objects}
1 48.288 48.288 48.288 48.288 {built-in method datatable.lib._datatable.gread}
2 21.834 10.917 25.749 12.874 Main.py:40(get_matrix_nbOcrrs_listStr_AhoCorasick)
2 20.747 10.374 2570.626 1285.313 model.py:233(test_groupe_score_dt)
4365 6.476 0.001 6.476 0.001 {method 'reduce' of 'numpy.ufunc' objects}
1 5.851 5.851 6492.121 6492.121 Main.py:309(main)
6710 3.705 0.001 3.705 0.001 {method 'copy' of 'list' objects}
400 2.548 0.006 2.548 0.006 {method 'predict' of 'sklearn.tree._tree.Tree' objects}
1 2.288 2.288 6495.453 6495.453 Main.py:1()
1 1.334 1.334 3889.596 3889.596 model.py:189(train_with_dt)
400 0.827 0.002 3.628 0.009 _classes.py:880(predict_proba)
4 0.522 0.131 4936.793 1234.198 _forest.py:591(predict)
400 0.354 0.001 3.982 0.010 _forest.py:442(_accumulate_prediction)
376662 0.150 0.000 0.150 0.000 {method 'add_word' of 'ahocorasick.Automaton' objects}
803 0.120 0.000 0.120 0.000 {built-in method marshal.loads}
2272/2260 0.070 0.000 0.144 0.000 {built-in method builtins.__build_class__}
1081/1 0.069 0.000 6495.453 6495.453 {built-in method builtins.exec}
143/119 0.064 0.000 0.116 0.001 {built-in method _imp.create_dynamic}
2 0.046 0.023 0.046 0.023 {method 'make_automaton' of 'ahocorasick.Automaton' objects}
...etc
6495.451秒内完成1619734次函数调用(1589052次基元调用)
订购人:内部时间
ncalls tottime percall cumtime percall文件名:lineno(函数)
4828 6248.349 1.294 6248.349 1.294{内置方法numpy.core.multiarray.array}
100 130.458 1.305 130.458 1.305{“sklearn.tree.\u tree.DepthFirstTreeBuilder”对象的方法“build”}
148.28848.28848.28848.28848{内置方法datatable.lib.\u datatable.gread}
221.83410.91725.74912.874 Main.py:40(获取矩阵
220.747 10.374 2570.626 1285.313型号。py:233(测试组分数)
4365 6.476 0.001 6.476 0.001{“numpy.ufunc”对象的方法“reduce”}
1 5.851 5.851 6492.121 6492.121 Main.py:309(Main)
6710 3.705 0.001 3.705 0.001{“列表”对象的方法“复制”}
400 2.548 0.006 2.548 0.006{“sklearn.tree.\u tree.tree”对象的“预测”方法}
1 2.288 2.288 6495.453 6495.453 Main.py:1()
1 1.334 1.334 3889.596 3889.596型号。py:189(带dt的列车)
400 0.827 0.002 3.628 0.009(预测概率)
40.5220.131 4936.793 1234.198_森林。py:591(预测)
400 0.354 0.001 3.982 0.010 _forest.py:442(累积预测)
376662 0.150 0.000 0.150 0.000{“ahocarasick.Automaton”对象的“添加单词”方法}
803 0.120 0.000 0.120 0.000{内置方法marshal.loads}
2272/2260.070 0.000 0.144 0.000{内置方法内置。uuu构建uuu类uuuu}
1081/1 0.069 0.000 6495.453 6495.453{内置方法builtins.exec}
143/119 0.064 0.000 0.116 0.001{内置方法_imp.create_dynamic}
2 0.046 0.023 0.046 0.023{方法'make_automaton'的'ahocorasick.automaton'对象}
等
感谢您的帮助。我发现了导致windows和Linux在执行时间上存在巨大差异的问题。
当我对python包进行更新时,系统说,需求已经满足了…,所以我认为包已经更新了。在@0x5453的注释之后,我尝试再次更新包,但没有成功,我在()中发现了一个关于更新问题的问题。
解决方案:我在Python3.5的旁边安装了Python3.8,并创建了一个虚拟环境,在那里安装了所需软件包的最新版本。而且它大大缩短了执行时间。 +---------------+-----------+----------+------------------+----------------+- | Dataset in Mo | Win Train | Win Pred | Ub Train (Py3.5) | Ub Pred (3.5) | +---------------+-----------+----------+------------------+----------------+- | 430 | 104 | 11 | 2420 (win x23) | 2019 (win x153)| +---------------+-----------+----------+------------------+----------------+- | 530 | 122 | 14 | 2948 (win x24) | 2162 (win x154)| +---------------+-----------+----------+------------------+----------------+- | 699 | 140 | 18 | 3672 (win x26) | 2500 (win x204)| +---------------+-----------+----------+------------------+----------------+- -+------------------+-----------------+ | Ub Train (Py3.8) | Ub Pred (Py3.8) | -+------------------+-----------------+ | 353 (win x3) | 270 (win x24) | -+------------------+-----------------+ | 771 (win x6) | 646 (win x46) | -+------------------+-----------------+ | 901 (win x6) | 430 (win x23) | -+------------------+-----------------+ +---------------+-----------+----------+------------------+----------------+- |Mo | Win Train | Win Pred | Ub Train(Py3.5)| Ub Pred(3.5)|中的数据集 +---------------+-----------+----------+------------------+----------------+- |430 | 104 | 11 | 2420(胜利x23)| 2019(胜利x153)| +---------------+-----------+----------+------------------+----------------+- |530 | 122 | 14 | 2948 | 2162 | x154 | +---------------+-----------+----------+------------------+----------------+- |699 | 140 | 18 | 3672(赢x26)| 2500(赢x204)| +---------------+-----------+----------+------------------+----------------+- -+------------------+-----------------+ |Ub列车(Py3.8)| Ub Pred(Py3.8)| -+------------------+-----------------+ |353(赢x3)| 270(赢x24)| -+------------------+-----------------+ |771(win x6)| 646(win x46)| -+------------------+-----------------+ |901(赢x6)| 430(赢x23)| -+------------------+-----------------+
现在,windows和Linux Python 3.8之间的剩余差异可以用所用机器的差异来解释。我发现了导致windows和Linux之间执行时间巨大差异的问题。
当我对python包进行更新时,系统会说Req +---------------+-----------+----------+------------------+----------------+- | Dataset in Mo | Win Train | Win Pred | Ub Train (Py3.5) | Ub Pred (3.5) | +---------------+-----------+----------+------------------+----------------+- | 430 | 104 | 11 | 2420 (win x23) | 2019 (win x153)| +---------------+-----------+----------+------------------+----------------+- | 530 | 122 | 14 | 2948 (win x24) | 2162 (win x154)| +---------------+-----------+----------+------------------+----------------+- | 699 | 140 | 18 | 3672 (win x26) | 2500 (win x204)| +---------------+-----------+----------+------------------+----------------+- -+------------------+-----------------+ | Ub Train (Py3.8) | Ub Pred (Py3.8) | -+------------------+-----------------+ | 353 (win x3) | 270 (win x24) | -+------------------+-----------------+ | 771 (win x6) | 646 (win x46) | -+------------------+-----------------+ | 901 (win x6) | 430 (win x23) | -+------------------+-----------------+