Python Linux和Windows之间的numpy性能差异_Python_Performance_Numpy_Scikit Learn

Python Linux和Windows之间的numpy性能差异

python performance numpy scikit-learn

Python Linux和Windows之间的numpy性能差异,python,performance,numpy,scikit-learn,Python,Performance,Numpy,Scikit Learn,我试图在两台不同的计算机上运行sklearn.decomposition.TruncatedSVD（），并了解性能差异计算机1（Windows 7，物理计算机）计算机2（Debian，亚马逊云上）计算机3（亚马逊云上的Windows 2008R2）这两台计算机都运行Python3.2和相同的sklearn、numpy、scipy版本我运行了cProfile，如下所示： print(vectors.shape) >>> (7500, 2042) _decomp = T

我试图在两台不同的计算机上运行sklearn.decomposition.TruncatedSVD（），并了解性能差异

计算机1（Windows 7，物理计算机）

计算机2（Debian，亚马逊云上）

计算机3（亚马逊云上的Windows 2008R2）

这两台计算机都运行Python3.2和相同的sklearn、numpy、scipy版本

我运行了cProfile，如下所示：

print(vectors.shape)
>>> (7500, 2042)

_decomp = TruncatedSVD(n_components=680, random_state=1)
global _o
_o = _decomp
cProfile.runctx('_o.fit_transform(vectors)', globals(), locals(), sort=1)

计算机1输出

>>>    833 function calls in 1.710 seconds
Ordered by: internal time

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1    0.767    0.767    0.782    0.782 decomp_svd.py:15(svd)
    1    0.249    0.249    0.249    0.249 {method 'enable' of '_lsprof.Profiler' objects}
    1    0.183    0.183    0.183    0.183 {method 'normal' of 'mtrand.RandomState' objects}
    6    0.174    0.029    0.174    0.029 {built-in method csr_matvecs}
    6    0.123    0.021    0.123    0.021 {built-in method csc_matvecs}
    2    0.110    0.055    0.110    0.055 decomp_qr.py:14(safecall)
    1    0.035    0.035    0.035    0.035 {built-in method dot}
    1    0.020    0.020    0.589    0.589 extmath.py:185(randomized_range_finder)
    2    0.018    0.009    0.019    0.010 function_base.py:532(asarray_chkfinite)
   24    0.014    0.001    0.014    0.001 {method 'ravel' of 'numpy.ndarray' objects}
    1    0.007    0.007    0.009    0.009 twodim_base.py:427(triu)
    1    0.004    0.004    1.710    1.710 extmath.py:232(randomized_svd)

>>>    858 function calls in 40.145 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    2   32.116   16.058   32.116   16.058 {built-in method dot}
    1    6.148    6.148    6.156    6.156 decomp_svd.py:15(svd)
    2    0.561    0.281    0.561    0.281 decomp_qr.py:14(safecall)
    6    0.561    0.093    0.561    0.093 {built-in method csr_matvecs}
    1    0.337    0.337    0.337    0.337 {method 'normal' of 'mtrand.RandomState' objects}
    6    0.202    0.034    0.202    0.034 {built-in method csc_matvecs}
    1    0.052    0.052    1.633    1.633 extmath.py:183(randomized_range_finder)
    1    0.045    0.045    0.054    0.054 _methods.py:73(_var)
    1    0.023    0.023    0.023    0.023 {method 'argmax' of 'numpy.ndarray' objects}
    1    0.023    0.023    0.046    0.046 extmath.py:531(svd_flip)
    1    0.016    0.016   40.145   40.145 <string>:1(<module>)
   24    0.011    0.000    0.011    0.000 {method 'ravel' of 'numpy.ndarray' objects}
    6    0.009    0.002    0.009    0.002 {method 'reduce' of 'numpy.ufunc' objects}
    2    0.008    0.004    0.009    0.004 function_base.py:532(asarray_chkfinite)

>>>         858 function calls in 2.223 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1    0.956    0.956    0.972    0.972 decomp_svd.py:15(svd)
    2    0.306    0.153    0.306    0.153 {built-in method dot}
    1    0.274    0.274    0.274    0.274 {method 'normal' of 'mtrand.RandomState' objects}
    6    0.205    0.034    0.205    0.034 {built-in method csr_matvecs}
    6    0.151    0.025    0.151    0.025 {built-in method csc_matvecs}
    2    0.133    0.067    0.133    0.067 decomp_qr.py:14(safecall)
    1    0.032    0.032    0.043    0.043 _methods.py:73(_var)
    1    0.030    0.030    0.030    0.030 {method 'argmax' of 'numpy.ndarray' objects}
   24    0.026    0.001    0.026    0.001 {method 'ravel' of 'numpy.ndarray' objects}
    2    0.019    0.010    0.020    0.010 function_base.py:532(asarray_chkfinite)
    1    0.019    0.019    0.773    0.773 extmath.py:183(randomized_range_finder)
    1    0.019    0.019    0.049    0.049 extmath.py:531(svd_flip)

计算机2输出

>>>    833 function calls in 1.710 seconds
Ordered by: internal time

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1    0.767    0.767    0.782    0.782 decomp_svd.py:15(svd)
    1    0.249    0.249    0.249    0.249 {method 'enable' of '_lsprof.Profiler' objects}
    1    0.183    0.183    0.183    0.183 {method 'normal' of 'mtrand.RandomState' objects}
    6    0.174    0.029    0.174    0.029 {built-in method csr_matvecs}
    6    0.123    0.021    0.123    0.021 {built-in method csc_matvecs}
    2    0.110    0.055    0.110    0.055 decomp_qr.py:14(safecall)
    1    0.035    0.035    0.035    0.035 {built-in method dot}
    1    0.020    0.020    0.589    0.589 extmath.py:185(randomized_range_finder)
    2    0.018    0.009    0.019    0.010 function_base.py:532(asarray_chkfinite)
   24    0.014    0.001    0.014    0.001 {method 'ravel' of 'numpy.ndarray' objects}
    1    0.007    0.007    0.009    0.009 twodim_base.py:427(triu)
    1    0.004    0.004    1.710    1.710 extmath.py:232(randomized_svd)

>>>    858 function calls in 40.145 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    2   32.116   16.058   32.116   16.058 {built-in method dot}
    1    6.148    6.148    6.156    6.156 decomp_svd.py:15(svd)
    2    0.561    0.281    0.561    0.281 decomp_qr.py:14(safecall)
    6    0.561    0.093    0.561    0.093 {built-in method csr_matvecs}
    1    0.337    0.337    0.337    0.337 {method 'normal' of 'mtrand.RandomState' objects}
    6    0.202    0.034    0.202    0.034 {built-in method csc_matvecs}
    1    0.052    0.052    1.633    1.633 extmath.py:183(randomized_range_finder)
    1    0.045    0.045    0.054    0.054 _methods.py:73(_var)
    1    0.023    0.023    0.023    0.023 {method 'argmax' of 'numpy.ndarray' objects}
    1    0.023    0.023    0.046    0.046 extmath.py:531(svd_flip)
    1    0.016    0.016   40.145   40.145 <string>:1(<module>)
   24    0.011    0.000    0.011    0.000 {method 'ravel' of 'numpy.ndarray' objects}
    6    0.009    0.002    0.009    0.002 {method 'reduce' of 'numpy.ufunc' objects}
    2    0.008    0.004    0.009    0.004 function_base.py:532(asarray_chkfinite)

>>>         858 function calls in 2.223 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1    0.956    0.956    0.972    0.972 decomp_svd.py:15(svd)
    2    0.306    0.153    0.306    0.153 {built-in method dot}
    1    0.274    0.274    0.274    0.274 {method 'normal' of 'mtrand.RandomState' objects}
    6    0.205    0.034    0.205    0.034 {built-in method csr_matvecs}
    6    0.151    0.025    0.151    0.025 {built-in method csc_matvecs}
    2    0.133    0.067    0.133    0.067 decomp_qr.py:14(safecall)
    1    0.032    0.032    0.043    0.043 _methods.py:73(_var)
    1    0.030    0.030    0.030    0.030 {method 'argmax' of 'numpy.ndarray' objects}
   24    0.026    0.001    0.026    0.001 {method 'ravel' of 'numpy.ndarray' objects}
    2    0.019    0.010    0.020    0.010 function_base.py:532(asarray_chkfinite)
    1    0.019    0.019    0.773    0.773 extmath.py:183(randomized_range_finder)
    1    0.019    0.019    0.049    0.049 extmath.py:531(svd_flip)

请注意{build-in-method dot}从0.035s/call到16.058s/call的差异，慢了450倍

------+---------+---------+---------+---------+--------------------------------------- ncalls| tottime | percall | cumtime | percall | filename:lineno(function) HARDWARE ------+---------+---------+---------+---------+--------------------------------------- 1 | 0.035 | 0.035 | 0.035 | 0.035 | {built-in method dot} Computer 1 2 | 32.116 | 16.058 | 32.116 | 16.058 | {built-in method dot} Computer 2 2 | 0.306 | 0.153 | 0.306 | 0.153 | {built-in method dot} Computer 3
我知道应该有性能差异，但我应该有那么高吗
有没有办法进一步调试这个性能问题
编辑
我测试了一台新电脑，电脑3，它的硬件与电脑2相似，并且有不同的操作系统
结果是0.153s/call，{内置方法dot}比Linux快100倍

------+---------+---------+---------+---------+--------------------------------------- ncalls| tottime | percall | cumtime | percall | filename:lineno(function) HARDWARE ------+---------+---------+---------+---------+--------------------------------------- 1 | 0.035 | 0.035 | 0.035 | 0.035 | {built-in method dot} Computer 1 2 | 32.116 | 16.058 | 32.116 | 16.058 | {built-in method dot} Computer 2 2 | 0.306 | 0.153 | 0.306 | 0.153 | {built-in method dot} Computer 3
编辑2
计算机1 numpy配置

>>> np.__config__.show() lapack_opt_info: libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'libiomp5md', 'libifportmd', 'mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'libiomp5md', 'libifportmd'] library_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/lib/intel64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/include'] blas_opt_info: libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'libiomp5md', 'libifportmd'] library_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/lib/intel64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/include'] openblas_info: NOT AVAILABLE lapack_mkl_info: libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'libiomp5md', 'libifportmd', 'mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'libiomp5md', 'libifportmd'] library_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/lib/intel64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/include'] blas_mkl_info: libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'libiomp5md', 'libifportmd'] library_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/lib/intel64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/include'] mkl_info: libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'libiomp5md', 'libifportmd'] library_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/lib/intel64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/include']

>>> np.__config__.show() lapack_info: NOT AVAILABLE lapack_opt_info: NOT AVAILABLE blas_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 atlas_threads_info: NOT AVAILABLE atlas_blas_info: NOT AVAILABLE lapack_src_info: NOT AVAILABLE openblas_info: NOT AVAILABLE atlas_blas_threads_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE blas_opt_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] atlas_info: NOT AVAILABLE lapack_mkl_info: NOT AVAILABLE mkl_info: NOT AVAILABLE
计算机2 numpy配置

>>> np.__config__.show() lapack_opt_info: libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'libiomp5md', 'libifportmd', 'mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'libiomp5md', 'libifportmd'] library_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/lib/intel64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/include'] blas_opt_info: libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'libiomp5md', 'libifportmd'] library_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/lib/intel64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/include'] openblas_info: NOT AVAILABLE lapack_mkl_info: libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'libiomp5md', 'libifportmd', 'mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'libiomp5md', 'libifportmd'] library_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/lib/intel64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/include'] blas_mkl_info: libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'libiomp5md', 'libifportmd'] library_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/lib/intel64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/include'] mkl_info: libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'libiomp5md', 'libifportmd'] library_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/lib/intel64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:/Program Files (x86)/Intel/Composer XE/mkl/include']

>>> np.__config__.show() lapack_info: NOT AVAILABLE lapack_opt_info: NOT AVAILABLE blas_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 atlas_threads_info: NOT AVAILABLE atlas_blas_info: NOT AVAILABLE lapack_src_info: NOT AVAILABLE openblas_info: NOT AVAILABLE atlas_blas_threads_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE blas_opt_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] atlas_info: NOT AVAILABLE lapack_mkl_info: NOT AVAILABLE mkl_info: NOT AVAILABLE
注意{build-in-method dot}的差异，从0.035s/call到16.058s/call，慢了450倍

时钟速度和缓存命中率是两个需要考虑的因素。Xeon E5-2670的缓存比核心i7-3770多得多。i7-3770在turbo模式下具有更高的峰值时钟速度。虽然您的Xeon在硬件中有一个很大的缓存，但在EC2上，您可以有效地与其他客户共享该缓存
有没有办法进一步调试这个性能问题
嗯，您有不同的测量（输出）和输入（操作系统和硬件）上的多个差异。考虑到不同的输入，这些不同的输出可能是预期的
CPU性能计数器将更好地隔离算法性能对不同系统的影响。Xeon具有更丰富的性能计数器，但它们都应该具有
CPU\u CLK\u UNHALTED
和
LLC\u MISSES
。它们通过将指令指针映射到事件（如正在执行的代码或缓存未命中）来工作。因此，您可以看到代码的哪些部分是CPU和缓存绑定的。由于目标之间的时钟速度和缓存大小不同，您可能会发现一个是缓存绑定的，另一个是CPU绑定的
Linux有一个名为
perf
（有时
perf\u事件
）的工具。另见

在Linux和Windows上，您也可以使用“英特尔VTune”。
{build-in-method dot}
是
np.dot
函数，它是用于矩阵、矩阵向量和向量向量乘法的CBLAS例程的NumPy包装器。您的Windows计算机使用经过大量调优的CBLAS版本。Linux机器正在使用缓慢的旧参考实现
如果您安装或（都可以通过Linux软件包管理器获得），或者实际上安装Intel MKL，您可能会看到巨大的加速。尝试
sudo-apt-get-install-libatlas-dev
，再次检查NumPy配置以查看是否拾取了ATLAS，然后再次测量
一旦决定了正确的CBLAS库，您可能需要重新编译scikit learn。大多数算法只是使用NumPy来满足其线性代数的需要，但有些算法（特别是k-means）直接使用CBLAS

操作系统与此无关。
我编辑了我的问题，添加了计算机3以消除不同的硬件问题。请还包括每个节点的
python--version
输出。同时比较
python-c'作为np导入numpy；np.\uuuu config\uuuu.show（）'
。您是如何安装numpy的？它使用哪种BLAS？如果windows框使用MKL，Linux框使用默认BLAS，这可能会解释很多。（http://www.AndreasMueller.AndreasMueller，我根据你的评论编辑了我的问题。看来你是对的。有没有关于如何使用MKL编译numpy的手册？如果您没有在Linux上访问MKL（它不是免费的），您可以使用OpenBLAS并获得相同的性能。2020年：现在，从pip安装的numpy是针对OpenBLAS编译的，并且具有与Intel MKL几乎相同的性能，除非CPU支持OpenBLAS尚未实现的AVX-512指令。来自anaconda的Numpy使用Intel MKL。了解Intel MKL在基于AMD Zen的CPU上运行速度非常慢，而没有覆盖所选代码路径的技巧，这一点可能很重要（MKL_DEBUG_CPU_TYPE=5）