Python Scipy-LAPACK函数是并行的吗?
我目前使用的是Python Scipy-LAPACK函数是并行的吗?,python,parallel-processing,scipy,lapack,Python,Parallel Processing,Scipy,Lapack,我目前使用的是scipy.linalg.lapack.zheevd()函数,它在所有内核上运行,如果我尝试使用ProcessPoolExecutor()或ThreadPoolExecutor()从concurrent.futures将函数映射到参数数组,就会产生挂起和内存溢出 它使用了和我的测试系统一样多的内核,但我的印象是,由于GIL,Python中的事情通常不会并行化。这是使用OpenMP运行底层Fortran代码的结果吗 可以安全地假设这是并行的,并且不能进一步并行吗?这对于我的代码来说并
scipy.linalg.lapack.zheevd()
函数,它在所有内核上运行,如果我尝试使用ProcessPoolExecutor()
或ThreadPoolExecutor()
从concurrent.futures
将函数映射到参数数组,就会产生挂起和内存溢出
它使用了和我的测试系统一样多的内核,但我的印象是,由于GIL,Python中的事情通常不会并行化。这是使用OpenMP运行底层Fortran代码的结果吗
可以安全地假设这是并行的,并且不能进一步并行吗?这对于我的代码来说并不是一个很大的瓶颈(找到400个唯一的1000x1000矩阵的本征系统;虽然可能需要放大,例如最终1000个2000x2000矩阵),但我正处于优化阶段
下面是一段有助于概念化的代码片段,但不代表实际的矩阵:
import numpy as np
from scipy import linalg as la
import concurrent.futures
# In real code
# various parameters are used to build the matrix function,
# it is presumably not sparse
# Matrix with independent variable x
def matrix_function(x):
# Define dimensions and pre-allocate space for matrix
#dim = 100 # For quicker evaluation/testing
dim = 1000 # For conveying the scale of the problem
matrix_dimensions = [dim, dim]
# The matrix is complex
mat = np.zeros(matrix_dimensions, dtype=complex)
for i in range(dim):
for j in range(i,dim):
mat[i,j] = x*np.random.rand(1) + np.random.rand(1)*1J
# Making the matrix Hermitian
mat[j,i] = np.conjugate( mat[i,j] )
return mat
# 400 Arguments for the defined matrix function
args = np.arange(0,10,0.025)
# Parallelizing evaluation of 400 matrices
with concurrent.futures.ProcessPoolExecutor() as pool:
evaluated_matrix_functions = pool.map( matrix_function, args )
''' This will hang,
which is what tipped me off to the issue
**not important to question
eigsystem = pool.map( la.lapack.zheevd,
evaluated_matrix_functions
)
'''
pool.shutdown()
''' This will cause a memory overflow,
depending on the size of the matrices
and how many of them; even with 32GB memory
with concurrent.futures.ThreadPoolExecutor() as pool:
eigsystem = pool.map( la.lapack.zheevd,
evaluated_matrix_functions
)
pool.shutdown()
'''
# The code which I run, in serial,
# but still uses all cores/threads my 2700x provides at full load
eigensystem_list = []
for matrix in evaluated_matrix_functions:
eigensystem_list.append( la.lapack.zheevd(matrix) )
# The eigensystem_list is then used in later calculations
这一切都由您在发动机罩下使用的LAPACK库控制。这一切都由您在发动机罩下使用的LAPACK库控制