Python 顺序编程与并行编程解决方案的差异
我已经创建了一个python代码来解决一个组套索惩罚线性模型。对于那些不习惯使用这些模型的人来说,基本思想是输入一个数据集(x)和一个响应变量(y),以及一个参数的值(lambda1),改变这个参数的值会改变模型的解。因此,我决定使用多处理库并解决不同的模型(与不同的参数值相关)。我创建了一个名为“model.py”的python文件,其中包含以下函数:Python 顺序编程与并行编程解决方案的差异,python,python-2.7,parallel-processing,multiprocessing,Python,Python 2.7,Parallel Processing,Multiprocessing,我已经创建了一个python代码来解决一个组套索惩罚线性模型。对于那些不习惯使用这些模型的人来说,基本思想是输入一个数据集(x)和一个响应变量(y),以及一个参数的值(lambda1),改变这个参数的值会改变模型的解。因此,我决定使用多处理库并解决不同的模型(与不同的参数值相关)。我创建了一个名为“model.py”的python文件,其中包含以下函数: # -*- coding: utf-8 -*- from __future__ import division import functool
# -*- coding: utf-8 -*-
from __future__ import division
import functools
import multiprocessing as mp
import numpy as np
from cvxpy import *
def lm_gl_preprocessing(x, y, index, lambda1=None):
lambda_vector = [lambda1]
m = x.shape[1]
n = x.shape[0]
lambda_param = Parameter(sign="positive")
m = m+1
index = np.append(0, index)
x = np.c_[np.ones(n), x]
group_sizes = []
beta_var = []
unique_index = np.unique(index)
for idx in unique_index:
group_sizes.append(len(np.where(index == idx)[0]))
beta_var.append(Variable(len(np.where(index == idx)[0])))
num_groups = len(group_sizes)
group_lasso_penalization = 0
model_prediction = x[:, np.where(index == unique_index[0])[0]] * beta_var[0]
for i in range(1, num_groups):
model_prediction += x[:, np.where(index == unique_index[i])[0]] * beta_var[i]
group_lasso_penalization += sqrt(group_sizes[i]) * norm(beta_var[i], 2)
lm_penalization = (1.0/n) * sum_squares(y - model_prediction)
objective = Minimize(lm_penalization + (lambda_param * group_lasso_penalization))
problem = Problem(objective)
response = {'problem': problem, 'beta_var': beta_var, 'lambda_param': lambda_param, 'lambda_vector': lambda_vector}
return response
def solver(problem, beta_var, lambda_param, lambda_vector):
beta_sol_list = []
for i in range(len(lambda_vector)):
lambda_param.value = lambda_vector[i]
problem.solve(solver=ECOS)
beta_sol = np.asarray(np.row_stack([b.value for b in beta_var])).flatten()
beta_sol_list.append(beta_sol)
return beta_sol_list
def parallel_solver(problem, beta_var, lambda_param, lambda_vector):
# Divide parameter vector into chunks to be executed in parallel
num_chunks = mp.cpu_count()
chunks = np.array_split(lambda_vector, num_chunks)
# Solve problem in parallel
pool = mp.Pool(num_chunks)
global_results = pool.map(functools.partial(solver, problem, beta_var, lambda_param), chunks)
pool.close()
pool.join()
return global_results
- lm_gl_preprocessing函数基本上定义了使用cvxpy模块求解的模型
- 函数解算器从previus函数中获取模型细节,并解决导致模型最终解的优化问题
- 函数parallel_solver使用多处理并行化解算器函数
from __future__ import division
from sklearn.datasets import load_boston
import numpy as np
import model as t
boston = load_boston()
x = boston.data
y = boston.target
index = np.array([1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5])
lambda1 = 1e-3
r1 = t.lm_gl_preprocessing(x=x, y=y, index=index, lambda1=lambda1)
s_parallel_1 = t.parallel_solver(problem=r1['problem'], beta_var=r1['beta_var'], lambda_param=r1['lambda_param'], lambda_vector=r1['lambda_vector'])
print(s_parallel_1)
[[array([ 4.61648376e+01, -1.22394832e-04, 0.00000000e+00,
0.00000000e+00, 1.37065733e-04, 1.51910696e-03,
0.00000000e+00, 1.51910696e-03, 0.00000000e+00,
7.00079603e-03, 1.52776114e-03, -8.67357376e-01,
7.16429750e-03, -8.67357376e-01])], [], [], []]
s_1 = t.solver(problem=r1['problem'], beta_var=r1['beta_var'], lambda_param=r1['lambda_param'], lambda_vector=r1['lambda_vector'])
print(s_1)
[array([ 3.62813738e+01, -1.06995338e-01, 4.64210526e-02,
1.97112192e-02, 2.68475527e+00, -1.75142155e+01,
3.80741843e+00, 5.14842823e-04, -1.47105323e+00,
3.04949407e-01, -1.23508259e-02, -9.50143293e-01,
9.40708993e-03, -5.25758097e-01])]
#####################################################
r1 = t.lm_gl_preprocessing(x=x, y=y, index=index, lambda1=lambda1)
s_1 = t.solver(problem=r1['problem'], beta_var=r1['beta_var'], lambda_param=r1['lambda_param'], lambda_vector=r1['lambda_vector'])
print(s_1)
[array([ 3.62813738e+01, -1.06995338e-01, 4.64210526e-02,
1.97112192e-02, 2.68475527e+00, -1.75142155e+01,
3.80741843e+00, 5.14842823e-04, -1.47105323e+00,
3.04949407e-01, -1.23508259e-02, -9.50143293e-01,
9.40708993e-03, -5.25758097e-01])]
s_parallel_1 = t.parallel_solver(problem=r1['problem'], beta_var=r1['beta_var'], lambda_param=r1['lambda_param'], lambda_vector=r1['lambda_vector'])
print(s_parallel_1)
[[array([ 3.62813738e+01, -1.06995338e-01, 4.64210526e-02,
1.97112192e-02, 2.68475527e+00, -1.75142155e+01,
3.80741843e+00, 5.14842823e-04, -1.47105323e+00,
3.04949407e-01, -1.23508259e-02, -9.50143293e-01,
9.40708993e-03, -5.25758097e-01])], [], [], []]
PS:我知道在这个例子中,我使用并行编程只是为了解决一个模型和一个可能的参数值,但这只是一个小例子,旨在说明顺序编程和并行编程提供的解决方案的差异。我会感谢任何提示,因为我在这里完全迷路了。如果我执行你的代码,我在所有情况下都会得到相同的结果。这是我正在运行的代码(我合并了两个文件): 和输出:
[[array([ 3.62813738e+01, -1.06995338e-01, 4.64210526e-02, 1.97112192e-02,
2.68475527e+00, -1.75142155e+01, 3.80741843e+00, 5.14842823e-04,
-1.47105323e+00, 3.04949407e-01, -1.23508259e-02, -9.50143293e-01,
9.40708993e-03, -5.25758097e-01])], [], [], []]
[array([ 3.62813738e+01, -1.06995338e-01, 4.64210526e-02, 1.97112192e-02,
2.68475527e+00, -1.75142155e+01, 3.80741843e+00, 5.14842823e-04,
-1.47105323e+00, 3.04949407e-01, -1.23508259e-02, -9.50143293e-01,
9.40708993e-03, -5.25758097e-01])]
#####################################################
[array([ 3.62813738e+01, -1.06995338e-01, 4.64210526e-02, 1.97112192e-02,
2.68475527e+00, -1.75142155e+01, 3.80741843e+00, 5.14842823e-04,
-1.47105323e+00, 3.04949407e-01, -1.23508259e-02, -9.50143293e-01,
9.40708993e-03, -5.25758097e-01])]
[[array([ 3.62813738e+01, -1.06995338e-01, 4.64210526e-02, 1.97112192e-02,
2.68475527e+00, -1.75142155e+01, 3.80741843e+00, 5.14842823e-04,
-1.47105323e+00, 3.04949407e-01, -1.23508259e-02, -9.50143293e-01,
9.40708993e-03, -5.25758097e-01])], [], [], []]
如您所见,我有相同数量的CPU(4)
我的环境是Linux上的Python2.7,以下是相关软件包的版本:
>>> import sklearn
>>> sklearn.__version__
'0.19.2'
>>> import scipy
>>> scipy.__version__
'1.1.0'
>>> import numpy
>>> numpy.__version__
'1.15.2'
>>> import cvxpy
>>> cvxpy.__version__
'0.4.0'
>>> import multiprocessing
>>> multiprocessing.__version__
'0.70a1'
从
parallel_solver
的输出中,我看到除一个进程外,所有进程都返回空列表。所以我猜只有一个进程在实际执行任务。不知道模型代码就很难回答。我的猜测是,当您调用解算器
时,某些参数(例如问题
)会被修改。因此,如果在solver
之后调用parallel\u solver
,则传递修改后的参数,从而得到不同的结果。是@Amedeo,在这种情况下,我只执行一个具有一个参数值的任务,因此并行版本的输出只是一个进程。关于你的猜测,这是一个很好的猜测,但我尝试了执行预处理函数+解算器,然后再次执行预处理+并行解算器(因此,输入,如可能已被解算器更改的问题,对于并行解算器,再次相同),并且我发现了与此处发布的结果相同的结果。请参阅下面我的答案。我复制了你的测试,得到了正确的结果。尝试我的代码并检查结果。如果是不同的,我唯一的猜测是你有一些过时的库,也许更新可以解决这个问题。例如,为了运行您的代码,我必须将cvxpy降级到v0.4。上一个版本(1.0)的“parameter()”有一个不同的参数(nonneg=True
而不是sign=“positive”
)。我正在Linux上运行它。我已经用环境信息更新了我的帖子。我还将在Windows上测试它以进行双重检查。使用Windows+Anaconda+cvxpy0.4,我还看到错误的输出。在Anaconda中,我可以更新到cvxpy1.0,但这需要对代码进行一些更改。
>>> import sklearn
>>> sklearn.__version__
'0.19.2'
>>> import scipy
>>> scipy.__version__
'1.1.0'
>>> import numpy
>>> numpy.__version__
'1.15.2'
>>> import cvxpy
>>> cvxpy.__version__
'0.4.0'
>>> import multiprocessing
>>> multiprocessing.__version__
'0.70a1'