Python错误:can';使用多处理和Tensorflow模型时pickle SwigPyObject对象
我无法对Tensorflow模型使用多处理。SciKit学习模型工作正常 我创建了一个滑动窗口对象检测器。该检测器从右到左、从上到下扫描图像,并提取子窗口。评估每个子窗口,以查看其是否包含学习对象 从子窗口中提取特征。我正在修改使用HOG特征提取器的代码,并将其替换为一个经过训练的带有顶部截断的VGG16模型。使用支持向量机进行预测。滑动窗口法速度较慢,但如果同时进行预测,则可以大大加快速度。我能够用一个SciKit学习模型(SVM)做到这一点,使用多线程没有问题。但是,当我添加Tensorflow模型时,我收到了错误消息:Python错误:can';使用多处理和Tensorflow模型时pickle SwigPyObject对象,python,multithreading,scikit-learn,python-multithreading,tensor,Python,Multithreading,Scikit Learn,Python Multithreading,Tensor,我无法对Tensorflow模型使用多处理。SciKit学习模型工作正常 我创建了一个滑动窗口对象检测器。该检测器从右到左、从上到下扫描图像,并提取子窗口。评估每个子窗口,以查看其是否包含学习对象 从子窗口中提取特征。我正在修改使用HOG特征提取器的代码,并将其替换为一个经过训练的带有顶部截断的VGG16模型。使用支持向量机进行预测。滑动窗口法速度较慢,但如果同时进行预测,则可以大大加快速度。我能够用一个SciKit学习模型(SVM)做到这一点,使用多线程没有问题。但是,当我添加Tensorfl
TypeError:无法pickle SwigPyObject对象
我创建了一个将模型作为参数的类。下面给出了一个简单的例子。当TF_model=[]
时,它工作
下面是这个问题的一个小例子
from multiprocessing import Process,Queue,Pool,Manager
class ObjectDetectorMinimal:
def __init__(self, SKLearnModel, TF_model):
# Note: in this minimal example, these models are not used
self.SKLmodel = SKLearnModel
self.TFmodel = TF_model
def f1(self): # Use multiple threads
# Uses multithreading instead of multiprocessing.
p = Pool()
print("Starting the Parallel Processing across multiple threads")
output1=[]
output2=[]
myArgs=[]
xx = range(0,5)
yy = range(20,25)
for i in range(0,len(xx)-1):
myArgs.append([xx[i],yy[i]])
print("MyArgs: {}".format(myArgs))
print("Starting p.map()")
pp=list(p.map(self.f2,myArgs)) # p.mat does not require a Queue to be defined.
for ppp in pp:
output1 = output1 + ppp[0]
output2 = output2 + ppp[1]
p.close()
p.join()
print("Finished f1()")
return(output1,output2)
def f2(self,myArgs):
output1=[]
output2 =[]
xx=myArgs[0]
yy=myArgs[1]
#print("xx: {} yy {} ".format(xx,yy))
b,p = self.f3(xx,yy)
if len(b)>0: # Check for empty
#print("b: {} p {}".format(b,p))
output1 = output1 + b
output2 = output2 + p
return(output1,output2)
else:
print("b is empty")
def f3(self,x,y):
self.processID = os.getpid()
output1=[]
output2=[]
output1.append([x ,2*x, 3*x])
output2.append([y, 2*y, 3*y])
return(output1,output2)
SKLmodel = svm.SVC(gamma='scale')
TF_model=VGG16(weights="imagenet", include_top=False)
od = ObjectDetectorMinimal(SKLmodel,TF_model)
#od = ObjectDetectorMinimal(model,[])
output1,output2=od.f1()
print("\nOutput1: {} \n \nOutput2: {}".format(output1,output2))
od=ObjectDetectorMinimal(model,[])
的输出为:
Starting the Parallel Processing across multiple threads
MyArgs: [[0, 20], [1, 21], [2, 22], [3, 23]]
Starting p.map()
Finished f1()
Output1: [[0, 0, 0], [1, 2, 3], [2, 4, 6], [3, 6, 9]]
Output2: [[20, 40, 60], [21, 42, 63], [22, 44, 66], [23, 46, 69]]`
如果我包括Tensorflow模型,我得到:
Starting the Parallel Processing across multiple threads
MyArgs: [[0, 20], [1, 21], [2, 22], [3, 23]]
Starting p.map()
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-110-a2b46430a3c6> in <module>
3 od = ObjectDetectorMinimal(model,featureextraction_model)
4 #od = ObjectDetectorMinimal(model,[])
----> 5 output1,output2=od.f1()
6 print("\nOutput1: {} \n \nOutput2: {}".format(output1,output2))
<ipython-input-106-d7922b5e2ae3> in f1(self)
18
19 print("Starting p.map()")
---> 20 pp=list(p.map(self.f2,myArgs)) # p.mat does not require a Queue to be defined.
21 for ppp in pp:
22 output1 = output1 + ppp[0]
/usr/lib/python3.5/multiprocessing/pool.py in map(self, func, iterable, chunksize)
258 in a list that is returned.
259 '''
--> 260 return self._map_async(func, iterable, mapstar, chunksize).get()
261
262 def starmap(self, func, iterable, chunksize=None):
/usr/lib/python3.5/multiprocessing/pool.py in get(self, timeout)
606 return self._value
607 else:
--> 608 raise self._value
609
610 def _set(self, i, obj):
/usr/lib/python3.5/multiprocessing/pool.py in _handle_tasks(taskqueue, put, outqueue, pool, cache)
383 break
384 try:
--> 385 put(task)
386 except Exception as e:
387 job, ind = task[:2]
/usr/lib/python3.5/multiprocessing/connection.py in send(self, obj)
204 self._check_closed()
205 self._check_writable()
--> 206 self._send_bytes(ForkingPickler.dumps(obj))
207
208 def recv_bytes(self, maxlength=None):
/usr/lib/python3.5/multiprocessing/reduction.py in dumps(cls, obj, protocol)
48 def dumps(cls, obj, protocol=None):
49 buf = io.BytesIO()
---> 50 cls(buf, protocol).dump(obj)
51 return buf.getbuffer()
52
`TypeError: can't pickle SwigPyObject objects`
跨多个线程启动并行处理
MyArgs:[[0,20],[1,21],[2,22],[3,23]]
启动p.map()
---------------------------------------------------------------------------
TypeError回溯(最近一次调用上次)
在里面
3 od=ObjectDetectorMinimal(模型、特征提取\模型)
4#od=ObjectDetectorMinimal(型号,[])
---->5输出1,输出2=od.f1()
6打印(“\nOutput1:{}\n\nOutput2:{}”。格式(output1,output2))
在f1中(自我)
18
19打印(“起始p.map()”)
--->20 pp=list(p.map(self.f2,myArgs))#p.mat不需要定义队列。
21对于pp中的ppp:
22输出1=输出1+ppp[0]
/映射中的usr/lib/python3.5/multiprocessing/pool.py(self、func、iterable、chunksize)
258在返回的列表中。
259 '''
-->260返回self.\u map\u async(func、iterable、mapstar、chunksize).get()
261
262 def星图(self、func、iterable、chunksize=None):
/get中的usr/lib/python3.5/multiprocessing/pool.py(self,timeout)
606返回自身值
607其他:
-->608提高自我价值
609
610 def_装置(自身、i、obj):
/usr/lib/python3.5/multiprocessing/pool.py in\u handle\u任务(taskqueue、put、outqueue、pool、cache)
383休息
384试试:
-->385 put(任务)
386例外情况除外,如e:
387作业,ind=任务[:2]
/发送中的usr/lib/python3.5/multiprocessing/connection.py(self,obj)
204自我检查关闭()
205自我检查可写()
-->206自发送字节(ForkingPickler.dumps(obj))
207
208 def recv_字节(self,maxlength=None):
/转储中的usr/lib/python3.5/multiprocessing/reduce.py(cls、obj、协议)
48个def转储(cls、obj、协议=无):
49 buf=io.BytesIO()
--->50 cls(buf,协议)。转储(obj)
51返回buf.getbuffer()
52
`TypeError:无法pickle SwigPyObject对象`
我希望使用Tensorflow模型的方式与使用多处理模块时使用SciKit学习模型的方式相同