Python错误:can';使用多处理和Tensorflow模型时pickle SwigPyObject对象

Python错误:can';使用多处理和Tensorflow模型时pickle SwigPyObject对象,python,multithreading,scikit-learn,python-multithreading,tensor,Python,Multithreading,Scikit Learn,Python Multithreading,Tensor,我无法对Tensorflow模型使用多处理。SciKit学习模型工作正常 我创建了一个滑动窗口对象检测器。该检测器从右到左、从上到下扫描图像,并提取子窗口。评估每个子窗口,以查看其是否包含学习对象 从子窗口中提取特征。我正在修改使用HOG特征提取器的代码,并将其替换为一个经过训练的带有顶部截断的VGG16模型。使用支持向量机进行预测。滑动窗口法速度较慢,但如果同时进行预测,则可以大大加快速度。我能够用一个SciKit学习模型(SVM)做到这一点,使用多线程没有问题。但是,当我添加Tensorfl

我无法对Tensorflow模型使用多处理。SciKit学习模型工作正常

我创建了一个滑动窗口对象检测器。该检测器从右到左、从上到下扫描图像,并提取子窗口。评估每个子窗口,以查看其是否包含学习对象

从子窗口中提取特征。我正在修改使用HOG特征提取器的代码,并将其替换为一个经过训练的带有顶部截断的VGG16模型。使用支持向量机进行预测。滑动窗口法速度较慢,但如果同时进行预测,则可以大大加快速度。我能够用一个SciKit学习模型(SVM)做到这一点,使用多线程没有问题。但是,当我添加Tensorflow模型时,我收到了错误消息:

TypeError:无法pickle SwigPyObject对象

我创建了一个将模型作为参数的类。下面给出了一个简单的例子。当
TF_model=[]
时,它工作

下面是这个问题的一个小例子

from multiprocessing import Process,Queue,Pool,Manager   
class ObjectDetectorMinimal:
  def __init__(self, SKLearnModel, TF_model):
    # Note: in this minimal example, these models are not used
    self.SKLmodel = SKLearnModel 
    self.TFmodel  = TF_model

  def f1(self):  # Use multiple threads
    # Uses multithreading instead of multiprocessing.
    p = Pool()
    print("Starting the Parallel Processing across multiple threads")
    output1=[]
    output2=[]
    myArgs=[]
    xx = range(0,5)
    yy = range(20,25)
    for i in range(0,len(xx)-1):
        myArgs.append([xx[i],yy[i]])
    print("MyArgs: {}".format(myArgs))    

    print("Starting p.map()")
    pp=list(p.map(self.f2,myArgs))  # p.mat does not require a Queue to be defined.
    for ppp in pp:
            output1 = output1 + ppp[0]
            output2 = output2 + ppp[1]
    p.close()
    p.join()
    print("Finished f1()")
    return(output1,output2)

  def f2(self,myArgs):
    output1=[]
    output2 =[]
    xx=myArgs[0]
    yy=myArgs[1]

    #print("xx: {} yy {} ".format(xx,yy))
    b,p = self.f3(xx,yy)
    if len(b)>0: # Check for empty
        #print("b: {} p {}".format(b,p))
        output1 = output1 + b
        output2 = output2 + p
        return(output1,output2)
    else:
        print("b is empty")

  def f3(self,x,y):
    self.processID = os.getpid()
    output1=[]
    output2=[]
    output1.append([x ,2*x, 3*x])
    output2.append([y, 2*y, 3*y])
    return(output1,output2)

 SKLmodel = svm.SVC(gamma='scale')
 TF_model=VGG16(weights="imagenet", include_top=False)
 od = ObjectDetectorMinimal(SKLmodel,TF_model)
 #od = ObjectDetectorMinimal(model,[])
 output1,output2=od.f1()
 print("\nOutput1: {} \n \nOutput2: {}".format(output1,output2))
od=ObjectDetectorMinimal(model,[])
的输出为:

Starting the Parallel Processing across multiple threads
MyArgs: [[0, 20], [1, 21], [2, 22], [3, 23]]
Starting p.map()
Finished f1()

Output1: [[0, 0, 0], [1, 2, 3], [2, 4, 6], [3, 6, 9]] 

Output2: [[20, 40, 60], [21, 42, 63], [22, 44, 66], [23, 46, 69]]`
如果我包括Tensorflow模型,我得到:

Starting the Parallel Processing across multiple threads
MyArgs: [[0, 20], [1, 21], [2, 22], [3, 23]]
Starting p.map()

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-110-a2b46430a3c6> in <module>
      3 od = ObjectDetectorMinimal(model,featureextraction_model)
      4 #od = ObjectDetectorMinimal(model,[])
----> 5 output1,output2=od.f1()
      6 print("\nOutput1: {} \n \nOutput2: {}".format(output1,output2))

<ipython-input-106-d7922b5e2ae3> in f1(self)
     18 
     19         print("Starting p.map()")
---> 20         pp=list(p.map(self.f2,myArgs))  # p.mat does not require a Queue to be defined.
     21         for ppp in pp:
     22                 output1 = output1 + ppp[0]

/usr/lib/python3.5/multiprocessing/pool.py in map(self, func, iterable, chunksize)
    258         in a list that is returned.
    259         '''
--> 260         return self._map_async(func, iterable, mapstar, chunksize).get()
    261 
    262     def starmap(self, func, iterable, chunksize=None):

/usr/lib/python3.5/multiprocessing/pool.py in get(self, timeout)
    606             return self._value
    607         else:
--> 608             raise self._value
    609 
    610     def _set(self, i, obj):

/usr/lib/python3.5/multiprocessing/pool.py in _handle_tasks(taskqueue, put, outqueue, pool, cache)
    383                         break
    384                     try:
--> 385                         put(task)
    386                     except Exception as e:
    387                         job, ind = task[:2]

/usr/lib/python3.5/multiprocessing/connection.py in send(self, obj)
    204         self._check_closed()
    205         self._check_writable()
--> 206         self._send_bytes(ForkingPickler.dumps(obj))
    207 
    208     def recv_bytes(self, maxlength=None):

/usr/lib/python3.5/multiprocessing/reduction.py in dumps(cls, obj, protocol)
     48     def dumps(cls, obj, protocol=None):
     49         buf = io.BytesIO()
---> 50         cls(buf, protocol).dump(obj)
     51         return buf.getbuffer()
     52 

`TypeError: can't pickle SwigPyObject objects`
跨多个线程启动并行处理
MyArgs:[[0,20],[1,21],[2,22],[3,23]]
启动p.map()
---------------------------------------------------------------------------
TypeError回溯(最近一次调用上次)
在里面
3 od=ObjectDetectorMinimal(模型、特征提取\模型)
4#od=ObjectDetectorMinimal(型号,[])
---->5输出1,输出2=od.f1()
6打印(“\nOutput1:{}\n\nOutput2:{}”。格式(output1,output2))
在f1中(自我)
18
19打印(“起始p.map()”)
--->20 pp=list(p.map(self.f2,myArgs))#p.mat不需要定义队列。
21对于pp中的ppp:
22输出1=输出1+ppp[0]
/映射中的usr/lib/python3.5/multiprocessing/pool.py(self、func、iterable、chunksize)
258在返回的列表中。
259         '''
-->260返回self.\u map\u async(func、iterable、mapstar、chunksize).get()
261
262 def星图(self、func、iterable、chunksize=None):
/get中的usr/lib/python3.5/multiprocessing/pool.py(self,timeout)
606返回自身值
607其他:
-->608提高自我价值
609
610 def_装置(自身、i、obj):
/usr/lib/python3.5/multiprocessing/pool.py in\u handle\u任务(taskqueue、put、outqueue、pool、cache)
383休息
384试试:
-->385 put(任务)
386例外情况除外,如e:
387作业,ind=任务[:2]
/发送中的usr/lib/python3.5/multiprocessing/connection.py(self,obj)
204自我检查关闭()
205自我检查可写()
-->206自发送字节(ForkingPickler.dumps(obj))
207
208 def recv_字节(self,maxlength=None):
/转储中的usr/lib/python3.5/multiprocessing/reduce.py(cls、obj、协议)
48个def转储(cls、obj、协议=无):
49 buf=io.BytesIO()
--->50 cls(buf,协议)。转储(obj)
51返回buf.getbuffer()
52
`TypeError:无法pickle SwigPyObject对象`
我希望使用Tensorflow模型的方式与使用多处理模块时使用SciKit学习模型的方式相同