制作一个Python装饰器,将任何@classmethod函数转换为并行处理值向量的函数
这是一个基本的Python问题。我正试图用我自己的装饰器来装饰一个制作一个Python装饰器,将任何@classmethod函数转换为并行处理值向量的函数,python,python-3.x,multiprocessing,decorator,vectorization,Python,Python 3.x,Multiprocessing,Decorator,Vectorization,这是一个基本的Python问题。我正试图用我自己的装饰器来装饰一个@classmethod,它将任何函数转换为一个函数,该函数接受原始函数的参数列表作为参数。这里有一个例子 import multiprocessing as mp def my_decorator(method): def method_in_vector_form(obj, L, *args, **kwargs): pool = mp.Pool(2) print('Starting d
@classmethod
,它将任何函数转换为一个函数,该函数接受原始函数的参数列表作为参数。这里有一个例子
import multiprocessing as mp
def my_decorator(method):
def method_in_vector_form(obj, L, *args, **kwargs):
pool = mp.Pool(2)
print('Starting decorator')
results = [pool.apply_async(method, args=(obj, value)) for value in L]
pool.close()
pool.join()
return [r.get() for r in results]
return method_in_vector_form
class Foo(object):
@classmethod
def boo(cls, i):
print('Running from Foo\'s boo' + i)
@classmethod
@my_decorator
def bar(cls, chunk_id):
print('Running from Foo\'s bar')
cls.boo(chunk_id)
return chunk_id * 2
class Child(Foo):
@classmethod
def boo(cls, i):
print('Running from Child\'s version of boo' + i)
if __name__ == '__main__':
print(Foo.bar([1, 2, 3]))
print(Child.bar([1, 2, 3]))
您有一个函数foo()
,如果您想在参数上按顺序运行它,a
、b
和c
。你会的
result_a = foo(a)
result_b = foo(b)
result_c = foo(c)
您只需创建一个在参数列表上工作的函数,L=[a,b,c]
,然后您就可以
results = foo_on_list([a, b, c])
我想做一个decorator,对任何函数foo()
都能做到这一点。更重要的是,我希望使用多处理
库能够异步执行foo(a)
的foo(b)
。我写了一个很好的修饰符,它适用于静态函数,但是我很难推广到必须使用cls
或self
参数的函数。这里有一个例子
import multiprocessing as mp
def my_decorator(method):
def method_in_vector_form(obj, L, *args, **kwargs):
pool = mp.Pool(2)
print('Starting decorator')
results = [pool.apply_async(method, args=(obj, value)) for value in L]
pool.close()
pool.join()
return [r.get() for r in results]
return method_in_vector_form
class Foo(object):
@classmethod
def boo(cls, i):
print('Running from Foo\'s boo' + i)
@classmethod
@my_decorator
def bar(cls, chunk_id):
print('Running from Foo\'s bar')
cls.boo(chunk_id)
return chunk_id * 2
class Child(Foo):
@classmethod
def boo(cls, i):
print('Running from Child\'s version of boo' + i)
if __name__ == '__main__':
print(Foo.bar([1, 2, 3]))
print(Child.bar([1, 2, 3]))
我希望得到以下输出(可能由于异步性而无序)
这是电流输出
Starting decorator
Traceback (most recent call last):
File "C:\....\dec_example.py", line 36, in <module>
print(Foo.bar([1, 2, 3]))
File "C:\....\TextFromPdf\dec_example.py", line 10, in method_in_vector_form
return [r.get() for r in results]
File "C:\....\dec_example.py", line 10, in <listcomp>
return [r.get() for r in results]
File "C:\....\pool.py", line 599, in get
raise self._value
File "C:\....\pool.py", line 383, in _handle_tasks
put(task)
File "C:\....\connection.py", line 206, in send
self._send_bytes(ForkingPickler.dumps(obj))
File "C:\....\reduction.py", line 50, in dumps
cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function Foo.bar at 0x0000000002897158>: attribute lookup bar on __main__ failed
启动装饰程序
回溯(最近一次呼叫最后一次):
文件“C:\..\dec\u example.py”,第36行,在
打印(Foo.bar([1,2,3]))
文件“C:\..\TextFromPdf\dec\u example.py”,第10行,方法为矢量形式
返回[r.get(),用于结果中的r]
文件“C:\..\dec\u example.py”,第10行,在
返回[r.get(),用于结果中的r]
get中第599行的文件“C:\..\pool.py”
提升自我价值
文件“C:\..\pool.py”,第383行,在任务处理中
放置(任务)
文件“C:\..\connection.py”,第206行,在send中
self.\u发送字节(ForkingPickler.dumps(obj))
文件“C:\..\reduce.py”,第50行,转储
cls(buf,协议).dump(obj)
_pickle.PicklingError:无法pickle:main上的属性查找栏失败
该问题/答案与编写自定义装饰程序无关。您确定希望从Foo's bar运行的只出现一次吗?它不会出现在每次调用boo
之前吗?这里的问题不在于装饰程序,而在于进程间通信所需的序列化(酸洗)。