Python:如何让挂起的进程在concurrent.futures池中释放控制?
我使用Python:如何让挂起的进程在concurrent.futures池中释放控制?,python,concurrency,multiprocessing,python-multiprocessing,concurrent.futures,Python,Concurrency,Multiprocessing,Python Multiprocessing,Concurrent.futures,我使用 concurrent.futuresProcessPoolExecutor方法,如下所示 with concurrent.futures.ProcessPoolExecutor(max_workers=workers) as executor: futures_ocr = ([ executor.submit( MyProcessor, folder ) for
concurrent.futuresProcessPoolExecutor
方法,如下所示
with concurrent.futures.ProcessPoolExecutor(max_workers=workers) as executor:
futures_ocr = ([
executor.submit(
MyProcessor,
folder
) for folder in sub_folders
])
is_cancel = wait_for(futures_ocr)
if is_cancel:
print 'shutting down executor'
executor.shutdown()
def wait_for(futures):
"""Handes the future tasks after completion"""
cancelled = False
try:
for future in concurrent.futures.as_completed(futures, timeout=200):
try:
result = future.result()
print 'successfully finished processing folder: ', result.source_folder_path
except concurrent.futures.TimeoutError:
print 'TimeoutError occured'
except TypeError:
print 'TypeError occured'
except KeyboardInterrupt:
print '****** cancelling... *******'
cancelled = True
for future in futures:
future.cancel()
return cancelled
在某些文件夹中,进程似乎被阻塞了很长时间,这不是因为代码中的某些错误,而是由于正在处理的文件的性质。所以,我想让这些类型的进程超时,以便在超过某个时间限制时返回。然后,池可以将该进程用于下一个可用任务
在as_completed()
函数中添加超时会在完成时出错
Traceback (most recent call last):
File "call_ocr.py", line 96, in <module>
main()
File "call_ocr.py", line 42, in main
is_cancel = wait_for(futures_ocr)
File "call_ocr.py", line 59, in wait_for
for future in concurrent.futures.as_completed(futures, timeout=200):
File "/Users/saurav/.pyenv/versions/ocr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 216, in as_completed
len(pending), len(fs)))
concurrent.futures._base.TimeoutError: 3 (of 3) futures unfinished
回溯(最近一次呼叫最后一次):
文件“call_ocr.py”,第96行,在
main()
文件“call_ocr.py”,第42行,主目录
是否取消=等待(期货ocr)
文件“call_ocr.py”,第59行,等待
对于concurrent.futures.as_中的future(期货,超时=200):
文件“/Users/saurav/.pyenv/versions/ocr/lib/python2.7/site-packages/concurrent/futures/_-base.py”,第216行,填写完成
len(待定),len(fs)))
concurrent.futures.\u base.timeouter错误:3个(共3个)futures未完成
我在这里做错了什么?什么是使timedout进程停止并将进程放弃回进程池的最佳方法?并发的
。futures
实现不支持这种用例
可以传递给函数和方法的超时允许设置等待结果的时间,但对实际计算本身没有影响
库支持这样的用例
from concurrent.futures import TimeoutError
from pebble import ProcessPool
def function(n):
return n
with ProcessPool() as pool:
future = pool.schedule(function, args=[1], timeout=10)
try:
results = future.result()
except TimeoutError as error:
print("function took longer than %d seconds" % error.args[1])