Python多处理过程是';即使任务完成了,也不要被杀
我已经编写了一个python脚本,它将从AmazonSQS中读取数据,并根据用户需要创建任意多个并行进程。它继承了Django BaseCommand,下面是代码Python多处理过程是';即使任务完成了,也不要被杀,python,django,multithreading,python-multiprocessing,boilerpipe,Python,Django,Multithreading,Python Multiprocessing,Boilerpipe,我已经编写了一个python脚本,它将从AmazonSQS中读取数据,并根据用户需要创建任意多个并行进程。它继承了Django BaseCommand,下面是代码 def handle(self, *args, **kwargs): self.set_up(*args, **kwargs) process_queue = JoinableQueue(self.threads) process_pool = Pool( self.threads,
def handle(self, *args, **kwargs):
self.set_up(*args, **kwargs)
process_queue = JoinableQueue(self.threads)
process_pool = Pool(
self.threads,
self.worker_process,
(process_queue,)
)
is_queue_empty = False
while not is_queue_empty:
message = self.get_next_message()
if len(message) == 0:
is_queue_empty = True
else:
process_queue.put(message[0])
process_queue.join()
raise CommandError('Number retries exceeded retry limit')
def worker_process(self, process_queue):
while True:
message = process_queue.get(True)
message_tuple = (message)
self.process_message(message_tuple)
process_queue.task_done()
这工作正常,一旦任务完成,所有进程都将停止。但不是针对某个特定的活动,我用它来提取一些数据
from boilerpipe.extract import Extractor
extractor = Extractor(extractor='DefaultExtractor', html=soup_html)
extractor.getText()
当我查看boilepipe代码时,我可以看到,在提取器的构造函数中有这样的代码
lock = threading.Lock()
class Extractor():
def __init__():
# code
try:
# code
lock.acquire()
# code
finally:
lock.release()
完整代码是
请提前告知,谢谢。线程?需要仔细地
中断
,否则,如果一个线程包含多个线程,则会出现更多错误!如果等待某件事,锁等不是个好主意。这是他们是如何做到的,不确定他们想要实现什么。基本上,python boilerpipe只是用Java编写的实际库的包装器。