Python 芹菜工人断开与经纪人的连接

我正在使用

Python

和

RabbitMQ

和

芹菜

将任务分发给工作人员。每个任务大约需要15分钟，并且99%的CPU限制。我的系统有24个内核，每当我的工作人员执行此任务时，我就会收到到代理的连接错误

[2019-10-12 08:49:57,695: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
[...]
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

我发现其他几个帖子都有这个问题，但没有一个能解决。特别是在CPU负载很重的情况下，你知道我该如何解决这个问题吗

Windows 10（工作程序）

macOS 10.14（RabbitMQ服务器）

Python3.7

芹菜4.3.0（大黄）

RabbitMQ3.7.16（Erlang22.0.7）

我的配置允许工作进程一次只消耗1个任务，即使在每个作业后重新启动工作进程，仍然没有运气：

CELERYD_MAX_TASKS_PER_CHILD = 1,
CELERYD_CONCURRENCY = 1,
CELERY_TASK_RESULT_EXPIRES=3600,
CELERYD_PREFETCH_MULTIPLIER = 1,
CELERY_MAX_CACHED_RESULTS = 1,
CELERY_ACKS_LATE = True,

这是整个调用堆栈：

[2019-10-12 08:49:57,695: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "C:\Python37\lib\site-packages\celery\worker\consumer\consumer.py", line 318, in start
    blueprint.start(self)
File "C:\Python37\lib\site-packages\celery\bootsteps.py", line 119, in start
    step.start(parent)
File "C:\Python37\lib\site-packages\celery\worker\consumer\consumer.py", line 596, in start
    c.loop(*c.loop_args())
File "C:\Python37\lib\site-packages\celery\worker\loops.py", line 118, in synloop
    qos.update()
File "C:\Python37\lib\site-packages\kombu\common.py", line 442, in update
    return self.set(self.value)
File "C:\Python37\lib\site-packages\kombu\common.py", line 435, in set
    self.callback(prefetch_count=new_value)
File "C:\Python37\lib\site-packages\celery\worker\consumer\tasks.py", line 47, in set_prefetch_count
    apply_global=qos_global,
File "C:\Python37\lib\site-packages\kombu\messaging.py", line 558, in qos
    apply_global)
File "C:\Python37\lib\site-packages\amqp\channel.py", line 1853, in basic_qos
    wait=spec.Basic.QosOk,
File "C:\Python37\lib\site-packages\amqp\abstract_channel.py", line 68, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
File "C:\Python37\lib\site-packages\amqp\abstract_channel.py", line 88, in wait
    self.connection.drain_events(timeout=timeout)
File "C:\Python37\lib\site-packages\amqp\connection.py", line 504, in drain_events
    while not self.blocking_read(timeout):
File "C:\Python37\lib\site-packages\amqp\connection.py", line 509, in blocking_read
    frame = self.transport.read_frame()
File "C:\Python37\lib\site-packages\amqp\transport.py", line 252, in read_frame
    frame_header = read(7, True)
File "C:\Python37\lib\site-packages\amqp\transport.py", line 438, in _read
    s = recv(n - len(rbuf))
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

---

我找到了解决这个问题的办法。我觉得问题出在芹菜的后端。就我而言，我使用的是redis

以下是我的配置

Broker - rabbitmq
Backend - redis
Python - 3.7
OS - Windows 10

在芹菜客户端，我尝试从客户端每隔60秒ping一次芹菜工人的状态。在这种情况下，我没有面临连接重置问题

while not doors_res.ready():
    sleep(60)
result = app.get()

其中app是芹菜实例

芹菜工人方面

celery worker -A <celery_file_name> -l info -P gevent

芹菜工人-A-l信息-P gevent

---

我的任务运行了大约2个小时，我没有遇到连接重置错误。

我在Windows上也遇到了同样的问题，你找到了解决这个问题的方法吗？不幸的是，我认为第4版的Windows不支持芹菜，所以我编写了自己的小系统，它足以满足我的一些需求。我不知道这一点，那么芹菜3版能在windows上运行吗？如果我没记错的话，是的。寻找芹菜4窗口，你可能会得到一些结果，因为他们的原因。我可以知道你采取了什么途径来构建自己的系统吗？有趣！谢谢！