芹菜RabbitMQ代理故障切换连接问题

在HA模式下，集群中有3个RabbitMQ节点。每个节点都位于单独的Docker容器上

我用的是芹菜4版和科姆布4版

我已使用此命令设置HA策略：

rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

芹菜配置如下所示：

CELERY = dict(
    broker_url=[
        'amqp://guest@rabbitmq1:5672',
        'amqp://guest@rabbitmq2:5672',
        'amqp://guest@rabbitmq3:5672',
    ],
   celery_queue_ha_policy='all',
   ...
)

在我停止master RabbitMQ应用程序以便使用以下命令测试芹菜故障切换功能之前，一切正常：

rabbitmqctl stop_app

RabbitMQ应用程序停止后，我立即开始在下面的日志中看到错误。日志消息的频率非常高，并且不会随着尝试次数的增加而减慢

根据日志，芹菜尝试使用下一次故障转移重新连接，但由于另一次尝试重新连接到已停止的主节点而中断。同样的事情一次又一次地发生，就像在无限循环中一样

[2017-03-17 15:10:28,084: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbitmq1:5672//: [Errno 111] Connection refused.
Will retry using next failover.

[2017-03-17 15:10:28,300: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2016 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@rabbitmq1', 'platform': 'Erlang/OTP', 'version': '3.6.6'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
[2017-03-17 15:10:28,302: DEBUG/MainProcess] ^-- substep ok
[2017-03-17 15:10:28,303: DEBUG/MainProcess] | Consumer: Starting Mingle
[2017-03-17 15:10:28,303: INFO/MainProcess] mingle: searching for neighbors
[2017-03-17 15:10:28,303: DEBUG/MainProcess] using channel_id: 1
[2017-03-17 15:10:28,318: DEBUG/MainProcess] Channel open
[2017-03-17 15:10:28,470: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 38, in start
    self.sync(c)
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 42, in sync
    replies = self.send_hello(c)
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 55, in send_hello
    replies = inspect.hello(c.hostname, our_revoked._data) or {}
  File "/usr/local/lib/python2.7/site-packages/celery/app/control.py", line 129, in hello
    return self._request('hello', from_node=from_node, revoked=revoked)
  File "/usr/local/lib/python2.7/site-packages/celery/app/control.py", line 81, in _request
    timeout=self.timeout, reply=True,
  File "/usr/local/lib/python2.7/site-packages/celery/app/control.py", line 436, in broadcast
    limit, callback, channel=channel,
  File "/usr/local/lib/python2.7/site-packages/kombu/pidbox.py", line 315, in _broadcast
    serializer=serializer)
  File "/usr/local/lib/python2.7/site-packages/kombu/pidbox.py", line 290, in _publish
    serializer=serializer,
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish
    exchange_name, declare,
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 187, in _publish
    channel = self.channel
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 209, in _get_channel
    channel = self._channel = channel()
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/functional.py", line 38, in __call__
    value = self.__value__ = self.__contract__()
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 224, in <lambda>
    channel = ChannelPromise(lambda: connection.default_channel)
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 819, in default_channel
    self.connection
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 802, in connection
    self._connection = self._establish_connection()
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 757, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
    conn.connect()
  File "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 294, in connect
    self.transport.connect()
  File "/usr/local/lib/python2.7/site-packages/amqp/transport.py", line 120, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File "/usr/local/lib/python2.7/site-packages/amqp/transport.py", line 161, in _connect
    self.sock.connect(sa)
  File "/usr/local/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
[2017-03-17 15:10:28,508: DEBUG/MainProcess] Closed channel #1
[2017-03-17 15:10:28,570: DEBUG/MainProcess] | Consumer: Restarting event loop...
[2017-03-17 15:10:28,572: DEBUG/MainProcess] | Consumer: Restarting Gossip...
[2017-03-17 15:10:28,575: DEBUG/MainProcess] | Consumer: Restarting Heart...
[2017-03-17 15:10:28,648: DEBUG/MainProcess] | Consumer: Restarting Control...
[2017-03-17 15:10:28,655: DEBUG/MainProcess] | Consumer: Restarting Tasks...
[2017-03-17 15:10:28,655: DEBUG/MainProcess] Canceling task consumer...
[2017-03-17 15:10:28,655: DEBUG/MainProcess] | Consumer: Restarting Mingle...
[2017-03-17 15:10:28,655: DEBUG/MainProcess] | Consumer: Restarting Events...
[2017-03-17 15:10:28,672: DEBUG/MainProcess] | Consumer: Restarting Connection...
[2017-03-17 15:10:28,673: DEBUG/MainProcess] | Consumer: Starting Connection
[2017-03-17 15:10:28,947: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbitmq1:5672//: [Errno 111] Connection refused.
Will retry using next failover.

[2017-03-17 15:10:29,345: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2016 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@rabbitmq1', 'platform': 'Erlang/OTP', 'version': '3.6.6'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
[2017-03-17 15:10:29,506: INFO/MainProcess] Connected to amqp://guest:**@rabbitmq2:5672//
[2017-03-17 15:10:29,535: DEBUG/MainProcess] ^-- substep ok
[2017-03-17 15:10:29,569: DEBUG/MainProcess] | Consumer: Starting Events
[2017-03-17 15:10:29,682: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbitmq1:5672//: [Errno 111] Connection refused.
Will retry using next failover.

[2017-03-17 15:10:29,740: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2016 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@rabbitmq1', 'platform': 'Erlang/OTP', 'version': '3.6.6'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
[2017-03-17 15:10:29,768: DEBUG/MainProcess] ^-- substep ok
[2017-03-17 15:10:29,770: DEBUG/MainProcess] | Consumer: Starting Mingle
[2017-03-17 15:10:29,770: INFO/MainProcess] mingle: searching for neighbors
[2017-03-17 15:10:29,771: DEBUG/MainProcess] using channel_id: 1
[2017-03-17 15:10:29,795: DEBUG/MainProcess] Channel open
[2017-03-17 15:10:29,874: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 38, in start
    self.sync(c)
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 42, in sync
    replies = self.send_hello(c)
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 55, in send_hello
    replies = inspect.hello(c.hostname, our_revoked._data) or {}
  File "/usr/local/lib/python2.7/site-packages/celery/app/control.py", line 129, in hello
    return self._request('hello', from_node=from_node, revoked=revoked)
  File "/usr/local/lib/python2.7/site-packages/celery/app/control.py", line 81, in _request
    timeout=self.timeout, reply=True,
  File "/usr/local/lib/python2.7/site-packages/celery/app/control.py", line 436, in broadcast
    limit, callback, channel=channel,
  File "/usr/local/lib/python2.7/site-packages/kombu/pidbox.py", line 315, in _broadcast
    serializer=serializer)
  File "/usr/local/lib/python2.7/site-packages/kombu/pidbox.py", line 290, in _publish
    serializer=serializer,
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish
    exchange_name, declare,
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 187, in _publish
    channel = self.channel
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 209, in _get_channel
    channel = self._channel = channel()
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/functional.py", line 38, in __call__
    value = self.__value__ = self.__contract__()
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 224, in <lambda>
    channel = ChannelPromise(lambda: connection.default_channel)
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 819, in default_channel
    self.connection
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 802, in connection
    self._connection = self._establish_connection()
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 757, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
    conn.connect()
  File "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 294, in connect
    self.transport.connect()
  File "/usr/local/lib/python2.7/site-packages/amqp/transport.py", line 120, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File "/usr/local/lib/python2.7/site-packages/amqp/transport.py", line 161, in _connect
    self.sock.connect(sa)
  File "/usr/local/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
[2017-03-17 15:10:29,887: DEBUG/MainProcess] Closed channel #1
[2017-03-17 15:10:29,907: DEBUG/MainProcess] | Consumer: Restarting event loop...
[2017-03-17 15:10:29,908: DEBUG/MainProcess] | Consumer: Restarting Gossip...
[2017-03-17 15:10:29,908: DEBUG/MainProcess] | Consumer: Restarting Heart...
[2017-03-17 15:10:29,908: DEBUG/MainProcess] | Consumer: Restarting Control...
[2017-03-17 15:10:29,909: DEBUG/MainProcess] | Consumer: Restarting Tasks...
[2017-03-17 15:10:29,910: DEBUG/MainProcess] Canceling task consumer...
[2017-03-17 15:10:29,911: DEBUG/MainProcess] | Consumer: Restarting Mingle...
[2017-03-17 15:10:29,912: DEBUG/MainProcess] | Consumer: Restarting Events...
[2017-03-17 15:10:29,953: DEBUG/MainProcess] | Consumer: Restarting Connection...
[2017-03-17 15:10:29,954: DEBUG/MainProcess] | Consumer: Starting Connection
[2017-03-17 15:10:30,036: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbitmq1:5672//: [Errno 111] Connection refused.
Will retry using next failover.

[2017-03-17 15:10:28084:错误/MainProcess]使用者：无法连接到amqp://guest:**@rabbitmq1:5672/:[Errno 111]连接被拒绝。
将使用下一次故障切换重试。
[2017-03-17 15:10:28300:DEBUG/MainProcess]从服务器启动，版本：0.9，属性：{'information'：'根据MPL授权。请参阅http://www.rabbitmq.com/“，”产品“：”RabbitMQ“，”版权“：”版权（C）2007-2016 Pivotal Software，Inc.，”功能“：{'exchange\u exchange\u bindings'：True，'connection.blocked'：True，'authentication\u failure\u close'：True，'direct\u reply\u to'：True，'basic.nack'：True，'per\u consumer\u qos'：True，'consumer\u priorities'：True，'consumer\u cancel\u notify'；True，'publisher\u confirmations'，'True}，'cluster\rabbit@rabbitmq1“，”平台“：”Erlang/OTP“，”版本“：”3.6.6'}，机制：[u'PLAIN'，u'AMQPLAIN']，地区：[u'en_US']
[2017-03-17 15:10:28302:调试/主流程]^--子步骤确定
[2017-03-17 15:10:28303:调试/主进程]|使用者：开始混合
[2017-03-17 15:10:28303:INFO/MainProcess]mingle:搜索邻居
[2017-03-17 15:10:28303:调试/主进程]使用通道id:1
[2017-03-17 15:10:28318:调试/主进程]通道打开
[2017-03-17 15:10:28470:警告/MainProcess]消费者：与代理的连接丢失。正在尝试重新建立连接。。。
回溯（最近一次呼叫最后一次）：
文件“/usr/local/lib/python2.7/site packages/celery/worker/consumer/consumer.py”，第318行，开始
蓝图。开始（自我）
文件“/usr/local/lib/python2.7/site packages/芹菜/bootsteps.py”，第119行，在开始处
步骤.开始（父级）
文件“/usr/local/lib/python2.7/site packages/芹菜/worker/consumer/mingle.py”，第38行，在开始处
自我同步（c）
文件“/usr/local/lib/python2.7/site packages/芹菜/worker/consumer/mingle.py”，第42行，同步
回复=self.send\u你好（c）
文件“/usr/local/lib/python2.7/site packages/芹菜/worker/consumer/mingle.py”，第55行，在send_hello中
repress=inspect.hello（c.hostname，我们的_撤销._数据）或{}
文件“/usr/local/lib/python2.7/site packages/芹菜/app/control.py”，第129行，在hello中
返回self.\u请求（'hello'，from\u node=from\u node，reversed=reversed）
文件“/usr/local/lib/python2.7/site packages/芹菜/app/control.py”，第81行，在请求中
timeout=self.timeout，reply=True，
文件“/usr/local/lib/python2.7/site packages/芹菜/app/control.py”，第436行，广播中
限制，回调，通道=通道，
文件“/usr/local/lib/python2.7/site packages/kombu/pidbox.py”，第315行，in_广播
序列化程序=序列化程序）
文件“/usr/local/lib/python2.7/site packages/kombu/pidbox.py”，第290行，in_publish
序列化程序=序列化程序，
文件“/usr/local/lib/python2.7/site packages/kombu/messaging.py”，第181行，在publish中
交换你的名字，声明，
文件“/usr/local/lib/python2.7/site packages/kombu/messaging.py”，第187行，in_publish
频道=自我频道
文件“/usr/local/lib/python2.7/site packages/kombu/messaging.py”，第209行，在“获取”频道中
通道=自身。_通道=通道（）
文件“/usr/local/lib/python2.7/site packages/kombu/utils/functional.py”，第38行，在调用中__
价值=自我。价值=自我。合同
文件“/usr/local/lib/python2.7/site packages/kombu/messaging.py”，第224行，在
通道=通道承诺（lambda:connection.default\u通道）
文件“/usr/local/lib/python2.7/site packages/kombu/connection.py”，第819行，默认频道
自我连接
文件“/usr/local/lib/python2.7/site packages/kombu/connection.py”，第802行，连接中
self.\u连接=self.\u建立连接（）
文件“/usr/local/lib/python2.7/site packages/kombu/connection.py”，第757行，在“建立连接”中
conn=self.transport.building_connection（）
文件“/usr/local/lib/python2.7/site packages/kombu/transport/pyamqp.py”，第130行，在建立连接中
连接
文件“/usr/local/lib/python2.7/site packages/amqp/connection.py”，第294行，在connect中
self.transport.connect（）
文件“/usr/local/lib/python2.7/site packages/amqp/transport.py”，第120行，在connect中
self.\u连接（self.host、self.port、self.connect\u超时）
文件“/usr/local/lib/python2.7/site packages/amqp/transport.py”，第161行，in\u connect
self.sock.connect（sa）
文件“/usr/local/lib/python2.7/socket.py”，第228行，meth格式
返回getattr（self.\u sock，name）（*args）
错误：[Errno 111]连接被拒绝
[2017-03-17 15:10:28508:调试/主进程]封闭通道#1
[2017-03-17 15:10:28570:调试/主进程]|使用者：重新启动事件循环。。。
[2017-03-17 15:10:28572:调试/主进程]|使用者：重新启动流言。。。
[2017-03-17 15:10:28575:调试/主进程]|消费者：重新启动心脏。。。
[2017-03-17 15:10:28648:调试/主进程]|使用者：重新启动控制。。。
[2017-03-17 15:10:28655:调试/主进程]|使用者：重新启动任务。。。
[2017-03-17 15:10:28655:调试/主进程]取消任务使用者。。。
[2017-03-17 15:10:28655:调试/主进程]|使用者：重新启动混合。。。
[2017-03-17 15:10:28655:调试/主进程]|使用者：重新启动事件。。。
[2017-03-17 15:10:28672:调试/主进程]|使用者：重新启动连接。。。
[2017-03-17 15:10:28673:调试/维护
celery worker -A app.tasks -l debug --without-mingle