Cluster computing arangodb群集重新启动失败

我们在5台服务器上设置了一个arangodb集群，其中包含3个代理、5个协调员和5台db服务器

环境：Centos 6

我们的经验是，如果超过其中一台服务器上的最大内存，集群将完全失败。为了避免这种情况，由于我们没有找到控制内存使用的方法，我们使用命令

top | grep arangod

定期观察每个节点，如果它们消耗太多，则重新启动它们。它通常很好用。但当我们尝试重新启动一个节点时，我们收到如下日志：

    2018/03/27 15:47:31 Failed to get master URL, retrying in 5sec (All 3 servers responded with temporary failure)
    2018/03/27 15:47:31 ## Start of dbserver log
        2018-03-27T07:46:31Z [37755] WARNING {memory} It is recommended to set NUMA to interleaved.
        2018-03-27T07:46:31Z [37755] WARNING {memory} put 'numactl --interleave=all' in front of your command
        2018-03-27T07:46:31Z [37755] INFO using storage engine rocksdb
        2018-03-27T07:46:31Z [37755] INFO {cluster} Starting up with role PRIMARY
        2018-03-27T07:46:41Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.30:8531. Unsuccessful consecutive tries: 21 (9.84s). Network checks advised.
        2018-03-27T07:46:42Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.28:8531. Unsuccessful consecutive tries: 22 (10.82s). Network checks advised.
        2018-03-27T07:46:43Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.29:8531. Unsuccessful consecutive tries: 23 (11.89s). Network checks advised.
        2018-03-27T07:46:44Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.30:8531. Unsuccessful consecutive tries: 24 (13.03s). Network checks advised.
        2018-03-27T07:46:46Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.28:8531. Unsuccessful consecutive tries: 25 (14.25s). Network checks advised.
        2018-03-27T07:46:47Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.29:8531. Unsuccessful consecutive tries: 26 (15.57s). Network checks advised.
        2018-03-27T07:46:48Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.30:8531. Unsuccessful consecutive tries: 27 (16.99s). Network checks advised.
        2018-03-27T07:46:50Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.28:8531. Unsuccessful consecutive tries: 28 (18.51s). Network checks advised.
        2018-03-27T07:46:51Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.29:8531. Unsuccessful consecutive tries: 29 (20.15s). Network checks advised.
        2018-03-27T07:46:53Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.30:8531. Unsuccessful consecutive tries: 30 (21.9s). Network checks advised.
        2018-03-27T07:46:55Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.28:8531. Unsuccessful consecutive tries: 31 (23.8s). Network checks advised.
        2018-03-27T07:46:57Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.29:8531. Unsuccessful consecutive tries: 32 (25.83s). Network checks advised.
        2018-03-27T07:46:59Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.30:8531. Unsuccessful consecutive tries: 33 (28.01s). Network checks advised.
        2018-03-27T07:47:02Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.28:8531. Unsuccessful consecutive tries: 34 (30.36s). Network checks advised.
        2018-03-27T07:47:04Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.29:8531. Unsuccessful consecutive tries: 35 (32.89s). Network checks advised.
        2018-03-27T07:47:04Z [37755] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.28:8531. Unsuccessful consecutive tries: 36 (32.89s). Network checks advised.
2018/03/27 15:47:31 ## End of dbserver log
2018/03/27 15:47:32 ## Start of coordinator log
        2018-03-27T07:46:32Z [37769] WARNING {memory} It is recommended to set NUMA to interleaved.
        2018-03-27T07:46:32Z [37769] WARNING {memory} put 'numactl --interleave=all' in front of your command
        2018-03-27T07:46:32Z [37769] INFO using storage engine rocksdb
        2018-03-27T07:46:32Z [37769] INFO {cluster} Starting up with role COORDINATOR
        2018-03-27T07:46:42Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.30:8531. Unsuccessful consecutive tries: 21 (9.84s). Network checks advised.
        2018-03-27T07:46:43Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.28:8531. Unsuccessful consecutive tries: 22 (10.82s). Network checks advised.
        2018-03-27T07:46:44Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.29:8531. Unsuccessful consecutive tries: 23 (11.89s). Network checks advised.
        2018-03-27T07:46:45Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.30:8531. Unsuccessful consecutive tries: 24 (13.03s). Network checks advised.
        2018-03-27T07:46:47Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.28:8531. Unsuccessful consecutive tries: 25 (14.25s). Network checks advised.
        2018-03-27T07:46:48Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.29:8531. Unsuccessful consecutive tries: 26 (15.57s). Network checks advised.
        2018-03-27T07:46:49Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.30:8531. Unsuccessful consecutive tries: 27 (16.99s). Network checks advised.
        2018-03-27T07:46:51Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.28:8531. Unsuccessful consecutive tries: 28 (18.51s). Network checks advised.
        2018-03-27T07:46:52Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.29:8531. Unsuccessful consecutive tries: 29 (20.14s). Network checks advised.
        2018-03-27T07:46:54Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.30:8531. Unsuccessful consecutive tries: 30 (21.9s). Network checks advised.
        2018-03-27T07:46:56Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.28:8531. Unsuccessful consecutive tries: 31 (23.8s). Network checks advised.
        2018-03-27T07:46:58Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.29:8531. Unsuccessful consecutive tries: 32 (25.83s). Network checks advised.
        2018-03-27T07:47:00Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.30:8531. Unsuccessful consecutive tries: 33 (28.01s). Network checks advised.
        2018-03-27T07:47:03Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.28:8531. Unsuccessful consecutive tries: 34 (30.36s). Network checks advised.
        2018-03-27T07:47:05Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.29:8531. Unsuccessful consecutive tries: 35 (32.89s). Network checks advised.
        2018-03-27T07:47:05Z [37769] INFO {agencycomm} Flaky agency communication to http+tcp://65.18.27.28:8531. Unsuccessful consecutive tries: 36 (32.89s). Network checks advised.
2018/03/27 15:47:32 ## End of coordinator log
2018/03/27 15:47:46 Failed to get master URL, retrying in 5sec (All 3 servers responded with temporary failure)

所有服务器之间的ping都很好，因此这不是网络问题

就在我写这个问题并收集日志信息时，集群成功重启。这有点古怪。 现在有两个节点将日志打印为

updated cluster config does not contain myself. rejecting

现在显示集合需要很长时间，而且集群无法正常工作。

---

有人知道为什么吗？

[引用github讨论]

请注意，只有在第一次启动群集时才必须使用命令--cluster.agency-size 5。这是因为启动器在第一次启动时写入了无法再更改的集群配置

因此，在您的情况下，如果需要在其他节点中添加更多代理，则必须在每个新节点上使用--cluster.start-agent true 如果您希望在关闭两个（随机）节点时确保5节点集群处于运行状态，那么您需要的是代理大小=5

如果代理未启动并运行，群集将无法工作。该机构利用RAFT协议。如果您的代理由3个代理组成，那么如果两个代理关闭，则代理关闭（对于集群也是如此）。如果您的代理由5个代理组成，那么如果两个代理失败，那么代理将继续存在（对于您的集群也是如此）

如果你想在3台机器下生存下来，那么其他设置是可能的

你也可以考虑使用独立的机器来做代理，例如：

• 3台机构专用机器
• 再加上DBServer+协调器的3台机器（总共6台机器），复制系数=3

---

上述设置将持续到1个代理关闭，2个数据库服务器关闭（总共3台机器关闭）

这可能是最好的地方：@Andrew，谢谢你的建议，我在这篇文章中提到了这个问题，根据github上的讨论将你的问题的答案发布出来是个好主意。你可以回答自己的问题并接受这个答案，这样人们就知道问题已经解决了。这个问题还存在吗？你需要帮助解决上述问题吗？@KavehVahedipour，是的，我仍然不知道这个问题的真正原因以及如何避免它。你能帮助我吗？