Apache zookeeper 带zookeeper集群的mesos主崩溃

我正在部署一个zookeeper集群，它有3个节点。我使用它来保持mesos master的高可用性。我下载zookeeper-3.4.6.tar.gz tarball并将其解压缩到/opt，将其重命名为/opt/zookeper，进入目录，编辑conf/zoo.cfg（粘贴在下面），在dataDir中创建一个myid文件（设置为/var/lib/zookeper in zoo.cfg），然后使用/bin/zkServer.sh start启动zookeeper，一切正常。我一个接一个地启动所有3个节点，它们看起来都很好。我使用./bin/zkCli.sh连接服务器，没有问题

但当我启动mesos（3个主节点和3个从节点，每个节点运行一个主节点和一个从节点）时，主节点很快一个接一个地崩溃，在网页的“从节点”选项卡中，没有显示从节点。但当我只管理一个动物园管理员时，这些都很好。所以我认为这是动物园管理员集群的问题

我的ubuntu服务器上有3台PV主机。他们都在运行ubuntu 14.04 LTS： 节点01，节点02，节点03， 我在这三个节点中都有

/etc/hosts

，如下所示：

172.16.2.70     node-01
172.16.2.81     node-02
172.16.2.80     node-03

F0817 15:09:19.995256  2250 master.cpp:1253] Recovery failed: Failed to recover registrar: Failed to perform fetch within 1mins
*** Check failure stack trace: ***
    @     0x7fa2b8be71a2  google::LogMessage::Fail()
    @     0x7fa2b8be70ee  google::LogMessage::SendToLog()
    @     0x7fa2b8be6af0  google::LogMessage::Flush()
    @     0x7fa2b8be9a04  google::LogMessageFatal::~LogMessageFatal()

▽
    @     0x7fa2b81a899a  mesos::internal::master::fail()

▽
    @     0x7fa2b8262f8f  _ZNSt5_BindIFPFvRKSsS1_EPKcSt12_PlaceholderILi1EEEE6__callIvJS1_EJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE

▽
    @     0x7fa2b823fba7  _ZNSt5_BindIFPFvRKSsS1_EPKcSt12_PlaceholderILi1EEEEclIJS1_EvEET0_DpOT_
    @     0x7fa2b820f9f3  _ZZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvRKSsS6_EPKcSt12_PlaceholderILi1EEEEvEERKS2_OT_NS2_6PreferEENUlS6_E_clES6_
    @     0x7fa2b826305c  _ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvS1_S1_EPKcSt12_PlaceholderILi1EEEEvEERKS6_OT_NS6_6PreferEEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_
    @           0x4a44e7  std::function<>::operator()()
    @           0x49f3a7  _ZN7process8internal3runISt8functionIFvRKSsEEJS4_EEEvRKSt6vectorIT_SaIS8_EEDpOT0_
    @           0x499480  process::Future<>::fail()
    @     0x7fa2b806b4b4  process::Promise<>::fail()
    @     0x7fa2b826011b  process::internal::thenf<>()
    @     0x7fa2b82a0757  _ZNSt5_BindIFPFvRKSt8functionIFN7process6FutureI7NothingEERKN5mesos8internal8RegistryEEERKSt10shared_ptrINS1_7PromiseIS3_EEERKNS2_IS7_EEESB_SH_St12_PlaceholderILi1EEEE6__callIvISM_EILm0ELm1ELm2EEEET_OSt5tupleIIDpT0_EESt12_Index_tupleIIXspT1_EEE
    @     0x7fa2b82962d9  std::_Bind<>::operator()<>()
    @     0x7fa2b827ee89  std::_Function_handler<>::_M_invoke()
I0817 15:09:20.098639  2248 http.cpp:283] HTTP GET for /master/state.json from 172.16.2.84:54542 with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36'
    @     0x7fa2b8296507  std::function<>::operator()()
    @     0x7fa2b827efaf  _ZZNK7process6FutureIN5mesos8internal8RegistryEE5onAnyIRSt8functionIFvRKS4_EEvEES8_OT_NS4_6PreferEENUlS8_E_clES8_
    @     0x7fa2b82a07fe  _ZNSt17_Function_handlerIFvRKN7process6FutureIN5mesos8internal8RegistryEEEEZNKS5_5onAnyIRSt8functionIS8_EvEES7_OT_NS5_6PreferEEUlS7_E_E9_M_invokeERKSt9_Any_dataS7_
    @     0x7fa2b8296507  std::function<>::operator()()
    @     0x7fa2b82e4419  process::internal::run<>()
    @     0x7fa2b82da22a  process::Future<>::fail()
    @     0x7fa2b83136b5  std::_Mem_fn<>::operator()<>()
    @     0x7fa2b830efdf  _ZNSt5_BindIFSt7_Mem_fnIMN7process6FutureIN5mesos8internal8RegistryEEEFbRKSsEES6_St12_PlaceholderILi1EEEE6__callIbIS8_EILm0ELm1EEEET_OSt5tupleIIDpT0_EESt12_Index_tupleIIXspT1_EEE
    @     0x7fa2b8307d7f  _ZNSt5_BindIFSt7_Mem_fnIMN7process6FutureIN5mesos8internal8RegistryEEEFbRKSsEES6_St12_PlaceholderILi1EEEEclIJS8_EbEET0_DpOT_
    @     0x7fa2b82fe431  _ZZNK7process6FutureIN5mesos8internal8RegistryEE8onFailedISt5_BindIFSt7_Mem_fnIMS4_FbRKSsEES4_St12_PlaceholderILi1EEEEbEERKS4_OT_NS4_6PreferEENUlS9_E_clES9_
    @     0x7fa2b830f065  _ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureIN5mesos8internal8RegistryEE8onFailedISt5_BindIFSt7_Mem_fnIMS8_FbS1_EES8_St12_PlaceholderILi1EEEEbEERKS8_OT_NS8_6PreferEEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_
    @           0x4a44e7  std::function<>::operator()()
    @           0x49f3a7  _ZN7process8internal3runISt8functionIFvRKSsEEJS4_EEEvRKSt6vectorIT_SaIS8_EEDpOT0_
    @     0x7fa2b82da202  process::Future<>::fail()
    @     0x7fa2b82d2d82  process::Promise<>::fail()
Aborted

我在所有三个节点上安装了zookeeper和mesos。Zookeeper配置文件如下（所有三个节点）：

它们可以正常启动并运行良好。然后我使用命令行

/bin/mesos-master.sh--zk=zk://172.16.2.70:2181172.16.2.81:2181172.16.2.80:2181/mesos--work_dir=/var/lib/mesos--quorum=2

，启动mesos主服务，几秒钟后，它会给出如下错误：

172.16.2.70     node-01
172.16.2.81     node-02
172.16.2.80     node-03

F0817 15:09:19.995256  2250 master.cpp:1253] Recovery failed: Failed to recover registrar: Failed to perform fetch within 1mins
*** Check failure stack trace: ***
    @     0x7fa2b8be71a2  google::LogMessage::Fail()
    @     0x7fa2b8be70ee  google::LogMessage::SendToLog()
    @     0x7fa2b8be6af0  google::LogMessage::Flush()
    @     0x7fa2b8be9a04  google::LogMessageFatal::~LogMessageFatal()

▽
    @     0x7fa2b81a899a  mesos::internal::master::fail()

▽
    @     0x7fa2b8262f8f  _ZNSt5_BindIFPFvRKSsS1_EPKcSt12_PlaceholderILi1EEEE6__callIvJS1_EJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE

▽
    @     0x7fa2b823fba7  _ZNSt5_BindIFPFvRKSsS1_EPKcSt12_PlaceholderILi1EEEEclIJS1_EvEET0_DpOT_
    @     0x7fa2b820f9f3  _ZZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvRKSsS6_EPKcSt12_PlaceholderILi1EEEEvEERKS2_OT_NS2_6PreferEENUlS6_E_clES6_
    @     0x7fa2b826305c  _ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvS1_S1_EPKcSt12_PlaceholderILi1EEEEvEERKS6_OT_NS6_6PreferEEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_
    @           0x4a44e7  std::function<>::operator()()
    @           0x49f3a7  _ZN7process8internal3runISt8functionIFvRKSsEEJS4_EEEvRKSt6vectorIT_SaIS8_EEDpOT0_
    @           0x499480  process::Future<>::fail()
    @     0x7fa2b806b4b4  process::Promise<>::fail()
    @     0x7fa2b826011b  process::internal::thenf<>()
    @     0x7fa2b82a0757  _ZNSt5_BindIFPFvRKSt8functionIFN7process6FutureI7NothingEERKN5mesos8internal8RegistryEEERKSt10shared_ptrINS1_7PromiseIS3_EEERKNS2_IS7_EEESB_SH_St12_PlaceholderILi1EEEE6__callIvISM_EILm0ELm1ELm2EEEET_OSt5tupleIIDpT0_EESt12_Index_tupleIIXspT1_EEE
    @     0x7fa2b82962d9  std::_Bind<>::operator()<>()
    @     0x7fa2b827ee89  std::_Function_handler<>::_M_invoke()
I0817 15:09:20.098639  2248 http.cpp:283] HTTP GET for /master/state.json from 172.16.2.84:54542 with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36'
    @     0x7fa2b8296507  std::function<>::operator()()
    @     0x7fa2b827efaf  _ZZNK7process6FutureIN5mesos8internal8RegistryEE5onAnyIRSt8functionIFvRKS4_EEvEES8_OT_NS4_6PreferEENUlS8_E_clES8_
    @     0x7fa2b82a07fe  _ZNSt17_Function_handlerIFvRKN7process6FutureIN5mesos8internal8RegistryEEEEZNKS5_5onAnyIRSt8functionIS8_EvEES7_OT_NS5_6PreferEEUlS7_E_E9_M_invokeERKSt9_Any_dataS7_
    @     0x7fa2b8296507  std::function<>::operator()()
    @     0x7fa2b82e4419  process::internal::run<>()
    @     0x7fa2b82da22a  process::Future<>::fail()
    @     0x7fa2b83136b5  std::_Mem_fn<>::operator()<>()
    @     0x7fa2b830efdf  _ZNSt5_BindIFSt7_Mem_fnIMN7process6FutureIN5mesos8internal8RegistryEEEFbRKSsEES6_St12_PlaceholderILi1EEEE6__callIbIS8_EILm0ELm1EEEET_OSt5tupleIIDpT0_EESt12_Index_tupleIIXspT1_EEE
    @     0x7fa2b8307d7f  _ZNSt5_BindIFSt7_Mem_fnIMN7process6FutureIN5mesos8internal8RegistryEEEFbRKSsEES6_St12_PlaceholderILi1EEEEclIJS8_EbEET0_DpOT_
    @     0x7fa2b82fe431  _ZZNK7process6FutureIN5mesos8internal8RegistryEE8onFailedISt5_BindIFSt7_Mem_fnIMS4_FbRKSsEES4_St12_PlaceholderILi1EEEEbEERKS4_OT_NS4_6PreferEENUlS9_E_clES9_
    @     0x7fa2b830f065  _ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureIN5mesos8internal8RegistryEE8onFailedISt5_BindIFSt7_Mem_fnIMS8_FbS1_EES8_St12_PlaceholderILi1EEEEbEERKS8_OT_NS8_6PreferEEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_
    @           0x4a44e7  std::function<>::operator()()
    @           0x49f3a7  _ZN7process8internal3runISt8functionIFvRKSsEEJS4_EEEvRKSt6vectorIT_SaIS8_EEDpOT0_
    @     0x7fa2b82da202  process::Future<>::fail()
    @     0x7fa2b82d2d82  process::Promise<>::fail()
Aborted

---

我想知道zookeeper在我的案例中是否部署和运行良好，以及如何找到问题所在。欢迎提供任何答案和建议。谢谢。

事实上，就我而言，这是因为我没有打开防火墙端口5050以允许三台服务器相互通信。更新防火墙规则后，它开始按预期工作。

将mesos masters的工作目录拆分为不同的目录，不要对所有主机使用共享工作目录，因为zk

我遇到了相同的问题，我尝试了不同的方法和不同的选项，最后，

--ip

选项对我有效。最初我使用了

--hostname

选项

mesos-master --ip=192.168.0.13 --quorum=2 --zk=zk://m1:2181,m2:2181,m3:2181/mesos --work_dir=/opt/mm1 --log_dir=/opt/mm1/logs

---

您需要检查所有mesos/zookeeper主节点是否能够正确通信。为此，您需要：

• Zookeeper端口打开：TCP 2181、2888、3888
• Mesos端口打开：TCP 5050
• ping可用（ICMP消息0和8）

---

如果在配置中使用FQDN而不是IP，请检查DNS解析是否也正常工作。

是否在所有节点中启动master？您可以使用zkCli.sh connect zk://172.16.2.70:2181172.16.2.81:2181172.16.2.80:2181在每个节点上成功吗？您好@haosdent，我已经在三个节点上启动了所有三个主节点。使用zkCli.sh，我可以连接任何zk服务器。后来我尝试只启动一个zookeeper节点，在这种情况下，无论启动多个主节点还是一个主节点，它们都工作得很好。所以我怀疑是动物园管理员的配置导致mesos master崩溃。我在这里发现了一个类似的错误。他在下面的评论中也有类似的错误，我们有一个jira问题来跟踪网络分区时的mesos崩溃。似乎与你的问题相似。嗨，你能为@haosdent上面提到的Mesos Jira问题附上一份完整的主日志（所有三个主日志）吗？我们有几个人在积极地研究这个问题，如果再举一个例子，我们将不胜感激。