Fatal编程技术网
  • linux/
  • Linux 在ActiveMQ Artemis中的两个节点之间发送消息失败

Linux 在ActiveMQ Artemis中的两个节点之间发送消息失败

linux

Linux 在ActiveMQ Artemis中的两个节点之间发送消息失败,linux,devops,activemq-artemis,Linux,Devops,Activemq Artemis,我使用共享存储HA策略通过JGroup协议配置了Artemis集群。当我禁用firewalld服务时，一切正常，包括故障切换和回切。但是当我启用firewalld服务时，消息无法从一个节点发送到另一个节点。 我已通过命令打开所需的端口61616和8161 sudo firewall-cmd --zone=public --permanent --add-port=61616/tcp sudo firewall-cmd --zone=public --permanent --add-port=81

我使用共享存储HA策略通过JGroup协议配置了Artemis集群。当我禁用firewalld服务时，一切正常，包括故障切换和回切。但是当我启用firewalld服务时，消息无法从一个节点发送到另一个节点。 我已通过命令打开所需的端口61616和8161

sudo firewall-cmd --zone=public --permanent --add-port=61616/tcp
sudo firewall-cmd --zone=public --permanent --add-port=8161/tcp
firewall-cmd --reload
master-broker.xml文件

<connectors>
         <connector name="netty-connector">tcp://172.16.20.129:61616</connector>
       </connectors>

       <!-- Acceptors -->
       <acceptors>
         <acceptor name="netty-acceptor">tcp://172.16.20.129:61616</acceptor>
       </acceptors>

       <!-- Clustering configuration -->
       <broadcast-groups>
         <broadcast-group name="Artemis-broadcast-group">
           <broadcast-period>5000</broadcast-period>
            <jgroups-file>test-jgroups-file_ping.xml</jgroups-file>
            <jgroups-channel>active_broadcast_channel</jgroups-channel>
            <connector-ref>netty-connector</connector-ref>
         </broadcast-group>
       </broadcast-groups>

       <discovery-groups>
         <discovery-group name="Artemis-discovery-group">
           <jgroups-file>test-jgroups-file_ping.xml</jgroups-file>
            <jgroups-channel>active_broadcast_channel</jgroups-channel>
            <refresh-timeout>10000</refresh-timeout>
         </discovery-group>
       </discovery-groups>

<!-- This is a symmetric cluster -->

       <cluster-connections>
         <cluster-connection name="Demo-Artemis-Cluster">
           <address></address>
           <connector-ref>netty-connector</connector-ref>
           <check-period>1000</check-period>
           <connection-ttl>5000</connection-ttl>
           <min-large-message-size>50000</min-large-message-size>
           <call-timeout>5000</call-timeout>
           <retry-interval>500</retry-interval>
           <retry-interval-multiplier>1.0</retry-interval-multiplier>
           <max-retry-interval>5000</max-retry-interval>
           <initial-connect-attempts>-1</initial-connect-attempts>
           <reconnect-attempts>-1</reconnect-attempts>
           <use-duplicate-detection>true</use-duplicate-detection>
           <message-load-balancing>ON_DEMAND</message-load-balancing>
           <max-hops>1</max-hops>
           <confirmation-window-size>32000</confirmation-window-size>
           <call-failover-timeout>30000</call-failover-timeout>
           <notification-interval>1000</notification-interval>
           <notification-attempts>2</notification-attempts>
           <discovery-group-ref discovery-group-name="Artemis-discovery-group"/>
         </cluster-connection>
       </cluster-connections>
从代理的日志是

[root@devops-centos-2 bin]# ./artemis run
     _        _               _
    / \  ____| |_  ___ __  __(_) _____
   / _ \|  _ \ __|/ _ \  \/  | |/  __/
  / ___ \ | \/ |_/  __/ |\/| | |\___ \
 /_/   \_\|   \__\____|_|  |_|_|/___ /
 Apache ActiveMQ Artemis 2.15.0


2020-09-28 11:05:41,421 INFO  [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2020-09-28 11:05:41,449 INFO  [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=/mnt/devops-nfs-share/shared-JGroup-data/data/journal,bindingsDirectory=/mnt/devops-nfs-share/shared-JGroup-data/data/bindings,largeMessagesDirectory=/mnt/devops-nfs-share/shared-JGroup-data/data/large-messages,pagingDirectory=/mnt/devops-nfs-share/shared-JGroup-data/data/paging)
2020-09-28 11:05:41,457 INFO  [org.apache.activemq.artemis.core.server] AMQ221032: Waiting to become backup node
2020-09-28 11:05:41,457 INFO  [org.apache.activemq.artemis.core.server] AMQ221033: ** got backup lock
2020-09-28 11:05:41,502 INFO  [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2020-09-28 11:05:41,545 INFO  [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2020-09-28 11:05:41,785 INFO  [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin    
-------------------------------------------------------------------
GMS: address=devops-centos-2-55890, cluster=active_broadcast_channel, physical address=172.16.24.85:43241
-------------------------------------------------------------------
2020-09-28 11:05:41,841 INFO  [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2020-09-28 11:05:42,167 INFO  [io.hawt.HawtioContextListener] Initialising hawtio services
2020-09-28 11:05:42,179 INFO  [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-09-28 11:05:42,181 INFO  [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-09-28 11:05:42,183 INFO  [io.hawt.jmx.UploadManager] Using file upload directory: /opt/J-sharedSlavebroker/tmp/uploads
2020-09-28 11:05:42,193 INFO  [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-09-28 11:05:42,212 INFO  [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/J-sharedSlavebroker/etc/jolokia-access.xml]
2020-09-28 11:05:42,232 INFO  [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-09-28 11:05:42,307 INFO  [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 172.16.24.85, devops-centos-2.com]
2020-09-28 11:05:43,163 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://localhost:8161    
2020-09-28 11:05:43,163 INFO  [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://localhost:8161/console
2020-09-28 11:05:44,818 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: JOIN(devops-centos-2-55890) sent to devops-centos-1-7492 timed out (after 3000 ms), on try 1
2020-09-28 11:05:47,822 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: JOIN(devops-centos-2-55890) sent to devops-centos-1-7492 timed out (after 3000 ms), on try 2
2020-09-28 11:05:50,090 SEVERE [org.jgroups.protocols.TCP] JGRP000029: devops-centos-2-55890: failed sending message to devops-centos-1-7492 (103 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[JOIN_REQ]: mbr=devops-centos-2-55890, UNICAST: DATA, seqno=3, TP: [cluster_name=active_broadcast_channel]
2020-09-28 11:05:50,825 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: JOIN(devops-centos-2-55890) sent to devops-centos-1-7492 timed out (after 3000 ms), on try 3
2020-09-28 11:05:51,131 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-2-55890: failure sending message to devops-centos-1-7492: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:05:52,091 SEVERE [org.jgroups.protocols.TCP] JGRP000036: devops-centos-2-55890: exception sending bundled msgs: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:05:53,829 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: JOIN(devops-centos-2-55890) sent to devops-centos-1-7492 timed out (after 3000 ms), on try 4    
2020-09-28 11:06:03,147 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-2-55890: failure sending message to devops-centos-1-7492: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:06:04,095 SEVERE [org.jgroups.protocols.TCP] JGRP000036: devops-centos-2-55890: exception sending bundled msgs: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:06:05,844 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: JOIN(devops-centos-2-55890) sent to devops-centos-1-7492 timed out (after 3000 ms), on try 8
2020-09-28 11:06:06,149 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-2-55890: failure sending message to devops-centos-1-18963: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:06:06,451 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-2-55890: failure sending message to devops-centos-1-7492: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:06:08,098 SEVERE [org.jgroups.protocols.TCP] JGRP000029: devops-centos-2-55890: failed sending message to devops-centos-1-7492 (103 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[JOIN_REQ]: mbr=devops-centos-2-55890, UNICAST: DATA, seqno=9, TP: [cluster_name=active_broadcast_channel]
2020-09-28 11:06:08,847 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: JOIN(devops-centos-2-55890) sent to devops-centos-1-7492 timed out (after 3000 ms), on try 9
2020-09-28 11:06:09,153 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-2-55890: failure sending message to devops-centos-1-7492: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:06:10,098 SEVERE [org.jgroups.protocols.TCP] JGRP000036: devops-centos-2-55890: exception sending bundled msgs: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:06:11,851 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: JOIN(devops-centos-2-55890) sent to devops-centos-1-7492 timed out (after 3000 ms), on try 10
2020-09-28 11:06:11,851 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: too many JOIN attempts (10): becoming singleton
2020-09-28 11:06:13,869 WARNING [org.jgroups.protocols.pbcast.FLUSH] devops-centos-2-55890: waiting for UNBLOCK timed out after 2000 ms
2020-09-28 11:06:13,893 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2020-09-28 11:06:13,893 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2020-09-28 11:06:13,893 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2020-09-28 11:06:13,894 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2020-09-28 11:06:13,894 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2020-09-28 11:06:13,894 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2020-09-28 11:06:13,943 INFO  [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.15.0 [9bc1eb6d-015f-11eb-a1d5-0050569b585d] started, waiting live to fail before it gets active
2020-09-28 11:06:23,946 WARN  [org.apache.activemq.artemis.core.server] AMQ222137: Unable to announce backup, retrying: ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219012: Timed out waiting to receive initial broadcast from cluster]            at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.executeDiscovery(ServerLocatorImpl.java:767) [artemis-core-client-2.15.0.jar:2.15.0]       at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:655) [artemis-core-client-2.15.0.jar:2.15.0]   at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549) [artemis-core-client-2.15.0.jar:2.15.0] at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528) [artemis-core-client-2.15.0.jar:2.15.0] at org.apache.activemq.artemis.core.server.cluster.BackupManager$BackupConnector$1.run(BackupManager.java:267) [artemis-server-2.15.0.jar:2.15.0] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.15.0.jar:2.15.0]  at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.15.0.jar:2.15.0]     at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.15.0.jar:2.15.0]  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_265]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_265]
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.15.0.jar:2.15.0]

2020-09-28 11:06:34,450 WARN  [org.apache.activemq.artemis.core.server] AMQ222137: Unable to announce backup, retrying: ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219012: Timed out waiting to receive initial broadcast from cluster]
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.executeDiscovery(ServerLocatorImpl.java:767) [artemis-core-client-2.15.0.jar:2.15.0]
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:655) [artemis-core-client-2.15.0.jar:2.15.0]
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549) [artemis-core-client-2.15.0.jar:2.15.0]
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528) [artemis-core-client-2.15.0.jar:2.15.0]
        at org.apache.activemq.artemis.core.server.cluster.BackupManager$BackupConnector$1.run(BackupManager.java:267) [artemis-server-2.15.0.jar:2.15.0]
        at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.15.0.jar:2.15.0]
        at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.15.0.jar:2.15.0]
        at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.15.0.jar:2.15.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_265]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_265]
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.15.0.jar:2.15.0]
是否有任何方法可以在不禁用firewalld服务的情况下发送消息。

我共享了上面的broker.xml和test-jgroups-file_ping.xml我共享了上面的broker.xml和test-jgroups-file_ping.xml 
[root@devops-centos-1 bin]# ./artemis run
     _        _               _
    / \  ____| |_  ___ __  __(_) _____
   / _ \|  _ \ __|/ _ \  \/  | |/  __/
  / ___ \ | \/ |_/  __/ |\/| | |\___ \
 /_/   \_\|   \__\____|_|  |_|_|/___ /
 Apache ActiveMQ Artemis 2.15.0


2020-09-28 10:45:46,892 INFO  [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2020-09-28 10:45:46,938 INFO  [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=/mnt/devops-nfs-share/shared-JGroup-data/data/journal,bindingsDirectory=/mnt/devops-nfs-share/shared-JGroup-data/data/bindings,largeMessagesDirectory=/mnt/devops-nfs-share/shared-JGroup-data/data/large-messages,pagingDirectory=/mnt/devops-nfs-share/shared-JGroup-data/data/paging)    
2020-09-28 10:45:47,214 WARNING [org.jgroups.stack.Configurator] JGRP000014: TP.enable_bundling has been deprecated: will be ignored as bundling is on by default
2020-09-28 10:45:47,235 WARNING [org.jgroups.stack.Configurator] JGRP000014: UNICAST.setTimeout has been deprecated: not used anymore    
-------------------------------------------------------------------
GMS: address=devops-centos-1-7492, cluster=active_broadcast_channel, physical address=172.16.20.129:45293
-------------------------------------------------------------------
2020-09-28 10:45:50,320 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-1-7492: JOIN(devops-centos-1-7492) sent to devops-centos-2-12055 timed out (after 3000 ms), on try 1
2020-09-28 10:45:53,326 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-1-7492: JOIN(devops-centos-1-7492) sent to devops-centos-2-12055 timed out (after 3000 ms), on try 2
2020-09-28 10:45:56,331 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-1-7492: JOIN(devops-centos-1-7492) sent to devops-centos-2-12055 timed out (after 3000 ms), on try 3
2020-09-28 10:45:56,638 SEVERE [org.jgroups.protocols.TCP] JGRP000029: devops-centos-1-7492: failed sending message to devops-centos-2-12055 (103 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[JOIN_REQ]: mbr=devops-centos-1-7492, UNICAST: DATA, seqno=4, TP: [cluster_name=active_broadcast_channel]
2020-09-28 10:45:57,607 SEVERE [org.jgroups.protocols.TCP] JGRP000036: devops-centos-1-7492: exception sending bundled msgs: java.net.SocketTimeoutException: connect timed out
2020-09-28 10:45:59,335 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-1-7492: JOIN(devops-centos-1-7492) sent to devops-centos-2-12055 timed out (after 3000 ms), on try 4
2020-09-28 10:45:59,641 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-1-7492: failure sending message to devops-centos-2-12055: java.net.SocketTimeoutException: connect timed out
2020-09-28 10:45:59,943 SEVERE [org.jgroups.protocols.TCP] JGRP000029: devops-centos-1-7492: failed sending message to devops-centos-2-12055 (103 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[JOIN_REQ]: mbr=devops-centos-1-7492, UNICAST: DATA, seqno=5, TP: [cluster_name=active_broadcast_channel]
2020-09-28 10:46:02,339 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-1-7492: JOIN(devops-centos-1-7492) sent to devops-centos-2-12055 timed out (after 3000 ms), on try 5
2020-09-28 10:46:02,648 SEVERE [org.jgroups.protocols.TCP] JGRP000029: devops-centos-1-7492: failed sending message to devops-centos-2-12055 (103 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[JOIN_REQ]: mbr=devops-centos-1-7492, UNICAST: DATA, seqno=6, TP: [cluster_name=active_broadcast_channel]
2020-09-28 10:46:03,611 SEVERE [org.jgroups.protocols.TCP] JGRP000036: devops-centos-1-7492: exception sending bundled msgs: java.net.SocketTimeoutException: connect timed out
2020-09-28 10:46:05,346 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-1-7492: JOIN(devops-centos-1-7492) sent to devops-centos-2-12055 timed out (after 3000 ms), on try 6
2020-09-28 10:46:05,654 SEVERE [org.jgroups.protocols.TCP] JGRP000029: devops-centos-1-7492: failed sending message to devops-centos-2-12055 (103 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[JOIN_REQ]: mbr=devops-centos-1-7492, UNICAST: DATA, seqno=7, TP: [cluster_name=active_broadcast_channel]
2020-09-28 10:46:08,350 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-1-7492: JOIN(devops-centos-1-7492) sent to devops-centos-2-12055 timed out (after 3000 ms), on try 7
2020-09-28 10:46:08,658 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-1-7492: failure sending message to devops-centos-2-12055: java.net.SocketTimeoutException: connect timed out
2020-09-28 10:46:09,613 SEVERE [org.jgroups.protocols.TCP] JGRP000036: devops-centos-1-7492: exception sending bundled msgs: java.net.SocketTimeoutException: connect timed out
2020-09-28 10:46:11,354 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-1-7492: JOIN(devops-centos-1-7492) sent to devops-centos-2-12055 timed out (after 3000 ms), on try 8
2020-09-28 10:46:11,661 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-1-7492: failure sending message to devops-centos-2-38597: java.net.SocketTimeoutException: connect timed out
2020-09-28 10:46:11,964 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-1-7492: failure sending message to devops-centos-2-12055: java.net.SocketTimeoutException: connect timed out
2020-09-28 10:46:13,616 SEVERE [org.jgroups.protocols.TCP] JGRP000036: devops-centos-1-7492: exception sending bundled msgs: java.net.SocketTimeoutException: connect timed out
2020-09-28 10:46:14,360 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-1-7492: JOIN(devops-centos-1-7492) sent to devops-centos-2-12055 timed out (after 3000 ms), on try 9
2020-09-28 10:46:14,669 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-1-7492: failure sending message to devops-centos-2-12055: java.net.SocketTimeoutException: connect timed out
2020-09-28 10:46:15,615 SEVERE [org.jgroups.protocols.TCP] JGRP000036: devops-centos-1-7492: exception sending bundled msgs: java.net.SocketTimeoutException: connect timed out
2020-09-28 10:46:17,365 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-1-7492: JOIN(devops-centos-1-7492) sent to devops-centos-2-12055 timed out (after 3000 ms), on try 10
2020-09-28 10:46:17,366 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-1-7492: too many JOIN attempts (10): becoming singleton
2020-09-28 10:46:19,400 WARNING [org.jgroups.protocols.pbcast.FLUSH] devops-centos-1-7492: waiting for UNBLOCK timed out after 2000 ms
2020-09-28 10:46:19,419 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2020-09-28 10:46:19,420 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2020-09-28 10:46:19,420 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2020-09-28 10:46:19,420 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2020-09-28 10:46:19,420 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2020-09-28 10:46:19,421 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2020-09-28 10:46:19,482 INFO  [org.apache.activemq.artemis.core.server] AMQ221034: Waiting indefinitely to obtain live lock
2020-09-28 10:46:19,483 INFO  [org.apache.activemq.artemis.core.server] AMQ221035: Live Server Obtained live lock       
2020-09-28 10:46:20,251 INFO  [org.apache.activemq.artemis.core.server] 
2020-09-28 10:46:20,256 INFO  [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2020-09-28 10:46:20,257 INFO  [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.15.0 [artemis-master, nodeID=9bc1eb6d-015f-11eb-a1d5-0050569b585d]    
2020-09-28 10:46:20,942 INFO  [io.hawt.HawtioContextListener] Initialising hawtio services
2020-09-28 10:46:20,954 INFO  [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-09-28 10:46:20,956 INFO  [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-09-28 10:46:20,957 INFO  [io.hawt.jmx.UploadManager] Using file upload directory: /opt/J-sharedMasterbroker/tmp/uploads
2020-09-28 10:46:20,968 INFO  [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-09-28 10:46:20,987 INFO  [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/J-sharedMasterbroker/etc/jolokia-access.xml]
2020-09-28 10:46:21,005 INFO  [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-09-28 10:46:21,080 INFO  [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 172.16.20.129]
2020-09-28 10:46:21,257 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://0.0.0.0:8161
2020-09-28 10:46:21,258 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2020-09-28 10:46:21,258 INFO  [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console

[root@devops-centos-2 bin]# ./artemis run
     _        _               _
    / \  ____| |_  ___ __  __(_) _____
   / _ \|  _ \ __|/ _ \  \/  | |/  __/
  / ___ \ | \/ |_/  __/ |\/| | |\___ \
 /_/   \_\|   \__\____|_|  |_|_|/___ /
 Apache ActiveMQ Artemis 2.15.0


2020-09-28 11:05:41,421 INFO  [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server
2020-09-28 11:05:41,449 INFO  [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=/mnt/devops-nfs-share/shared-JGroup-data/data/journal,bindingsDirectory=/mnt/devops-nfs-share/shared-JGroup-data/data/bindings,largeMessagesDirectory=/mnt/devops-nfs-share/shared-JGroup-data/data/large-messages,pagingDirectory=/mnt/devops-nfs-share/shared-JGroup-data/data/paging)
2020-09-28 11:05:41,457 INFO  [org.apache.activemq.artemis.core.server] AMQ221032: Waiting to become backup node
2020-09-28 11:05:41,457 INFO  [org.apache.activemq.artemis.core.server] AMQ221033: ** got backup lock
2020-09-28 11:05:41,502 INFO  [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2020-09-28 11:05:41,545 INFO  [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824
2020-09-28 11:05:41,785 INFO  [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin    
-------------------------------------------------------------------
GMS: address=devops-centos-2-55890, cluster=active_broadcast_channel, physical address=172.16.24.85:43241
-------------------------------------------------------------------
2020-09-28 11:05:41,841 INFO  [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2020-09-28 11:05:42,167 INFO  [io.hawt.HawtioContextListener] Initialising hawtio services
2020-09-28 11:05:42,179 INFO  [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2020-09-28 11:05:42,181 INFO  [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.12 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2020-09-28 11:05:42,183 INFO  [io.hawt.jmx.UploadManager] Using file upload directory: /opt/J-sharedSlavebroker/tmp/uploads
2020-09-28 11:05:42,193 INFO  [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2020-09-28 11:05:42,212 INFO  [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/J-sharedSlavebroker/etc/jolokia-access.xml]
2020-09-28 11:05:42,232 INFO  [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control
2020-09-28 11:05:42,307 INFO  [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 172.16.24.85, devops-centos-2.com]
2020-09-28 11:05:43,163 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://localhost:8161    
2020-09-28 11:05:43,163 INFO  [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://localhost:8161/console
2020-09-28 11:05:44,818 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: JOIN(devops-centos-2-55890) sent to devops-centos-1-7492 timed out (after 3000 ms), on try 1
2020-09-28 11:05:47,822 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: JOIN(devops-centos-2-55890) sent to devops-centos-1-7492 timed out (after 3000 ms), on try 2
2020-09-28 11:05:50,090 SEVERE [org.jgroups.protocols.TCP] JGRP000029: devops-centos-2-55890: failed sending message to devops-centos-1-7492 (103 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[JOIN_REQ]: mbr=devops-centos-2-55890, UNICAST: DATA, seqno=3, TP: [cluster_name=active_broadcast_channel]
2020-09-28 11:05:50,825 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: JOIN(devops-centos-2-55890) sent to devops-centos-1-7492 timed out (after 3000 ms), on try 3
2020-09-28 11:05:51,131 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-2-55890: failure sending message to devops-centos-1-7492: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:05:52,091 SEVERE [org.jgroups.protocols.TCP] JGRP000036: devops-centos-2-55890: exception sending bundled msgs: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:05:53,829 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: JOIN(devops-centos-2-55890) sent to devops-centos-1-7492 timed out (after 3000 ms), on try 4    
2020-09-28 11:06:03,147 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-2-55890: failure sending message to devops-centos-1-7492: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:06:04,095 SEVERE [org.jgroups.protocols.TCP] JGRP000036: devops-centos-2-55890: exception sending bundled msgs: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:06:05,844 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: JOIN(devops-centos-2-55890) sent to devops-centos-1-7492 timed out (after 3000 ms), on try 8
2020-09-28 11:06:06,149 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-2-55890: failure sending message to devops-centos-1-18963: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:06:06,451 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-2-55890: failure sending message to devops-centos-1-7492: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:06:08,098 SEVERE [org.jgroups.protocols.TCP] JGRP000029: devops-centos-2-55890: failed sending message to devops-centos-1-7492 (103 bytes): java.net.SocketTimeoutException: connect timed out, headers: GMS: GmsHeader[JOIN_REQ]: mbr=devops-centos-2-55890, UNICAST: DATA, seqno=9, TP: [cluster_name=active_broadcast_channel]
2020-09-28 11:06:08,847 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: JOIN(devops-centos-2-55890) sent to devops-centos-1-7492 timed out (after 3000 ms), on try 9
2020-09-28 11:06:09,153 SEVERE [org.jgroups.protocols.TCP] JGRP000034: devops-centos-2-55890: failure sending message to devops-centos-1-7492: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:06:10,098 SEVERE [org.jgroups.protocols.TCP] JGRP000036: devops-centos-2-55890: exception sending bundled msgs: java.net.SocketTimeoutException: connect timed out
2020-09-28 11:06:11,851 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: JOIN(devops-centos-2-55890) sent to devops-centos-1-7492 timed out (after 3000 ms), on try 10
2020-09-28 11:06:11,851 WARNING [org.jgroups.protocols.pbcast.GMS] devops-centos-2-55890: too many JOIN attempts (10): becoming singleton
2020-09-28 11:06:13,869 WARNING [org.jgroups.protocols.pbcast.FLUSH] devops-centos-2-55890: waiting for UNBLOCK timed out after 2000 ms
2020-09-28 11:06:13,893 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2020-09-28 11:06:13,893 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2020-09-28 11:06:13,893 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2020-09-28 11:06:13,894 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2020-09-28 11:06:13,894 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2020-09-28 11:06:13,894 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2020-09-28 11:06:13,943 INFO  [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.15.0 [9bc1eb6d-015f-11eb-a1d5-0050569b585d] started, waiting live to fail before it gets active
2020-09-28 11:06:23,946 WARN  [org.apache.activemq.artemis.core.server] AMQ222137: Unable to announce backup, retrying: ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219012: Timed out waiting to receive initial broadcast from cluster]            at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.executeDiscovery(ServerLocatorImpl.java:767) [artemis-core-client-2.15.0.jar:2.15.0]       at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:655) [artemis-core-client-2.15.0.jar:2.15.0]   at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549) [artemis-core-client-2.15.0.jar:2.15.0] at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528) [artemis-core-client-2.15.0.jar:2.15.0] at org.apache.activemq.artemis.core.server.cluster.BackupManager$BackupConnector$1.run(BackupManager.java:267) [artemis-server-2.15.0.jar:2.15.0] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.15.0.jar:2.15.0]  at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.15.0.jar:2.15.0]     at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.15.0.jar:2.15.0]  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_265]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_265]
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.15.0.jar:2.15.0]

2020-09-28 11:06:34,450 WARN  [org.apache.activemq.artemis.core.server] AMQ222137: Unable to announce backup, retrying: ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219012: Timed out waiting to receive initial broadcast from cluster]
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.executeDiscovery(ServerLocatorImpl.java:767) [artemis-core-client-2.15.0.jar:2.15.0]
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:655) [artemis-core-client-2.15.0.jar:2.15.0]
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549) [artemis-core-client-2.15.0.jar:2.15.0]
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528) [artemis-core-client-2.15.0.jar:2.15.0]
        at org.apache.activemq.artemis.core.server.cluster.BackupManager$BackupConnector$1.run(BackupManager.java:267) [artemis-server-2.15.0.jar:2.15.0]
        at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.15.0.jar:2.15.0]
        at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.15.0.jar:2.15.0]
        at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.15.0.jar:2.15.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_265]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_265]
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.15.0.jar:2.15.0]