Caching 在wildfly 10群集(2个节点)中添加的infinispan复制缓存中经常发生复制超时异常

Caching 在wildfly 10群集(2个节点)中添加的infinispan复制缓存中经常发生复制超时异常,caching,wildfly-10,infinispan,Caching,Wildfly 10,Infinispan,我们正在使用wildfly 10(共2个节点)建立集群环境。我们添加了一个新的缓存容器和名为“userSessionData”的复制缓存,用于跨两台服务器共享数据。使用Standalone-full-ha.xml。在我们的开发环境中,一切正常 但在客户机预生产环境中,我们经常会遇到复制超时异常。在我们的开发环境中,我们使用默认的远程超时设置,即17500毫秒,我认为这与复制超时相对应。在客户机预生产环境中,我们将其增加到60000毫秒,即1分钟,但仍然经常遇到相同的异常。超时问题场景之一:-当用

我们正在使用wildfly 10(共2个节点)建立集群环境。我们添加了一个新的缓存容器和名为“userSessionData”的复制缓存,用于跨两台服务器共享数据。使用Standalone-full-ha.xml。在我们的开发环境中,一切正常

但在客户机预生产环境中,我们经常会遇到复制超时异常。在我们的开发环境中,我们使用默认的远程超时设置,即17500毫秒,我认为这与复制超时相对应。在客户机预生产环境中,我们将其增加到60000毫秒,即1分钟,但仍然经常遇到相同的异常。超时问题场景之一:-当用户登录到应用程序时,我们将用户会话详细信息放入缓存。在写入缓存期间,发生异常

我附带了infinispan配置,缓存写入和异常期间使用的Java代码。也贴在下面

下面是添加缓存的infinispan配置

<cache-container name="userSessionData_cache" default-cache="default" jndi-name="infinispan/userSessionData_cache">
                <transport lock-timeout="60000"/>
                <replicated-cache name="userSessionData" jndi-name="infinispan/userSessionData_cache/userSessionData" mode="SYNC" remote-timeout="60000">
                    <transaction mode="BATCH"/>
                </replicated-cache>
</cache-container>
下面是用于缓存的java类(使用类的单例模式):


我们遇到了一个问题,这些超时在使用默认吞吐量垃圾收集的垃圾收集事件期间发生。它可以通过多种方式进行调整以减少暂停,包括并行gc或cms收集

根据您使用jgroups的方式,还可能出现某种网络问题,但根据我的观察,这种情况更可能发生在“停止世界”垃圾收集条件下,因为服务在内存压力下增长

-XX:+useConMarkSweepGC-XX:+UseParNewGC

由于infinispan是一个缓存,因此超时非常小,超过了jvm暂停毫秒,这通常被认为是某些应用程序可以接受的


请阅读-总结是,这不是一种向志愿者讲话的理想方式,可能会对获得答案产生反作用。请不要将此添加到您的问题中。注意要点。谢谢下次不会再重复了。好的,谢谢。关于您的编辑,大小写是否正确:它是驼峰式的。这里要做的典型事情是找出复制超时出现的原因。有各种各样的可能性,但是您应该首先检查内存消耗以及GC是否在另一个节点中发生。您还应该看看是否可以在超时的节点中获得一些线程转储,可能是挂起了什么?…等等
public class UserSessionTracker {

    private UserSessionTracker() {
    }
    private static UserSessionTracker instance = new UserSessionTracker();

    public static void putUserSession(String userName, String sessionId) {
        synchronized (instance) {
                UserSessionDataCacheManager.getInstance().putUserSession(userName, sessionId);
        }
    }
 }
@ManagedBean(name = "userSessionDataCacheManager", eager = true)
@ApplicationScoped
public class UserSessionDataCacheManager {

@Resource(name = "java:jboss/infinispan/userSessionData_cache/userSessionData")
    private Cache<String, java.util.Hashtable<String, Object>> entityCache = null;

    private static UserSessionDataCacheManager configInstance = null;

    public UserSessionDataCacheManager() {
        configInstance = this;
    }

    @PostConstruct
    public void start() {
System.out.println("Cache name: " + entityCache.getName());
    }

    public static UserSessionDataCacheManager getInstance() {
        if (configInstance == null) {
            createInstance();
        }
        return configInstance;
    }

    private static void createInstance() {
        configInstance = new UserSessionDataCacheManager();
    }

    public void putUserSession(String userName, String sessionId) {
System.out.println("inside  putUserSession");
    java.util.Hashtable<String, Object> sessionData = new java.util.Hashtable<String, Object>();
        sessionData.put("Session", sessionId);
        entityCache.put(userName, sessionData);

    }
}
2017-07-27 11:53:54,539 INFO  [stdout] (default task-15) inside putUserSession

2017-07-27 11:54:54,571 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-15) ISPN000136: Error executing command PrepareCommand, writing keys [F2959267]: org.infinispan.util.concurrent.TimeoutException: Replication timeout for fit-r3stems01
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
    at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
    at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

2017-07-27 11:54:54,571 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-15) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: Replication timeout for fit-r3stems01
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
    at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
    at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

2017-07-27 11:55:54,587 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-15) ISPN000136: Error executing command RollbackCommand, writing keys [F2959267]: org.infinispan.util.concurrent.TimeoutException: Replication timeout for fit-r3stems01
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
    at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
    at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

2017-07-27 11:55:54,587 WARN  [org.infinispan.transaction.impl.TransactionCoordinator] (default task-15) ISPN000141: Could not rollback prepared 1PC transaction. This transaction will be rolled back by the recovery process, if enabled. Transaction: LocalXaTransaction{xid=DummyXid{id=77}} LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=[], backupKeyLocks=[], topologyId=3, stateTransferFlag=null} org.infinispan.transaction.xa.LocalXaTransaction@8b: org.infinispan.util.concurrent.TimeoutException: Replication timeout for fit-r3stems01
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
    at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
    at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

2017-07-27 11:55:54,587 WARN  [org.infinispan.transaction.tm.DummyTransaction] (default task-15) ISPN000112: exception while committing: javax.transaction.xa.XAException
    at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:207)
    at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
    at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:114)
    at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:401)
    at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:448)
    at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:321)
    at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:108)
    at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:71)
    at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:81)
    at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:814)
    at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:565)
    at io.undertow.servlet.spec.HttpServletResponseImpl.sendRedirect(HttpServletResponseImpl.java:202)
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
    at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
    at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
    at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
    at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
    at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
    at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
    at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
    at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
    at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
    at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
    at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for fit-r3stems01
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
    at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
    at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    ... 3 more

2017-07-27 11:55:54,712 WARN  [org.wildfly.clustering.web.undertow] (default task-15) javax.transaction.HeuristicMixedException: org.infinispan.commons.CacheException: javax.transaction.HeuristicMixedException
    at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:74)
    at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:81)
    at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:814)
    at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:565)
    at io.undertow.servlet.spec.HttpServletResponseImpl.sendRedirect(HttpServletResponseImpl.java:202)
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
    at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
    at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
    at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
    at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
    at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
    at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
    at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
    at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
    at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
    at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
    at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.transaction.HeuristicMixedException
    at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:439)
    at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:448)
    at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:321)
    at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:108)
    at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:71)
    ... 161 more
Caused by: javax.transaction.xa.XAException
    at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:207)
    at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
    at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:114)
    at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:401)
    ... 165 more
Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for fit-r3stems01
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
    at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
    at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    ... 3 more