Java Ehcache线程死锁
我们有一个运行ehcache多年的生产应用程序，目前是ehcache 2.10.1。最近，我们注意到每天运行ehcache的所有服务器上都会出现死锁，大约10个死锁，如下所示 也许我们的负载会随着时间的推移而增加，并且对ehcache的推动力度过大。这个错误是在死锁几个小时后造成JVM挂起的情况。有什么想法吗Java Ehcache线程死锁,java,ehcache,Java,Ehcache,我们有一个运行ehcache多年的生产应用程序，目前是ehcache 2.10.1。最近，我们注意到每天运行ehcache的所有服务器上都会出现死锁，大约10个死锁，如下所示 也许我们的负载会随着时间的推移而增加，并且对ehcache的推动力度过大。这个错误是在死锁几个小时后造成JVM挂起的情况。有什么想法吗 /home> ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args|grep 89366 | awk ' $5 >= 2 ' (
/home> ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args|grep 89366 | awk ' $5 >= 2 ' (Wildfly PID 89366, find Linux lightweight process (LWP) with CPU usage greater than 2%)
89366 81979 118 prddepl+ 98.2 09:45 03:57:44 /apps/jdk1.8.0_92/bin/java -server -Xms1024m -Xmx5120m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=512m -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+ScavengeBeforeFullGC -XX:+CMSScavengeBeforeRemark -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/apps/wildfly/standalone/log -Dsun.net.inetaddr.ttl=60 -Djava.awt.headless=true -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.tanukisoftware.wrapper.WrapperManager.mbean=false -Dlogging.configuration=file:/apps/wildfly/standalone/configuration/logging.properties -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dorg.jboss.logging.Logger.pluginClass=org.jboss.logging.logmanager.LoggerPluginImpl -Djava.io.tmpdir=/tmp -Djava.net.preferIPv4Stack=true -javaagent:/apps/wildfly/standalone/lib/notsoserial-1.0-SNAPSHOT.jar -Dnotsoserial.whitelist=/apps/wildfly/bin/is-deserialized.txt -XX:-OmitStackTraceInFastThrow -Djava.library.path=/apps/wildfly/standalone/lib -classpath /apps/wildfly/standalone/lib/wrapper.jar:/apps/wildfly/standalone/lib/notsoserial-1.0-SNAPSHOT.jar:/apps/wildfly/standalone/lib/wrapper.jar -Dwrapper.key=u02jJraVjSs0PwNd -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=89364 -Dwrapper.version=3.5.13 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperJarApp /apps/wildfly/jboss-modules.jar -mp /apps/wildfly/modules -logmodule org.jboss.logmanager -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/apps/wildfly -b=0.0.0.0
[my-web1.idm.local(myuser)]
/home> printf "%x\n" 81979 (convert LWP to hex)
1403b
[my-web1.idm.local(myuser)]
/home> jstack -l 89366 | grep -A 10 1403b
"EventsThread <3096>" #248148 prio=5 os_prio=0 tid=0x00007f815c076000 nid=0x1403b runnable [0x00007f81326e5000]
java.lang.Thread.State: RUNNABLE
at net.sf.ehcache.store.disk.Segment.clearFaultedBit(Segment.java:1051)
at net.sf.ehcache.store.disk.DiskStore.clearFaultedBit(DiskStore.java:675)
at net.sf.ehcache.store.CacheStore.flush(CacheStore.java:436)
at net.sf.ehcache.Cache.flush(Cache.java:2650)
- locked <0x000000069dd8d428> (a net.sf.ehcache.Cache)
at com.mycompany.ticker.cache.ExternalTickerCacheManager.onUpdate(ExternalTickerCacheManager.java:945)
/home>ps-eLo-pid、lwp、nlwp、ruser、pcpu、stime、etime、args | grep 89366 | awk'$5>=2'（Wildfly-pid 89366，查找CPU使用率大于2%的Linux轻量级进程（lwp）
89366 81979 118 prddepl+98.2 09:45 03:57:44/apps/jdk1.8.092/bin/java-server-Xms1024m-Xmx5120m-XX:MetaspaceSize=96M-XX:MaxMetaspaceSize=512m-XX:+cmsclassUnloadinabled-XX:+CMSParallelRemarkEnabled-XX:+UseCmSinitiatingOccinecy-XX:cmSinitingOccinecyFraction=70-XX:+CleavenebeFoullgc-XX:+cmsscavengeBefore备注-XX:+UseConMarkSweepGC-XX:+HeapDumpOnOutOfMemoryError-XX:HeapDumpPath=/apps/wildfly/standalone/log-Dsun.net.inetaddr.ttl=60-Djava.awt.headless=true-djbboss.modules.system.pkgs=org.jboss.byteman-Dorg.tanukisotware.wrapper.WrapperManager.mbean=false-dloging.configuration=file:/apps/wildfly/standalone/configuration/logging.properties-Djava.util.logging.manager=org.jboss.logmanager.logmanager-Dorg.jboss.loggin.logmanager.LoggerPluginImpl-Djava.io.tmpdir=/tmp-Djava.net.preferIPv4Stack=true-javaagent:/apps/wildfly/standalone/lib/notsoserial-1.0-SNAPSHOT.jar-Dnotsoserial.whitelist=/apps/wildfly/bin/is-deserialized.txt-XX:-省略stacktraceinfastthrow-Djava.library.path=/apps/wildfly/standalone/lib-classpath/apps/wildfly/standalone/lib/notsoserial-1.0-SNAPSHOT.jar:/apps/wildfly/standalone/lib/wrapper.jar-Dwrapper.key=u02jravjss0pwnd-Dwrapper.port=32000-Dwrapper.jvm.port.min=31000-Dwrapper.jvm.port.max=31999-Dwrapper.disable_console_input=TRUE-Dwrapper.pid=89364-Dwrapper.version=3.5.13-Dwrapper.native_library=wrapper-Dwrapper.service=TRUE-Dwrapper.cpu.timeout=10-Dwrapper.jvmid=1 org.tanukisoftware.wrapper.wrapperjarapps/wildfly/jboss-modules.jar-mp/apps/wildfly/modules-logmodules-logmodules-org.jboss.logmanager-jaxpmodule-providerorg.jboss.as.standalone-Djboss.home.dir=/apps/wildfly-b=0.0.0.0
[my-web1.idm.local（myuser）]
/主页>打印文件“%x\n”81979（将LWP转换为十六进制）
1403b
[my-web1.idm.local（myuser）]
/home>jstack-l 89366 | grep-A 10 1403b
“事件线程”#248148优先级=5 os_优先级=0 tid=0x00007f815c076000 nid=0x1403b可运行[0x00007f81326e5000]
java.lang.Thread.State:可运行
位于net.sf.ehcache.store.disk.Segment.clearFaultedBit（Segment.java:1051）
位于net.sf.ehcache.store.disk.DiskStore.clearFaultedBit（DiskStore.java:675）
位于net.sf.ehcache.store.CacheStore.flush（CacheStore.java:436）
位于net.sf.ehcache.Cache.flush（Cache.java:2650）
-锁定（net.sf.ehcache.Cache）
在com.mycompany.ticker.cache.ExternalTickerCacheManager.onUpdate（ExternalTickerCacheManager.java:945）
<ehcache name="mycompany-tickerEhcache-dev" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="ehcahce.xsd">
<!--Default Cache configuration. These will applied to caches programmatically created through
the CacheManager.
The following attributes are required for defaultCache:
maxInMemory - Sets the maximum number of objects that will be created in memory
eternal - Sets whether elements are eternal. If eternal, timeouts are ignored and the element
is never expired.
timeToIdleSeconds - Sets the time to idle for an element before it expires.
i.e. The maximum amount of time between accesses before an element expires
Is only used if the element is not eternal.
Optional attribute. A value of 0 means that an Element can idle for infinity
timeToLiveSeconds - Sets the time to live for an element before it expires.
i.e. The maximum time between creation time and when an element expires.
Is only used if the element is not eternal.
overflowToDisk - Sets whether elements can overflow to disk when the in-memory cache
has reached the maxInMemory limit.
timeToIdleSeconds="500"
timeToLiveSeconds="700"
-->
<diskStore path="/apps/wildfly/conf/tickerEhcache-dev"/>
<defaultCache maxElementsInMemory="0" eternal="true" memoryStoreEvictionPolicy="LRU" diskPersistent="true" >
</defaultCache>
<cache name="com.mycompany.ticker.ehcache"
maxElementsInMemory="0"
eternal="true"
overflowToDisk="false"
memoryStoreEvictionPolicy="LFU"
diskPersistent="true" >
<bootstrapCacheLoaderFactory class="com.mycompany.ticker.cache.MyCompanyDiskStoreBootstrapCacheLoaderFactory"
properties="bootstrapAsynchronously=true, interval=5"/>
</cache>
<cache name="com.mycompany.ticker.history.ehcache" eternal="false" maxElementsInMemory="0" overflowToDisk="false" diskPersistent="true" >
<cacheWriter writeMode="write_behind" notifyListenersOnException="true" maxWriteDelay="0"
writeCoalescing="false" writeBatching="true" writeBatchSize="25"
retryAttempts="1" retryAttemptDelaySeconds="1" rateLimitPerSecond="0">
</cacheWriter>
</cache>
</ehcache>