Java性能问题-Tomcat WebappClassLoader已锁定

在对我的产品进行性能测试时，我在过去几天一直面临这个奇怪的问题，我使用的是Java6、Struts3框架和Tomcat7服务器

在性能测试期间，我们开始加载数千个UI请求，这些请求最初命中服务器，它运行正常，但在几个小时后，请求开始被阻止，随后CPU使用率达到100%，甚至UI变得无法访问。当我进行线程转储分析时，下面的问题是我一直得到的

""http-nio-8443"-exec-1305" daemon prio=10 tid=0x00007f0458538000 nid=0x56dd waiting for monitor entry [0x00007f04345c9000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1663)
    - locked <0x00000000a162f0c0> (a org.apache.catalina.loader.WebappClassLoader)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1521)
    at java.beans.Introspector.instantiate(Introspector.java:1448)
    at java.beans.Introspector.findExplicitBeanInfo(Introspector.java:431)
    at java.beans.Introspector. (Introspector.java:380)
    at java.beans.Introspector.getBeanInfo(Introspector.java:232)
    at java.beans.Introspector.getBeanInfo(Introspector.java:218)
    at com.googlecode.jsonplugin.JSONWriter.bean(JSONWriter.java:169)
    at com.googlecode.jsonplugin.JSONWriter.process(JSONWriter.java:152)
    at com.googlecode.jsonplugin.JSONWriter.value(JSONWriter.java:120)
    at com.googlecode.jsonplugin.JSONWriter.write(JSONWriter.java:88)
    at com.googlecode.jsonplugin.JSONUtil.serialize(JSONUtil.java:90)
    at com.googlecode.jsonplugin.JSONResult.execute(JSONResult.java:119)
    at com.opensymphony.xwork2.DefaultActionInvocation.executeResult(DefaultActionInvocation.java:348)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253)
    at com.facetime.imauditor.sreach.action.CustomRequestInterceptorJson.intercept(CustomRequestInterceptorJson.java:69)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:221)
    at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:150)
    at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:48)
    at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:123)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:167)
    at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:105)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:83)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:207)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:74)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:127)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at org.apache.struts2.interceptor.ProfilingActivationInterceptor.intercept(ProfilingActivationInterceptor.java:107)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:206)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:115)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:143)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:121)
    at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:170)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:123)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:176)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:50)
    at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:504)
    at org.apache.struts2.dispatcher.FilterDispatcher.doFilter(FilterDispatcher.java:419)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at com.facetime.imcoreserver.registration.AbsoluteSendRedirectFilter.doFilter(AbsoluteSendRedirectFilter.java:48)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:403)
    at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:369)
    at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:317)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1532)
    - locked <0x00000000abb53778> (a org.apache.tomcat.util.net.SecureNioChannel)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

---

这方面的任何帮助都会非常有用，请提前感谢。

更新：

我最近也面临着类似的问题。有时，当web应用程序没有很好地编程时，它可能无法正确清理网络（TCP）套接字。。。例如，数据库连接类。导致主机（Tomcat JVM）进程上打开的文件描述符过多。这可能会导致你面临同样的责任

因此，在我的例子中，我通过使用ThreadLocal generic在每个线程的基础上设计数据库连接来解决这个问题。这是在我的Web应用服务托管bean中管理的：

private static ThreadLocal<YourConnectionClass> connection = new ThreadLocal<YourConnectionClass>();

public static YourConnectionClass getConnection(){
    if (connection.get() == null) connection.set(driver.getConnection());
    return connection.get();
}

检查以下链接：

---

这是struts2 json插件中的一个错误：

---

我发现了一个类似的问题，原因是WebappClassLoader上的同步；任何时候出现大量loadClass调用—许多第三方库不幸都会这样做。理想的解决方案是不要一直调用loadClass，但这可能不是一个选项

这源于在bug 48903/44041/48694中引入Tomcat的一个更改，以修复一些用户遇到的死锁

---

在我看来，这个修复对同步过于悲观，希望我能得到一个被接受的修复，允许在进入监视器之前检查本地缓存；这个问题可以通过升级到Tomcat 8.0.16或更高版本并将Loader类设置为org.apache.catalina.Loader.ParallelWebappClassLoader来解决

在[TOMCAT_HOME]/conf/context.xml中，添加以下行并重新启动

我们遇到了一个类似的问题，Hibernate和SpringJDBC将尝试将db resultset行转换为对象。这种锁定行为严重影响了Tomcat运行大量具有大结果集的线程

---

我建议用户在Tomcat7到Tomcat8上运行Hibernate和Spring JDBC应用程序，并打开ParallelWebappClassLoader，以获得更好的性能。

您是否将垃圾收集器的活动与遇到的cpu峰值进行了比较？垃圾收集器似乎工作正常。在上面的转储中，是什么“正在等待监视器条目[0x00007f04345c9000]“干什么？也许我们看错了线索。内存消耗正常还是在增加用户负载时增加了很多？内存消耗正常，只是UI请求的响应时间增加了，并且在某一点之后请求超时。上面的线程转储保留了锁和剩余线程（1097）正在等待它释放锁。等待的请求执行的操作与上面的操作类似。与数据库或tomcat连接池的连接数或允许的会话数不是问题吗？您是否尝试将探查器连接到jvm，以便更仔细地了解所做的工作？很难说您的跑步是否有限制，或者只是一个共享集合是否同步不良。我也不会跳过内存问题。也许线程必须处理不断增加的数据量，并且不会因此而结束。感谢will的回复，我也尝试了上面的JVM选项，但仍然没有成功。

private static ThreadLocal<YourConnectionClass> connection = new ThreadLocal<YourConnectionClass>();

public static YourConnectionClass getConnection(){
    if (connection.get() == null) connection.set(driver.getConnection());
    return connection.get();
}

netstat | grep <YOUR_TCP_PORT>

lsof <YOUR_JVM_PID>

-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:SurvivorRatio=2 -XX:NewRatio=8