Linux top命令报告处于等待状态的Java ThreadPoolExecutor线程的高CPU使用率（将Jetty server从9.0.6更新到9.4.22后）_Java_Multithreading_Jetty_Embedded Jetty_Corretto

Linux top命令报告处于等待状态的Java ThreadPoolExecutor线程的高CPU使用率（将Jetty server从9.0.6更新到9.4.22后）

java multithreading jetty

Linux top命令报告处于等待状态的Java ThreadPoolExecutor线程的高CPU使用率（将Jetty server从9.0.6更新到9.4.22后）,java,multithreading,jetty,embedded-jetty,corretto,Java,Multithreading,Jetty,Embedded Jetty,Corretto,我有一个web服务，它接收请求，并在执行某些验证后将数据转储到消息队列。此服务在内部创建ThreadPoolExecutor。我使用Prometheus和Grafana来可视化应用程序指标，并观察到即使在低工作负载时（即使我将应用程序与负载平衡器分离并且不提供任何流量），CPU使用量也始终在增加，而从未减少早些时候，我使用Jetty 9.0.6运行应用程序，并在切换到Jetty 9.4.22后观察到这种行为。回到Jetty 9.0.6（甚至9.4.13）似乎可以解决这个问题。因此，我怀疑这可能

我有一个web服务，它接收请求，并在执行某些验证后将数据转储到消息队列。此服务在内部创建ThreadPoolExecutor。我使用Prometheus和Grafana来可视化应用程序指标，并观察到即使在低工作负载时（即使我将应用程序与负载平衡器分离并且不提供任何流量），CPU使用量也始终在增加，而从未减少

早些时候，我使用Jetty 9.0.6运行应用程序，并在切换到Jetty 9.4.22后观察到这种行为。回到Jetty 9.0.6（甚至9.4.13）似乎可以解决这个问题。因此，我怀疑这可能与Jetty（9.4.13+）有关

所以我尝试分析top命令输出，同时使用VisualVM连接到应用程序

Linux top命令报告来自ThreadPoolExecutor的某个线程正在使用高CPU，CPU时间超过500分钟，但是Visual VM显示该线程处于等待状态

top

命令输出（少数线程在实际处于等待状态时报告CPU使用率较高，持续时间较长，如下所示）

线程转储中的线程状态

pool-5-thread-1
priority:5 - threadId:0x00007f13b0033000 - nativeId:0x8c9 - nativeId (decimal):2249 - state:WAITING
stackTrace:
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000037ca83da0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

pool-5-thread-5
priority:5 - threadId:0x00007f13a0020800 - nativeId:0x8cd - nativeId (decimal):2253 - state:WAITING
stackTrace:
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000037ca83da0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

pool-5-thread-7
priority:5 - threadId:0x00007f139c021800 - nativeId:0x8cf - nativeId (decimal):2255 - state:WAITING
stackTrace:
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000037ca83da0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

pool-5-thread-2
priority:5 - threadId:0x00007f13bc021800 - nativeId:0x8ca - nativeId (decimal):2250 - state:WAITING
stackTrace:
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000037ca83da0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

正在使用的JVM标志-

-XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=70
-Xss228k
-XX:G1HeapRegionSize=4
-XX:MaxGCPauseMillis=200
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-XX:NativeMemoryTracking=detail
-XX:+PrintTenuringDistribution
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=20
-XX:GCLogFileSize=512M
-Xloggc:/home/app/logs/gc/${CURRENT_DATETIME}.log
-Xms100m
-Xmx${JAVA_XMX}m
-XX:ActiveProcessorCount=${CPUS}
-XX:MaxMetaspaceSize=256m
-XX:CompressedClassSpaceSize=200m
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=6001
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.rmi.port=6001
-Djava.rmi.server.hostname=${HOST}
-XX:+UsePerfData

top - 12:16:34 up  5:03,  0 users,  load average: 10.64, 9.74, 9.72
Threads: 7123 total,  10 running, 7113 sleeping,   0 stopped,   0 zombie
%Cpu(s): 28.3 us, 16.0 sy,  0.0 ni, 54.8 id,  0.0 wa,  0.0 hi,  0.9 si,  0.0 st
KiB Mem:  32121548 total, 26088752 used,  6032796 free,   134028 buffers
KiB Swap:        0 total,        0 used,        0 free.  2008744 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 5660 app        20   0 30.582g 0.021t  97556 S 47.0 69.6 128:42.34 pool-5-thread-3
 4804 app        20   0 30.582g 0.021t  97556 R 49.6 69.6  98:41.07 pool-5-thread-1
 2695 app        20   0 30.582g 0.021t  97556 R 49.9 69.6  93:10.97 pool-5-thread-5
 2392 app        20   0 30.582g 0.021t  97556 S 48.0 69.6  79:47.46 pool-5-thread-9
 2284 app        20   0 30.582g 0.021t  97556 S  3.2 69.6  13:04.52 pool-5-thread-2

-XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=70
-Xss228k
-XX:G1HeapRegionSize=4
-XX:MaxGCPauseMillis=200
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-XX:NativeMemoryTracking=detail
-XX:+PrintTenuringDistribution
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=20
-XX:GCLogFileSize=512M
-Xloggc:/home/app/logs/gc/${CURRENT_DATETIME}.log
-Xms100m
-Xmx${JAVA_XMX}m
-XX:ActiveProcessorCount=${CPUS}
-XX:MaxMetaspaceSize=256m
-XX:CompressedClassSpaceSize=200m
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=6001
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.rmi.port=6001
-Djava.rmi.server.hostname=${HOST}
-XX:+UsePerfData