为什么JVM会因一个“";java.lang.OutOfMemoryError:Metaspace“;即使元空间没有被填满?

为什么JVM会因一个“";java.lang.OutOfMemoryError:Metaspace“;即使元空间没有被填满?,java,java-8,garbage-collection,out-of-memory,metaspace,Java,Java 8,Garbage Collection,Out Of Memory,Metaspace,我目前正在调查一个有趣的“java.lang.OutOfMemoryError:Metaspace”问题。原因我认为这是“有趣的”,因为元空间的使用远远小于最大元空间大小(1GB)。堆和GC配置包括: OS: Windows Server 2012 R2 Java HotSpot(TM) 64-Bit Server VM (25.92-b14) for windows-amd64 JRE (1.8.0_92-b14), built on Mar 31 2016 21:03:04 by "java

我目前正在调查一个有趣的“java.lang.OutOfMemoryError:Metaspace”问题。原因我认为这是“有趣的”,因为元空间的使用远远小于最大元空间大小(1GB)。堆和GC配置包括:

OS: Windows Server 2012 R2
Java HotSpot(TM) 64-Bit Server VM (25.92-b14) for windows-amd64 JRE (1.8.0_92-b14), built on Mar 31 2016 21:03:04 by "java_re" with MS VC++ 10.0 (VS2010)
Memory: 4k page, physical 29359668k(24768956k free), swap 37748276k(29744220k free)
CommandLine flags: -XX:CompressedClassSpaceSize=314572800 -XX:GCLogFileSize=104857600 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=4294967296 -XX:InitiatingHeapOccupancyPercent=45 -XX:+ManagementServer -XX:MaxHeapSize=6442450944 -XX:MaxMetaspaceFreeRatio=80 -XX:MaxMetaspaceSize=1073741824 -XX:MetaspaceSize=536870912 -XX:MinMetaspaceFreeRatio=50 -XX:NumberOfGCLogFiles=20  -XX:+PrintAdaptiveSizePolicy -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseGCLogFileRotation -XX:-UseLargePagesIndividualAllocation -XX:+UseParallelGC -XX:+UseParallelOldGC 
以下是JVM崩溃前的最后6个GC事件:

PSAdaptiveSizePolicy::compute_eden_space_size: costs minor_time: 0.001800 major_cost: 0.000613 mutator_cost: 0.997587 throughput_goal: 0.990000 live_space: 1064594816 free_space: 2587885568 old_eden_size: 1486356480 desired_eden_size: 1443889152
PSAdaptiveSizePolicy::compute_old_gen_free_space: costs minor_time: 0.001800 major_cost: 0.000613 mutator_cost: 0.997587 throughput_goal: 0.990000 live_space: 1095533824 free_space: 2545418240 old_promo_size: 1101529088 desired_promo_size: 1078460416
AdaptiveSizePolicy::old generation size: collection: 1121 (2867855360) -> (2863661056) 
AdaptiveSizeStop: collection: 1121 
[PSYoungGen: 4825K->0K(1456640K)] [ParOldGen: 2799035K->902778K(2796544K)] 2803860K->902778K(4253184K), [Metaspace: 330658K->329502K(614400K)], 1.0547274 secs] [Times: user=3.03 sys=0.00, real=1.06 secs] 
2020-02-05T07:47:01.085-0500: 23601.918: [GC (Allocation Failure) AdaptiveSizePolicy::update_averages:  survived: 55971960  promoted: 8192  overflow: false
AdaptiveSizeStart: 23601.976 collection: 1122 
  avg_survived_padded_avg: 107655896.000000  avg_promoted_padded_avg: 77367808.000000  avg_pretenured_padded_avg: 0.000000  tenuring_thresh: 1  target_size: 108003328
PSAdaptiveSizePolicy::compute_eden_space_size: costs minor_time: 0.001995 major_cost: 0.000613 mutator_cost: 0.997392 throughput_goal: 0.990000 live_space: 1098490112 free_space: 2522349568 old_eden_size: 1443889152 desired_eden_size: 1443364864
AdaptiveSizePolicy::survivor space sizes: collection: 1122 (106954752, 524288) -> (56098816, 108003328) 
AdaptiveSizeStop: collection: 1122 
[PSYoungGen: 1451520K->54660K(1464320K)] 2354298K->957446K(4260864K), 0.0585204 secs] [Times: user=0.14 sys=0.00, real=0.06 secs] 
2020-02-05T07:47:02.815-0500: 23603.648: [GC (Metadata GC Threshold) AdaptiveSizePolicy::update_averages:  survived: 10372288  promoted: 54161296  overflow: false
AdaptiveSizeStart: 23603.701 collection: 1123 
  avg_survived_padded_avg: 104833904.000000  avg_promoted_padded_avg: 84380448.000000  avg_pretenured_padded_avg: 0.000000  tenuring_thresh: 1  target_size: 104857600
PSAdaptiveSizePolicy::compute_eden_space_size: costs minor_time: 0.009104 major_cost: 0.000613 mutator_cost: 0.990283 throughput_goal: 0.990000 live_space: 1096590848 free_space: 2521825280 old_eden_size: 1443364864 desired_eden_size: 1402470400
AdaptiveSizePolicy::survivor space sizes: collection: 1123 (10485760, 56098816) -> (10485760, 104857600) 
AdaptiveSizeStop: collection: 1123 
[PSYoungGen: 221390K->10129K(1379840K)] 1124177K->965807K(4176384K), 0.0625170 secs] [Times: user=0.19 sys=0.00, real=0.06 secs] 
2020-02-05T07:47:02.878-0500: 23603.711: [Full GC (Metadata GC Threshold) AdaptiveSizeStart: 23604.878 collection: 1124 
PSAdaptiveSizePolicy::compute_eden_space_size: costs minor_time: 0.009104 major_cost: 0.012509 mutator_cost: 0.978387 throughput_goal: 0.990000 live_space: 1093844224 free_space: 2480930816 old_eden_size: 1402470400 desired_eden_size: 1520959488
PSAdaptiveSizePolicy::compute_old_gen_free_space: costs minor_time: 0.009104 major_cost: 0.012509 mutator_cost: 0.978387 throughput_goal: 0.990000 live_space: 1097600256 free_space: 2599419904 old_promo_size: 1078460416 desired_promo_size: 1328545792
AdaptiveSizeStop: collection: 1124 
[PSYoungGen: 10129K->0K(1379840K)] [ParOldGen: 955678K->804848K(2796544K)] 965807K->804848K(4176384K), [Metaspace: 336946K->336898K(620544K)], 1.1672615 secs] [Times: user=3.28 sys=0.00, real=1.17 secs] 
2020-02-05T07:47:04.046-0500: 23604.879: [GC (Last ditch collection) AdaptiveSizePolicy::update_averages:  survived: 0  promoted: 0  overflow: false
AdaptiveSizeStart: 23604.893 collection: 1125 
  avg_survived_padded_avg: 103366768.000000  avg_promoted_padded_avg: 82075424.000000  avg_pretenured_padded_avg: 0.000000  tenuring_thresh: 2  target_size: 103809024
PSAdaptiveSizePolicy::compute_eden_space_size: costs minor_time: 0.009750 major_cost: 0.012509 mutator_cost: 0.977741 throughput_goal: 0.990000 live_space: 1095128320 free_space: 2849505280 old_eden_size: 1520959488 desired_eden_size: 1525678080
AdaptiveSizePolicy::survivor space sizes: collection: 1125 (104857600, 10485760) -> (104857600, 103809024) 
AdaptiveSizeStop: collection: 1125 
[PSYoungGen: 0K->0K(1513984K)] 804848K->804848K(4310528K), 0.0142603 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] 
2020-02-05T07:47:04.061-0500: 23604.893: [Full GC (Last ditch collection) AdaptiveSizeStart: 23607.663 collection: 1126 
PSAdaptiveSizePolicy::compute_eden_space_size: costs minor_time: 0.009750 major_cost: 0.258044 mutator_cost: 0.732206 throughput_goal: 0.990000 live_space: 1092903552 free_space: 2854223872 old_eden_size: 1525678080 desired_eden_size: 1456472064
PSAdaptiveSizePolicy::compute_old_gen_free_space: costs minor_time: 0.009750 major_cost: 0.258044 mutator_cost: 0.732206 throughput_goal: 0.990000 live_space: 1093117440 free_space: 2785017856 old_promo_size: 1328545792 desired_promo_size: 1713373184
AdaptiveSizeStop: collection: 1126 
[PSYoungGen: 0K->0K(1513984K)] [ParOldGen: 804848K->787083K(2796544K)] 804848K->787083K(4310528K), [Metaspace: 336962K->328764K(620544K)], 2.7696424 secs] [Times: user=6.91 sys=0.03, real=2.77 secs] 
如果您查看最后一次元空间的使用情况,大约为328MB:
[元空间:336962K->328764K(620544K)]

根据:“MaxMetaspaceSize应用于提交的压缩类空间和其他类元数据的空间之和。”。我还检查了压缩类空间的使用情况,大约50MB


我检查了虚拟内存的使用情况。没有短缺。这就是我困惑的地方。正如您所看到的,元空间的使用是正常的,因此您似乎意识到,由于启用了
UseCompressedClassPointers
,您将得到:
MaxMetaspaceSize=CompressedClassSpaceSize+“其他元空间”
,其中
CompressedClassSpaceSize
在您的情况下大约为300MB,“其他元空间”大约是
600MB
。因此,从您的日志来看,您有大约“300MB”的空闲空间,但GC仍然失败。这里的问题可能是元空间的碎片化——您确实有空间,但它不是连续的,可能会失败。(continue:)有趣的一点是,似乎没有出现分配失败的情况,而是由于超过阈值(
Full GC(Metadata GC threshold)
)触发了GC。但是时间看起来很高。日志的这一部分不足以明确说明这一点,但JVM在GC中花费的时间似乎比其他任何东西都多。也许,这相当于“超出了GC开销”,但由于上一次GC是由Metaspace人体工程学触发的,它又回到了误导性的信息“Metaspace”。只是一个猜测。假设“最后的收集”是问题的症状,我甚至在这里只看到一个完整的GC,标记为“完整GC(元数据GC阈值)”,这并不奇怪,因为您指定了
-XX:MinMetaspaceFreeRatio=50
,并且元空间使用率超过了50%的阈值。但是这个完整的GC花费了1.167秒,之前的次要GC花费了0.063秒,而两者之间的时间和上一次分配触发的GC之间的时间是1.67秒。这是一个非常糟糕的比率,但不足以让JVM摆脱困境。另一方面,我在顶部看到另一个GC>1s,其开始时间不包括在内。@Holger这是预期的行为。当某些元数据无法分配到元空间时。因此,它首先触发了一个,如果这没有帮助,它会尝试我甚至没有看到
disch
集合,尽管在jdk-13中(但有它存在的版本),我猜他们在“某些”版本中删除了它。我仍然倾向于在这里说分裂。虽然我不认为jcmd VM.metaspace有帮助(尽管我只是为了好玩才使用它,所以我的知识有限)。似乎比我聪明得多的人(感谢Luke!)说要使用
jcmd PID VM.classloader_stats
或者对于您的情况:
jmap-clstats PID