Linux与Java进程

在我们的生产环境中，我经常面临Tomcat进程被Linux杀死的问题

检查/var/log/messages，它表示java没有被污染，java调用了OOM killer

-Xms20480m-Xmx20480m安装在32 GB的机箱上

我看到下面的崩溃-

是OOM导致了这次事故吗？还是因为OOM而发生了车祸？ 如何调试此问题

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f4c3230aad7, pid=16248, tid=139964439296320
#
# JRE version: Java(TM) SE Runtime Environment (7.0_45-b18) (build 1.7.0_45-b18)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.45-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x674ad7]  JVM_Clone+0x97
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x0000000001313800):  JavaThread "http-bio-14080-exec-17" daemon [_thread_in_native, id=18943, stack(0x00007f4c029f7000,0x00007f4c02af8000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=128 (), si_addr=0x0000000000000000

Registers:
RAX=0x0000000000010000, RBX=0x80000001000004db, RCX=0x0000000302bd6fb0, RDX=0x00007f4c32acda50
RSP=0x00007f4c02af2c88, RBP=0x00007f4c02af2d70, RSI=0x00007f4c02af2d80, RDI=0x00000000013139e8
R8 =0x0000000000000001, R9 =0x00000002f8191228, R10=0x00007f4c2e231658, R11=0x00007f4c2e231638
R12=0x0000000302bd6fb0, R13=0x00000007e0002420, R14=0x0000000000000000, R15=0x0000000001313800
RIP=0x00007f4c3230aad7, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000000
  TRAPNO=0x000000000000000d

Top of Stack: (sp=0x00007f4c02af2c88)
0x00007f4c02af2c88:   00007f4c3230aa63 00000000129196d1
0x00007f4c02af2c98:   00000002f817b620 00000000013139e8
0x00007f4c02af2ca8:   00007f4c02af2d18 00000002f8191080
0x00007f4c02af2cb8:   0000000000000009 0000000001313800
0x00007f4c02af2cc8:   0000000000000000 0000000001313800
0x00007f4c02af2cd8:   00000000138f5509 0000000000000000
0x00007f4c02af2ce8:   00007f4c2e382ae8 0000000000000000
0x00007f4c02af2cf8:   00007f4c02af2cf8 00000002f817b620
0x00007f4c02af2d08:   00000002f8190fd8 00000007e01ab278
0x00007f4c02af2d18:   00000002f8190ec0 00000007e0002420
0x00007f4c02af2d28:   0000000000000000 00000007e0002420
0x00007f4c02af2d38:   0000000000000000 0000000000000000
0x00007f4c02af2d48:   00000007e0002420 0000000000000000
0x00007f4c02af2d58:   00007f4c02af2de0 0000000000000000
0x00007f4c02af2d68:   0000000001313800 00007f4c02af2dc0
0x00007f4c02af2d78:   00007f4c2e2316c6 0000000302bd6fb0
0x00007f4c02af2d88:   0000000000000000 00007f4c02af2e08
0x00007f4c02af2d98:   00007f4c2e1bc8e1 00007f4c02af2db8
0x00007f4c02af2da8:   00007f4c2e1bc8e1 00000002f8190fd8
0x00007f4c02af2db8:   00000002f817b620 00007f4c02af2e28
0x00007f4c02af2dc8:   00007f4c2e1bc233 00000007e1ec4e9f
0x00007f4c02af2dd8:   00007f4c2e1bc233 0000000302bd6fb0
0x00007f4c02af2de8:   00007f4c02af2de8 00000007e1b7a5db
0x00007f4c02af2df8:   00007f4c02af2e30 00000007e37575a8
0x00007f4c02af2e08:   0000000000000000 00000007e1b7a5e8
0x00007f4c02af2e18:   00007f4c02af2de0 00007f4c02af2e40
0x00007f4c02af2e28:   00007f4c02af2ed0 00007f4c2edaaf34
0x00007f4c02af2e38:   00007f4c2edaaf34 00007f4c0000000a
0x00007f4c02af2e48:   00000007e416ec20 0000000000000000
0x00007f4c02af2e58:   00000007e416e2b8 00007f4c02af2ed0
0x00007f4c02af2e68:   00007f4c2e1bc233 00007f4c02af2ed0
0x00007f4c02af2e78:   00007f4c2e1bc233 000000000000000a 

Instructions: (pc=0x00007f4c3230aad7)
0x00007f4c3230aab7:   85 60 02 00 00 06 00 00 00 4c 89 6d b0 4c 8b 23
0x00007f4c3230aac7:   4d 85 e4 0f 84 68 02 00 00 49 8b 9d 20 01 00 00
0x00007f4c3230aad7:   48 83 7b 10 f7 0f 87 6e 02 00 00 48 8b 43 10 48
0x00007f4c3230aae7:   8d 50 08 48 3b 53 18 0f 87 9c 02 00 00 48 89 53 

Register to memory mapping:

RAX=0x0000000000010000 is an unknown value
RBX=0x80000001000004db is an unknown value
RCX=0x0000000302bd6fb0 is an oop
[Lcom.mycompany.MyClass$Type; 
 - klass: 'com/mycompany/MyClass$Type'[]
 - length: 16
RDX=0x00007f4c32acda50: <offset 0xe37a50> in /usr/java/jre64-1.7.0_45/jre/lib/amd64/server/libjvm.so at 0x00007f4c31c96000
RSP=0x00007f4c02af2c88 is pointing into the stack for thread: 0x0000000001313800
RBP=0x00007f4c02af2d70 is pointing into the stack for thread: 0x0000000001313800
RSI=0x00007f4c02af2d80 is pointing into the stack for thread: 0x0000000001313800
RDI=0x00000000013139e8 is an unknown value
R8 =0x0000000000000001 is an unknown value
R9 =
[error occurred during error reporting (printing register info), id 0xb]

Stack: [0x00007f4c029f7000,0x00007f4c02af8000],  sp=0x00007f4c02af2c88,  free space=1007k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x674ad7]  JVM_Clone+0x97
J  java.lang.Object.clone()Ljava/lang/Object;
j  com.mycompany.MyClass$Type.values()[Lcom/mycompany/MyClass$Type;+3

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J  java.lang.Object.clone()Ljava/lang/Object;
j  com.mycompany.MyClass$Type.values()[Lcom/mycompany/MyClass$Type;+3

#
#Java运行时环境检测到一个致命错误：
#
#pc=0x00007f4c3230aad7、pid=16248、tid=139964439296320时的SIGSEGV（0xb）
#
#JRE版本：Java（TM）SE运行时环境（7.0_45-b18）（build 1.7.0_45-b18）
#Java虚拟机：Java热点（TM）64位服务器虚拟机（24.45-b08混合模式linux-amd64压缩oops）
#有问题的框架：
#V[libjvm.so+0x674ad7]JVM_Clone+0x97
#
#无法写入核心转储。核心转储已被禁用。要启用核心转储，请在再次启动Java之前尝试“ulimit-c unlimited”
#
#如果您想提交错误报告，请访问：
#   http://bugreport.sun.com/bugreport/crash.jsp
#崩溃发生在Java虚拟机外部的本机代码中。
#有关报告错误的位置，请参见问题框。
#
---------------R E A D---------------
当前线程（0x0000000001313800）：JavaThread“http-bio-14080-exec-17”守护程序[_thread_in_native，id=18943，堆栈（0x00007f4c029f7000,0x00007f4c02af8000）]
siginfo:si_signo=SIGSEGV:si_errno=0，si_code=128（），si_addr=0x0000000000000000
登记册：
RAX=0x0000000000010000，RBX=0x800000100004DB，RCX=0x0000000302bd6fb0，RDX=0x00007f4c32acda50
RSP=0x00007f4c02af2c88，RBP=0x00007F4C02AF2AD70，RSI=0x00007F4C02AF2AD80，RDI=0x00000000013139e8
R8=0x0000000000000001，R9=0x00000002f8191228，R10=0x00007f4c2e231658，R11=0x00007f4c2e231638
R12=0x0000000302bd6fb0，R13=0x00000007e0002420，R14=0x0000000000000000，R15=0x0000000001313800
RIP=0x00007f4c3230aad7，EFLAGS=0x000000000001022，CSGSFS=0x0000000000000033，ERR=0x0000000000000000
TRAPNO=0x000000000000000d
堆栈顶部：（sp=0x00007f4c02af2c88）
0x00007f4c02af2c88:00007f4c3230aa63 00000000 129196D1
0x00007f4c02af2c98:0000000 2F817B620 000000000 13139E8
0x00007f4c02af2ca8:00007F4C02AF2AD18 0000000 2F819180
0x00007f4c02af2cb8:000000000000000 9 000000000 1313800
0x00007f4c02af2cc8:000000000000000000000000000 1313800
0x00007f4c02af2cd8:00000000 138F5509 0000000000000000
0x00007f4c02af2ce8:00007f4c2e382ae8 0000000000000000
0x00007f4c02af2cf8:00007f4c02af2cf8 0000000 2F817B620
0x00007F4C02AF2AD08:0000000 2F8190FD8 0000000 7E01AB278
0x00007F4C02AF2AD18:0000000 2F8190EC0 0000000 7E0002420
0x00007F4C02AF2AD28:0000000000000000000000000 7E0002420
0x00007F4C02AF2AD38:0000000000000000 0000000000000000
0x00007F4C02AF2AD48:0000000 7E0002420 0000000000000000
0x00007F4C02AF2AD58:00007F4C02AF2AD0 0000000000000000
0x00007F4C02AF2AD68:000000000 1313800 00007F4C02AF2ADC0
0x00007F4C02AF2AD78:00007F4C2E2316C60000000 302BD6FB0
0x00007F4C02AF2AD88:0000000000000000 00007F4C02AF2AE08
0x00007F4C02AF2AD98:00007f4c2e1bc8e1 00007F4C02AF2AD8
0x00007F4C02AF2AF2DA8:00007f4c2e1bc8e1 0000000 2F8190FD8
0x00007f4c02af2db8:0000000 2F817B620 00007f4c02af2e28
0x00007f4c02af2dc8:00007F4C2E1BC2330000000 7E1EC4E9F
0x00007F4C02AF2ADD8:00007F4C2E1BC2330000000 302BD6FB0
0x00007F4C02AF2ADE8:00007F4C02AF2ADE8 0000000 7E1B7A5DB
0x00007f4c02af2df8:00007f4c02af2e30 0000000 7E37575A8
0x00007F4C02AF2AE08:0000000000000000000000000 7E1B7A5E8
0x00007F4C02AF2AE18:00007F4C02AF2AF2DE0 00007F4C02AF2AE40
0x00007F4C02AF2AE28:00007F4C02AF2AED0 00007F4C2AEDAF34
0x00007F4C02AF2AE38:00007f4c2edaaf34 00007F4C000000A
0x00007F4C02AF2AE48:0000000 7E416EC20 0000000000000000
0x00007F4C02AF2AE58:0000000 7E416E2B8 00007F4C02AF2AED0
0x00007F4C02AF2AF2E68:00007f4c2e1bc233 00007F4C02AF2AED0
0x00007f4c02af2e78:00007F4C2E1BC233000000000000000 A
说明：（pc=0x00007f4c3230aad7）
0x00007F4C230AAB7:85 60 02 00 06 00 00 4c 89 6d b0 4c 8b 23
0x00007f4c3230aac7:4d 85 e4 0f 84 68 02 00 49 8b 9d 20 01 00
0x00007f4c3230aad7:48 83 7b 10 f7 0f 87 6e 02 00 48 8b 43 10 48
0x00007f4c3230aae7:8d 50 08 48 3b 53 18 0f 87 9c 02 00 48 89 53
寄存器到内存映射：
RAX=0x0000000000010000是未知值
RBX=0x800000100004dB是未知值
RCX=0x0000000302bd6fb0是一个oop
[Lcom.mycompany.MyClass$类型；
-klass:'com/mycompany/MyClass$Type'[]
-长度：16
RDX=0x00007f4c32acda50:in/usr/java/jre64-1.7.0_45/jre/lib/amd64/server/libjvm.so在0x00007f4c31c96000处
RSP=0x00007f4c02af2c88指向线程0x0000000001313800的堆栈
RBP=0x00007F4C02AF2AD70指向线程0x0000000001313800的堆栈
RSI=0x00007F4C02AF2AD80指向线程0x0000000001313800的堆栈
RDI=0x00000000013139e8是未知值
R8=0x0000000000000001是未知值
R9=
[错误报告期间发生错误（打印注册表信息），id 0xb]
堆栈：[0x00007f4c029f7000,0x00007f4c02af8000]，sp=0x00007f4c02af2c88，可用空间=1007k
本机框架：（J=编译的Java代码，J=解释的，Vv=虚拟机代码，C=本机代码）
V[libjvm.so+0x674ad7]JVM_Clone+0x97
J java.lang.Object.clone（）Ljava/lang/Object；
j com.mycompany.MyClass$Type.values（）[Lcom/mycompany/MyClass$Type；+3
Java框架：（J=编译的Java代码，J=解释的，Vv=虚拟机代码）
J java.lang.Object.clone（）Ljava/lang/Object；
j com.mycompany.MyClass$Type.values（）[Lcom/mycompany/MyClass$Type；+3

/var/log/messages输出-

myhostname kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
myhostname kernel: java cpuset=/ mems_allowed=0
myhostname kernel: Pid: 32307, comm: java Not tainted 2.6.39-400.209.1.el5uek #1
myhostname kernel: Call Trace:
myhostname kernel:  [<ffffffff811136b4>] dump_header+0x94/0xe0
myhostname kernel:  [<ffffffff811137fd>] oom_kill_process+0x6d/0x160
myhostname kernel:  [<ffffffff811139ec>] out_of_memory+0xfc/0x210
myhostname kernel:  [<ffffffff811187ec>] __alloc_pages_slowpath+0x64c/0x660
myhostname kernel:  [<ffffffff811189b4>] __alloc_pages_nodemask+0x1b4/0x200
myhostname kernel:  [<ffffffff8111b140>] ? __do_page_cache_readahead+0xe0/0x170
myhostname kernel:  [<ffffffff81150893>] alloc_pages_current+0xb3/0x120
myhostname kernel:  [<ffffffff811100da>] __page_cache_alloc+0x9a/0xb0
myhostname kernel:  [<ffffffff8111097f>] page_cache_read+0x4f/0xb0
myhostname kernel:  [<ffffffff81111a54>] filemap_fault+0x174/0x270
myhostname kernel:  [<ffffffff81137a2c>] __do_fault+0x5c/0x550
myhostname kernel:  [<ffffffff81137fc6>] do_linear_fault+0x36/0x40
myhostname kernel:  [<ffffffff81510b6e>] ? call_function_interrupt+0xe/0x20
myhostname kernel:  [<ffffffff81138044>] handle_pte_fault+0x74/0x190
myhostname kernel:  [<ffffffff815106ae>] ? apic_timer_interrupt+0xe/0x20
myhostname kernel:  [<ffffffff8113828f>] handle_mm_fault+0x12f/0x1b0
myhostname kernel:  [<ffffffff8150b1cd>] do_page_fault+0x17d/0x4b0
myhostname kernel:  [<ffffffff8117b821>] ? user_path_at+0x11/0x20
myhostname kernel:  [<ffffffff81170516>] ? vfs_fstatat+0x56/0x90
myhostname kernel:  [<ffffffff8117067b>] ? vfs_stat+0x1b/0x20
myhostname kernel:  [<ffffffff81507cd5>] page_fault+0x25/0x30
myhostname kernel: Mem-Info:
myhostname kernel: Node 0 DMA per-cpu:
myhostname kernel: CPU    0: hi:    0, btch:   1 usd:   0
myhostname kernel: CPU    1: hi:    0, btch:   1 usd:   0
myhostname kernel: CPU    2: hi:    0, btch:   1 usd:   0
myhostname kernel: CPU    3: hi:    0, btch:   1 usd:   0
myhostname kernel: CPU    4: hi:    0, btch:   1 usd:   0
myhostname kernel: CPU    5: hi:    0, btch:   1 usd:   0
myhostname kernel: Node 0 DMA32 per-cpu:
myhostname kernel: CPU    0: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    1: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    2: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    3: hi:  186, btch:  31 usd:  11
myhostname kernel: CPU    4: hi:  186, btch:  31 usd:  30
myhostname kernel: CPU    5: hi:  186, btch:  31 usd:   0
myhostname kernel: Node 0 Normal per-cpu:
myhostname kernel: CPU    0: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    1: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    2: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    3: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    4: hi:  186, btch:  31 usd:   0
myhostname kernel: CPU    5: hi:  186, btch:  31 usd:   0
myhostname kernel: active_anon:4372468 inactive_anon:275213 isolated_anon:0
myhostname kernel:  active_file:17 inactive_file:21 isolated_file:0
myhostname kernel:  unevictable:6002 dirty:24 writeback:4 unstable:0
myhostname kernel:  free:38369 slab_reclaimable:3708 slab_unreclaimable:9265
myhostname kernel:  mapped:1253 shmem:67 pagetables:10880 bounce:0
myhostname kernel: Node 0 DMA free:15880kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15688kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
myhostname kernel: lowmem_reserve[]: 0 3000 32290 32290
myhostname kernel: Node 0 DMA32 free:119288kB min:2136kB low:2668kB high:3204kB active_anon:30224kB inactive_anon:9152kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:16kB writeback:4kB mapped:36kB shmem:0kB slab_reclaimable:40kB slab_unreclaimable:232kB kernel_stack:24kB pagetables:48kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:67 all_unreclaimable? yes
myhostname kernel: lowmem_reserve[]: 0 0 29290 29290
myhostname kernel: Node 0 Normal free:17564kB min:20856kB low:26068kB high:31284kB active_anon:17459648kB inactive_anon:1091700kB active_file:120kB inactive_file:88kB unevictable:24008kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:24008kB dirty:80kB writeback:12kB mapped:4976kB shmem:268kB slab_reclaimable:14792kB slab_unreclaimable:36828kB kernel_stack:2768kB pagetables:43472kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:14972 all_unreclaimable? yes
myhostname kernel: lowmem_reserve[]: 0 0 0 0
myhostname kernel: Node 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15880kB
myhostname kernel: Node 0 DMA32: 458*4kB 494*8kB 284*16kB 55*32kB 7*64kB 4*128kB 5*256kB 7*512kB 7*1024kB 6*2048kB 20*4096kB = 119288kB
myhostname kernel: Node 0 Normal: 3333*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 17428kB
myhostname kernel: 115783 total pagecache pages
myhostname kernel: 114466 pages in swap cache
myhostname kernel: Swap cache stats: add 2183060, delete 2068594, find 1860109/1985849
myhostname kernel: Free swap  = 0kB
myhostname kernel: Total swap = 2097148kB
myhostname kernel: 8388592 pages RAM
myhostname kernel: 134806 pages reserved
myhostname kernel: 6845 pages shared
myhostname kernel: 8210059 pages non-shared

myhostname内核：java调用的oom killer:gfp\u mask=0x201da，order=0，oom\u adj=0，oom\u score\u adj=0
myhostname内核：java cpuset=/mems\u allowed=0
myhostname内核：Pid:32307，comm:java未受污染2.6.39-400.209.1.el5uek#1
myhostname内核：调用跟踪：
myhostname内核：[]转储头+0x94/0xe0
myhostname内核：[]oom\u kill\u进程+0x6d/0x160
myhostname内核：[]内存不足+0xfc/0x210
myhostname内核：[]所有页面慢路径+0x64c/0x660
myhostname内核：[]所有页面节点掩码+0x1b4/0x200
MyKern