devbox
你好赵伟
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Java堆内存溢出造成OS卡顿/服务中断的一种情况_java.util.concurrent.timeoutexception: waited 3000

作者：你好赵伟 | 2024-03-16 11:50:54

赞

踩

java.util.concurrent.timeoutexception: waited 3000 milliseconds

前提

top看内存情况

在这里插入图片描述

目标

测试内存临界情况下，内存溢出对已运行Java服务的影响

过程

1、制造麻烦

public static void main(String[] argv)  {
        Thread.setDefaultUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {
            @Override
            public void uncaughtException(Thread t, Throwable e) {
                System.out.println("出错啦");
                e.printStackTrace();
            }
        });
        new Thread(new Runnable() {
            @Override
            public void run() {
                test_outofmem();
            }
        }).start();
    }
    public static void test_outofmem()  {
        List list = new ArrayList<>();
        for(int i = 0; i < 10000; i++) {
            try {
                Thread.sleep(100);
            } catch (InterruptedException e) {
            }
            list.add(new int[1024*1024*4]);
        }
    }

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25

2、人为产生坏影响

可能是两种情况：
在这里插入图片描述
或者
在这里插入图片描述

3、查看内存

在这里插入图片描述

4、表现

1、系统出现一次卡顿；
2、再次看系统内存情况就这样子的，内存多出来2G

3、服务之间访问超时，主要表现在nacos的grpc通信超时，造成各个服务状态变为UNHEALTHY。(3000ms是写在nacos的代码中的）

[com.alibaba.nacos.client.remote.worker] ERROR c.a.n.c.r.c.g.GrpcClient - Server check fail, please check server localhost ,port 9848 is available , error ={}
java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 59165 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@77129150[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@3e2b9293, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@5aa48400, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@4951cc97}}}}}]]
	at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
	at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.serverCheck(GrpcClient.java:146)
	at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.connectToServer(GrpcClient.java:268)

  • 1
  • 2
  • 3
  • 4
  • 5

4、已建立连接的socket通信中断

ERROR c.n.s.s.SocketServerConsumer - 读输入流时发生错误
java.net.SocketException: Socket closed
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)

  • 1
  • 2
  • 3
  • 4
  • 5

5、如果代码强壮，经过短暂停顿后，服务恢复正常

初步分析

查看系统日志：

sudo tail -f /var/log/messages

  • 1
Oct 14 15:27:31 localhost kernel: [ 1853]     0  1853   480464    29007  3391488   324668             0 packagekitd

...

Oct 14 15:28:09 localhost kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/packagekit.service,task=packagekitd,pid=1853,uid=0
Oct 14 15:28:09 localhost kernel: Out of memory: Killed process 1853 (packagekitd) total-vm:1921856kB, anon-rss:116028kB, file-rss:0kB, shmem-rss:0kB, UID:0
Oct 14 15:28:10 localhost systemd[1]: packagekit.service: Main process exited, code=killed, status=9/KILL
Oct 14 15:28:11 localhost systemd[1]: packagekit.service: Failed with result 'signal'.

...

Oct 14 16:08:29 localhost kernel: [2046832]  1002 2046832     1883       16    57344        0             0 tail
Oct 14 16:08:29 localhost kernel: [2048057]  1002 2048057   757176   275803  2437120        0             0 java
Oct 14 16:08:29 localhost kernel: [2048083]  1002 2048083   757176   275107  2445312        0             0 java
Oct 14 16:08:29 localhost kernel: [2048128]  1002 2048128   757176   132692  1363968        0             0 java

...

6:08:29 localhost kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1002.slice/session-160.scope,task=java,pid=2048057,uid=1002
Oct 14 16:08:29 localhost kernel: Out of memory: Killed process 2048057 (java) total-vm:3028704kB, anon-rss:1103212kB, file-rss:0kB, shmem-rss:0kB, UID:1002
Oct 14 16:08:30 localhost kernel: oom_reaper: reaped process 2048057 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21

第1次发生oom-kill时，packagekitd占用内存是最高的，历时38秒
第2次oom-kill，2048057 的pgtables_bytes不算最高，但是是高的，历时…没记录

再次启动三个java -XX:+PrintGC -Xms1024m -Xmx1024m -XX:+UseG1GC OutOfHeap
内存严重的不够用，查看message日志，看到连续杀死了两个占用pgtables_bytes高的Java进程。
这个过程经历了45秒!

Oct 14 17:10:11 localhost kernel: java invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Oct 14 17:10:15 localhost kernel: CPU: 0 PID: 2063149 Comm: java Kdump: loaded Tainted: G                 ---------r-  - 4.18.0-240.el8.x86_64 #1
Oct 14 17:10:15 localhost kernel: Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Oct 14 17:10:15 localhost kernel: Call Trace:
Oct 14 17:10:22 localhost kernel: dump_stack+0x5c/0x80
Oct 14 17:10:24 localhost kernel: dump_header+0x51/0x308
Oct 14 17:10:25 localhost kernel: ? try_to_free_pages+0xe8/0x1c0
Oct 14 17:10:25 localhost kernel: oom_kill_process.cold.28+0xb/0x10
Oct 14 17:10:25 localhost kernel: out_of_memory+0x1c1/0x4b0
Oct 14 17:10:26 localhost kernel: __alloc_pages_slowpath+0xc24/0xd40
Oct 14 17:10:26 localhost kernel: __alloc_pages_nodemask+0x245/0x280
Oct 14 17:10:26 localhost kernel: filemap_fault+0x3b8/0x840
Oct 14 17:10:27 localhost kernel: ? update_load_avg+0x7e/0x5e0
Oct 14 17:10:27 localhost kernel: ? account_entity_enqueue+0x9c/0xd0
Oct 14 17:10:27 localhost kernel: ? page_add_file_rmap+0x15/0x170
Oct 14 17:10:27 localhost kernel: ? alloc_set_pte+0x203/0x480
Oct 14 17:10:28 localhost kernel: ? _cond_resched+0x15/0x30
Oct 14 17:10:28 localhost kernel: __xfs_filemap_fault+0x6d/0x200 [xfs]
Oct 14 17:10:28 localhost kernel: __do_fault+0x38/0xc0
Oct 14 17:10:28 localhost kernel: do_fault+0x191/0x3c0
Oct 14 17:10:28 localhost kernel: __handle_mm_fault+0x3e6/0x7c0
Oct 14 17:10:30 localhost kernel: handle_mm_fault+0xc2/0x1d0
Oct 14 17:10:30 localhost kernel: __do_page_fault+0x21b/0x4d0
Oct 14 17:10:30 localhost kernel: do_page_fault+0x32/0x110
Oct 14 17:10:30 localhost kernel: ? page_fault+0x8/0x30
Oct 14 17:10:30 localhost kernel: page_fault+0x1e/0x30
Oct 14 17:10:30 localhost kernel: RIP: 0033:0x7fb619520759
Oct 14 17:10:30 localhost kernel: Code: Bad RIP value.
Oct 14 17:10:30 localhost kernel: RSP: 002b:00007fb5d4b98c00 EFLAGS: 00010207
Oct 14 17:10:31 localhost kernel: RAX: 00007fb5d4a99000 RBX: 00007fb619ba0160 RCX: 00007fb619cf0967
Oct 14 17:10:31 localhost kernel: RDX: 0000000000000000 RSI: 0000000000003000 RDI: 00007fb5d4a99000
Oct 14 17:10:31 localhost kernel: RBP: 00007fb5d4b98c00 R08: 00000000ffffffff R09: 0000000000000000
Oct 14 17:10:31 localhost kernel: R10: 0000000000004032 R11: 0000000000000206 R12: 0000000000000003
Oct 14 17:10:31 localhost kernel: R13: 00007fb5d4a99000 R14: 0000000000003000 R15: 00007fb5e4d5a000
Oct 14 17:10:31 localhost kernel: Mem-Info:
Oct 14 17:10:32 localhost kernel: active_anon:2181877 inactive_anon:384255 isolated_anon:0#012 active_file:1367 inactive_file:2024 isolated_file:34#012 unevictable:0 dirty:1215 writeback:4 unstable:0#012 slab_reclaimable:12550 slab_unreclaimable:28201#012 mapped:461 shmem:414 pagetables:22586 bounce:0#012 free:27865 free_pcp:30 free_cma:0
Oct 14 17:10:32 localhost kernel: Node 0 active_anon:8727508kB inactive_anon:1537020kB active_file:5468kB inactive_file:8096kB unevictable:0kB isolated(anon):0kB isolated(file):136kB mapped:1844kB dirty:4860kB writeback:16kB shmem:1656kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 5519360kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Oct 14 17:10:32 localhost kernel: Node 0 DMA free:15356kB min:96kB low:120kB high:144kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct 14 17:10:32 localhost kernel: lowmem_reserve[]: 0 3258 10425 10425 10425
Oct 14 17:10:33 localhost kernel: Node 0 DMA32 free:49724kB min:21096kB low:26368kB high:31640kB active_anon:2559556kB inactive_anon:759972kB active_file:928kB inactive_file:1244kB unevictable:0kB writepending:732kB present:3653568kB managed:3391424kB mlocked:0kB kernel_stack:3056kB pagetables:7116kB bounce:0kB free_pcp:120kB local_pcp:120kB free_cma:0kB
Oct 14 17:10:33 localhost kernel: lowmem_reserve[]: 0 0 7166 7166 7166
Oct 14 17:10:33 localhost kernel: Node 0 Normal free:46380kB min:46388kB low:57984kB high:69580kB active_anon:6167920kB inactive_anon:777044kB active_file:4504kB inactive_file:6808kB unevictable:0kB writepending:4144kB present:7560192kB managed:7344720kB mlocked:0kB kernel_stack:24400kB pagetables:83228kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct 14 17:10:34 localhost kernel: lowmem_reserve[]: 0 0 0 0 0
Oct 14 17:10:34 localhost kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 0*1024kB 1*2048kB (M) 3*4096kB (M) = 15356kB
Oct 14 17:10:34 localhost kernel: Node 0 DMA32: 37*4kB (UE) 253*8kB (UME) 446*16kB (UE) 247*32kB (UE) 162*64kB (UE) 63*128kB (UE) 21*256kB (UME) 9*512kB (UME) 4*1024kB (UM) 0*2048kB 0*4096kB = 49724kB
Oct 14 17:10:34 localhost kernel: Node 0 Normal: 610*4kB (UME) 699*8kB (UME) 860*16kB (UME) 192*32kB (UME) 79*64kB (UME) 7*128kB (UME) 1*256kB (U) 0*512kB 12*1024kB (M) 0*2048kB 0*4096kB = 46432kB
Oct 14 17:10:34 localhost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 14 17:10:35 localhost kernel: 30324 total pagecache pages
Oct 14 17:10:35 localhost kernel: 26483 pages in swap cache
Oct 14 17:10:35 localhost kernel: Swap cache stats: add 7058695, delete 7032212, find 2830039/3852094
Oct 14 17:10:35 localhost kernel: Free swap  = 0kB
Oct 14 17:10:35 localhost kernel: Total swap = 4579324kB
Oct 14 17:10:35 localhost kernel: 2807438 pages RAM
Oct 14 17:10:36 localhost kernel: 0 pages HighMem/MovableOnly
Oct 14 17:10:36 localhost kernel: 119562 pages reserved
Oct 14 17:10:36 localhost kernel: 0 pages hwpoisoned
Oct 14 17:10:36 localhost kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Oct 14 17:10:36 localhost kernel: [  671]     0   671    36015      477   311296      536             0 systemd-journal
...
Oct 14 17:10:52 localhost kernel: [2053212]  1002 2053212     1924       16    57344        0             0 tail
Oct 14 17:10:52 localhost kernel: [2060331]  1002 2060331   664237   101251  1667072        0             0 java
Oct 14 17:10:52 localhost kernel: [2061177]  1002 2061177   757176   277190  2449408        0             0 java
Oct 14 17:10:52 localhost kernel: [2061649]     0 2061649    51353     5263   438272      236             0 sssd_kcm
Oct 14 17:10:52 localhost kernel: [2061840]     0 2061840    52182      426   413696        0             0 sudo
Oct 14 17:10:52 localhost kernel: [2062505]  1002 2062505     1883       16    61440        0             0 tail
Oct 14 17:10:52 localhost kernel: [2063006]     0 2063006     1883       16    53248        0             0 tail
Oct 14 17:10:52 localhost kernel: [2063011]     0 2063011     1874       16    57344        0             0 sleep
Oct 14 17:10:52 localhost kernel: [2063075]  1002 2063075  1148860   144650  1376256        0             0 java
Oct 14 17:10:52 localhost kernel: [2063095]  1002 2063095   757176   141674  1445888        0             0 java
Oct 14 17:10:52 localhost kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1002.slice/session-160.scope,task=java,pid=2061177,uid=1002
Oct 14 17:10:52 localhost kernel: Out of memory: Killed process 2061177 (java) total-vm:3028704kB, anon-rss:1108760kB, file-rss:0kB, shmem-rss:0kB, UID:1002
Oct 14 17:10:52 localhost kernel: oom_reaper: reaped process 2061177 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Oct 14 17:10:52 localhost kernel: gmain invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Oct 14 17:10:52 localhost kernel: CPU: 0 PID: 1816 Comm: gmain Kdump: loaded Tainted: G                 ---------r-  - 4.18.0-240.el8.x86_64 #1
Oct 14 17:10:52 localhost kernel: Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Oct 14 17:10:52 localhost kernel: Call Trace:
Oct 14 17:10:52 localhost kernel: dump_stack+0x5c/0x80
Oct 14 17:10:52 localhost kernel: dump_header+0x51/0x308
Oct 14 17:10:52 localhost kernel: ? try_to_free_pages+0xe8/0x1c0
Oct 14 17:10:52 localhost kernel: oom_kill_process.cold.28+0xb/0x10
Oct 14 17:10:52 localhost kernel: out_of_memory+0x1c1/0x4b0
Oct 14 17:10:52 localhost kernel: __alloc_pages_slowpath+0xc24/0xd40
Oct 14 17:10:53 localhost kernel: __alloc_pages_nodemask+0x245/0x280
Oct 14 17:10:53 localhost kernel: filemap_fault+0x3b8/0x840
Oct 14 17:10:53 localhost kernel: ? __mod_lruvec_state+0x44/0x110
Oct 14 17:10:53 localhost kernel: ? page_add_file_rmap+0x103/0x170
Oct 14 17:10:53 localhost kernel: ? pmd_devmap_trans_unstable+0x2a/0x40
Oct 14 17:10:53 localhost kernel: ? alloc_set_pte+0x38a/0x480
Oct 14 17:10:53 localhost kernel: ? _cond_resched+0x15/0x30
Oct 14 17:10:53 localhost kernel: __xfs_filemap_fault+0x6d/0x200 [xfs]
Oct 14 17:10:53 localhost kernel: __do_fault+0x38/0xc0
Oct 14 17:10:53 localhost kernel: do_fault+0x191/0x3c0
Oct 14 17:10:53 localhost kernel: __handle_mm_fault+0x3e6/0x7c0
Oct 14 17:10:53 localhost kernel: handle_mm_fault+0xc2/0x1d0
Oct 14 17:10:53 localhost kernel: __do_page_fault+0x21b/0x4d0
Oct 14 17:10:53 localhost kernel: do_page_fault+0x32/0x110
Oct 14 17:10:53 localhost kernel: ? page_fault+0x8/0x30
Oct 14 17:10:53 localhost kernel: page_fault+0x1e/0x30
Oct 14 17:10:53 localhost kernel: RIP: 0033:0x7fa3a26da770
Oct 14 17:10:53 localhost kernel: Code: Bad RIP value.
Oct 14 17:10:53 localhost kernel: RSP: 002b:00007fa39da99a88 EFLAGS: 00010246
Oct 14 17:10:53 localhost kernel: RAX: 0000000000000000 RBX: 000055e9ce97d620 RCX: 000055e9ce9690c0
Oct 14 17:10:53 localhost kernel: RDX: 00000387eb3c2073 RSI: 000055e9ce985510 RDI: 0000000000000000
Oct 14 17:10:53 localhost kernel: RBP: 0000000000000000 R08: 0000000000000001 R09: 000055e9ce97d6a8
Oct 14 17:10:53 localhost kernel: R10: 000055e9ce96b600 R11: 0000000000000000 R12: 00007fa39da99ab0
Oct 14 17:10:53 localhost kernel: R13: 000000007fffffff R14: 00007fa39da99aa8 R15: 0000000000000000
Oct 14 17:10:53 localhost kernel: Mem-Info:
Oct 14 17:10:53 localhost kernel: active_anon:2184023 inactive_anon:384007 isolated_anon:0#012 active_file:643 inactive_file:687 isolated_file:53#012 unevictable:0 dirty:579 writeback:16 unstable:0#012 slab_reclaimable:12565 slab_unreclaimable:28176#012 mapped:432 shmem:439 pagetables:22557 bounce:0#012 free:27858 free_pcp:75 free_cma:0
Oct 14 17:10:53 localhost kernel: Node 0 active_anon:8736092kB inactive_anon:1536028kB active_file:2572kB inactive_file:2748kB unevictable:0kB isolated(anon):0kB isolated(file):212kB mapped:1728kB dirty:2316kB writeback:64kB shmem:1756kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 5492736kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Oct 14 17:10:53 localhost kernel: Node 0 DMA free:15356kB min:96kB low:120kB high:144kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct 14 17:10:53 localhost kernel: lowmem_reserve[]: 0 3258 10425 10425 10425
Oct 14 17:10:53 localhost kernel: Node 0 DMA32 free:49692kB min:21096kB low:26368kB high:31640kB active_anon:2580088kB inactive_anon:738892kB active_file:1400kB inactive_file:1036kB unevictable:0kB writepending:1440kB present:3653568kB managed:3391424kB mlocked:0kB kernel_stack:3204kB pagetables:7104kB bounce:0kB free_pcp:208kB local_pcp:208kB free_cma:0kB
Oct 14 17:10:53 localhost kernel: lowmem_reserve[]: 0 0 7166 7166 7166
Oct 14 17:10:53 localhost kernel: Node 0 Normal free:46384kB min:46388kB low:57984kB high:69580kB active_anon:6155988kB inactive_anon:797100kB active_file:1156kB inactive_file:1712kB unevictable:0kB writepending:888kB present:7560192kB managed:7344720kB mlocked:0kB kernel_stack:24124kB pagetables:83124kB bounce:0kB free_pcp:92kB local_pcp:92kB free_cma:0kB
Oct 14 17:10:53 localhost kernel: lowmem_reserve[]: 0 0 0 0 0
Oct 14 17:10:53 localhost kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 0*1024kB 1*2048kB (M) 3*4096kB (M) = 15356kB
Oct 14 17:10:53 localhost kernel: Node 0 DMA32: 41*4kB (UME) 219*8kB (UE) 446*16kB (UE) 246*32kB (UE) 162*64kB (UE) 63*128kB (UE) 20*256kB (UE) 8*512kB (UE) 5*1024kB (UM) 0*2048kB 0*4096kB = 49692kB
Oct 14 17:10:53 localhost kernel: Node 0 Normal: 332*4kB (UME) 730*8kB (UME) 925*16kB (UME) 181*32kB (UME) 75*64kB (UE) 6*128kB (UME) 1*256kB (U) 1*512kB (M) 12*1024kB (M) 0*2048kB 0*4096kB = 46384kB
Oct 14 17:10:53 localhost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 14 17:10:53 localhost kernel: 29031 total pagecache pages
Oct 14 17:10:53 localhost kernel: 27194 pages in swap cache
Oct 14 17:10:53 localhost kernel: Swap cache stats: add 7061377, delete 7034183, find 2830505/3853548
Oct 14 17:10:53 localhost kernel: Free swap  = 0kB
Oct 14 17:10:53 localhost kernel: Total swap = 4579324kB
Oct 14 17:10:53 localhost kernel: 2807438 pages RAM
Oct 14 17:10:53 localhost kernel: 0 pages HighMem/MovableOnly
Oct 14 17:10:53 localhost kernel: 119562 pages reserved
Oct 14 17:10:53 localhost kernel: 0 pages hwpoisoned
Oct 14 17:10:53 localhost kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Oct 14 17:10:53 localhost kernel: [  671]     0   671    36015      505   311296      534             0 systemd-journal
Oct 14 17:10:53 localhost kernel: [  707]     0   707    30291       19   245760     1146         -1000 systemd-udevd
...
Oct 14 17:10:56 localhost kernel: [2063075]  1002 2063075  1148860   304952  2662400        0             0 java
Oct 14 17:10:56 localhost kernel: [2063095]  1002 2063095   757176   258618  2420736        0             0 java
Oct 14 17:10:56 localhost kernel: [2063186]     0 2063186     6831      293    77824       59             0 ksmtuned
Oct 14 17:10:56 localhost kernel: [2063195]     0 2063195      255        1    36864        0             0 awk
Oct 14 17:10:56 localhost kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1002.slice/session-160.scope,task=java,pid=2063075,uid=1002
Oct 14 17:10:56 localhost kernel: Out of memory: Killed process 2063075 (java) total-vm:4595440kB, anon-rss:1219808kB, file-rss:0kB, shmem-rss:0kB, UID:1002
Oct 14 17:10:56 localhost kernel: oom_reaper: reaped process 2063075 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
^C

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  • 89
  • 90
  • 91
  • 92
  • 93
  • 94
  • 95
  • 96
  • 97
  • 98
  • 99
  • 100
  • 101
  • 102
  • 103
  • 104
  • 105
  • 106
  • 107
  • 108
  • 109
  • 110
  • 111
  • 112
  • 113
  • 114
  • 115
  • 116
  • 117
  • 118
  • 119
  • 120
  • 121
  • 122
  • 123
  • 124
  • 125
  • 126
  • 127
  • 128
  • 129
  • 130
  • 131
  • 132
  • 133
  • 134
  • 135
  • 136
  • 137
  • 138
  • 139
  • 140

操作系统只是kill掉了某个进程， 那么如果某个Java进程被kill了，只能说写的Java代码中存在问题或者访问量激增，需要限流。
只要Java代码够强壮，出现oom-killer情况，出于一些通信超时的限制，服务会出现短暂异常，内存释放，通信恢复，异常也就恢复了。

参考资料：
《Linux内核OOM killer机制》，里面有源代码，与message中打印的日志可以呼应
When Linux Runs Out of Memory

好了，我只所以做这个分析，是因为云服务器中的内存被我使用的已经就剩余100m+了，测试的虚拟机中的内存也处于不够用边缘。一次在云服务中启动新服务，OS短暂中断，其中一个服务异常不再堆外提供服务了，于是做了上面的测试和分析。

代码强壮之后，现在放心了。

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/你好赵伟/article/detail/249297
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号