Follow Excellent, Success will Chase you

0%

OOM

内核出现OOM的log分析:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
[  128.496873] min_free_kbytes invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
[ 128.505497] min_free_kbytes cpuset=/ mems_allowed=0
[ 128.510968] CPU: 0 PID: 118 Comm: min_free_kbytes Not tainted 3.10.14-00062-g277665d-dirty #322
[ 128.523967] Stack : 00000000 00000000 00000000 00000000 80624582 00000053 805b0000 805ad500
8c3933a0 805ad407 8053d15c 00000076 80623d20 805ad500 00000000 00000000
805ad500 8046cffc 805c0000 8003b3bc 80b3324c 00000000 8053eecc 890019ac
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 89001938
...
[ 128.607831] Call Trace:
[ 128.612474] [<800209e0>] show_stack+0x48/0x70
[ 128.621191] [<8046daf0>] dump_header.isra.4+0x88/0x260
[ 128.628626] [<800bd78c>] oom_kill_process+0xd0/0x478
[ 128.635875] [<800be070>] out_of_memory+0x318/0x390
[ 128.642951] [<800c1ef4>] __alloc_pages_nodemask+0x8b8/0x900
[ 128.650833] [<800d9104>] handle_pte_fault+0xae4/0xc5c
[ 128.660245] [<800d9358>] handle_mm_fault+0xdc/0x11c
[ 128.667406] [<800298d8>] do_page_fault+0x158/0x480
[ 128.674477] [<8001a784>] resume_userspace_check+0x0/0x10
[ 128.684178]
[ 128.687833] Mem-Info:
[ 128.692287] Normal per-cpu:
[ 128.699351] CPU 0: hi: 90, btch: 15 usd: 32
[ 128.706406] HighMem per-cpu:
[ 128.711486] CPU 0: hi: 90, btch: 15 usd: 15
[ 128.718564] active_anon:120239 inactive_anon:11 isolated_anon:0
[ 128.718564] active_file:5 inactive_file:2 isolated_file:0
[ 128.718564] unevictable:0 dirty:0 writeback:0 unstable:0
[ 128.718564] free:4617 slab_reclaimable:150 slab_unreclaimable:600
[ 128.718564] mapped:1 shmem:15 pagetables:150 bounce:0
[ 128.718564] free_cma:0
[ 128.754330] Normal free:18244kB min:10084kB low:12604kB high:15124kB active_anon:219220kB inactive_anon:0kB active_file
:20kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:262144kB managed:241940kB mlocked:0k
B dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:600kB slab_unreclaimable:2400kB kernel_stack:320kB pagetab
les:600kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:426 all_unreclaimable? yes
[ 128.798925] lowmem_reserve[]: 0 2048 2048
[ 128.807365] HighMem free:224kB min:256kB low:2984kB high:5716kB active_anon:261736kB inactive_anon:44kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:262144kB managed:262144kB mlocked:0kB dir
ty:0kB writeback:0kB mapped:0kB shmem:60kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB uns
table:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:31 all_unreclaimable? yes
[ 128.850710] lowmem_reserve[]: 0 0 0
[ 128.858541] Normal: 1*4kB (R) 0*8kB 0*16kB 0*32kB 1*64kB (R) 0*128kB 1*256kB (R) 1*512kB (R) 1*1024kB (R) 0*2048kB 0*40
96kB 2*8192kB (R) 0*16384kB 0*32768kB 0*65536kB = 18244kB
[ 128.896190] HighMem: 0*4kB 0*8kB 0*16kB 1*32kB (R) 1*64kB (R) 1*128kB (R) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*
8192kB 0*16384kB 0*32768kB 0*65536kB = 224kB
[ 128.928406] 22 total pagecache pages
[ 128.988524] 262144 pages RAM
[ 128.993703] 131072 pages HighMem
[ 129.001226] 136074 pages reserved
[ 129.006754] 30 pages shared
[ 129.011751] 121318 pages non-shared
[ 129.017458] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[ 129.027697] [ 61] 0 61 883 18 3 0 0 syslogd
[ 129.040259] [ 64] 0 64 883 18 3 0 0 klogd
[ 129.050561] [ 86] 1000 86 753 38 4 0 0 dbus-daemon
[ 129.061408] [ 96] 0 96 1327 70 4 0 -1000 sshd
[ 129.071623] [ 103] 0 103 899 23 5 0 0 sh
[ 129.081664] [ 104] 0 104 883 17 3 0 0 telnetd
[ 129.094214] [ 108] 0 108 753 36 4 0 0 min_free_kbytes
[ 129.105410] [ 118] 0 118 120817 120018 121 0 0 min_free_kbytes
[ 129.116617] Out of memory: Kill process 118 (min_free_kbytes) score 922 or sacrifice child
[ 129.129353] Killed process 118 (min_free_kbytes) total-vm:483268kB, anon-rss:480072kB, file-rss:0kB

相关变量说明

file-rss

1
[  129.129353] Killed process 118 (min_free_kbytes) total-vm:483268kB, anon-rss:480072kB, file-rss:0kB

rss:”Resident Set Size”, 实际驻留”在内存中”的内存数. 不包括已经交换出去的代码. 举一个例子: 如果你有一个程序使用了100K内存, 操作系统交换出40K内存, 那么RSS为60K. RSS还包括了与其它进程共享的内存区域. 这些区域通常用于libc库等.

处理流程

参考

  1. Linux中进程内存RSS与cgroup内存的RSS统计 - 差异
  2. 进程实际内存占用: 私有驻留内存数(Private RSS)介绍
  3. Out of Memory(OOM)处理流程
  4. Taming the OOM killer
-------------本文结束感谢您的阅读-------------
  • 本文作者: Winddoing
  • 本文链接: https://winddoing.github.io/post/53957.html
  • 作者声明: 本博文为个人笔记, 由于个人能力有限,难免出现错误,欢迎大家批评指正。
  • 版权声明: 本博客所有文章除特别声明外,均采用 BY-NC-SA 许可协议。转载请注明出处!