[19381.817493] python3 invoked oom-killer: gfp_mask=0x14280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=728 [19381.826966] python3 cpuset=505c48dbdddb7a6c0ab880466281ce3860e3e459f37a15d8812a844301057e7b mems_allowed=0 [19381.834681] CPU: 0 PID: 2022 Comm: python3 Not tainted 4.14.123-111.109.amzn2.x86_64 #1 [19381.841500] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 [19381.846487] Call Trace: [19381.849061] dump_stack+0x5c/0x82 [19381.852292] dump_header+0x94/0x229 [19381.855580] ? get_page_from_freelist+0x525/0xba0 [19381.859676] oom_kill_process+0x223/0x420 [19381.863348] out_of_memory+0x2af/0x4d0 [19381.866806] __alloc_pages_slowpath+0x9b6/0xd20 [19381.870889] __alloc_pages_nodemask+0x25d/0x280 [19381.874806] alloc_pages_vma+0x7c/0x1f0 [19381.878259] __handle_mm_fault+0x8cd/0x12b0 [19381.881886] handle_mm_fault+0xaa/0x1e0 [19381.885238] __do_page_fault+0x23e/0x4c0 [19381.888699] ? page_fault+0x2f/0x50 [19381.892097] page_fault+0x45/0x50 [19381.895324] RIP: 4429e880:0x7ffc4a081280 [19381.899020] RSP: 0003:0000000000000001 EFLAGS: 00000006 [19381.899036] Mem-Info: [19381.905952] active_anon:909458 inactive_anon:39 isolated_anon:0 [19381.905952] active_file:18 inactive_file:17 isolated_file:0 [19381.905952] unevictable:0 dirty:0 writeback:3 unstable:0 [19381.905952] slab_reclaimable:10464 slab_unreclaimable:14564 [19381.905952] mapped:39 shmem:140 pagetables:2963 bounce:0 [19381.905952] free:14957 free_pcp:0 free_cma:0 [19381.930372] Node 0 active_anon:3637832kB inactive_anon:156kB active_file:72kB inactive_file:68kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:156kB dirty:0kB writeback:12kB shmem:560kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [19381.950140] Node 0 DMA free:15024kB min:184kB low:228kB high:272kB active_anon:860kB inactive_anon:0kB active_file:0kB inactive_file:20kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [19381.970018] lowmem_reserve[]: 0 3714 3714 3714 [19381.973918] Node 0 DMA32 free:44804kB min:44868kB low:56084kB high:67300kB active_anon:3636788kB inactive_anon:156kB active_file:360kB inactive_file:412kB unevictable:0kB writepending:12kB present:3915776kB managed:3829972kB mlocked:0kB kernel_stack:4816kB pagetables:11852kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [19381.995869] lowmem_reserve[]: 0 0 0 0 [19381.999357] Node 0 DMA: 1*4kB (M) 0*8kB 0*16kB 2*32kB (UM) 2*64kB (U) 2*128kB (UM) 1*256kB (U) 0*512kB 2*1024kB (UM) 0*2048kB 3*4096kB (M) = 15044kB [19382.009652] Node 0 DMA32: 1767*4kB (UME) 1080*8kB (UME) 557*16kB (UME) 432*32kB (UME) 93*64kB (UME) 6*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45164kB [19382.021619] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [19382.028708] 151 total pagecache pages [19382.032312] 0 pages in swap cache [19382.035654] Swap cache stats: add 0, delete 0, find 0/0 [19382.040256] Free swap = 0kB [19382.043116] Total swap = 0kB [19382.045938] 982941 pages RAM [19382.049070] 0 pages HighMem/MovableOnly [19382.052840] 21472 pages reserved [19382.056091] 0 pages hwpoisoned [19382.059244] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [19382.066467] [ 1690] 0 1690 10291 99 25 3 0 0 systemd-journal [19382.074372] [ 1875] 0 1875 29206 67 26 3 0 0 lvmetad [19382.081856] [ 2629] 0 2629 10653 231 24 3 0 -1000 systemd-udevd [19382.089580] [ 2933] 0 2933 15562 147 29 3 0 -1000 auditd [19382.096877] [ 2994] 0 2994 24997 79 17 3 0 0 irqbalance [19382.104728] [ 3015] 0 3015 3286 27 13 3 0 0 rngd [19382.111996] [ 3058] 81 3058 14584 136 34 3 0 -900 dbus-daemon [19382.119294] [ 3089] 32 3089 18457 179 38 3 0 0 rpcbind [19382.126772] [ 3109] 0 3109 52089 162 39 3 0 0 gssproxy [19382.134388] [ 3122] 0 3122 6609 91 18 3 0 0 systemd-logind [19382.142073] [ 3157] 999 3157 30140 126 29 4 0 0 chronyd [19382.149542] [ 3426] 0 3426 26363 501 50 3 0 0 dhclient [19382.157114] [ 3515] 0 3515 26363 494 49 3 0 0 dhclient [19382.164569] [ 3683] 0 3683 22084 259 40 3 0 0 master [19382.172114] [ 3685] 89 3685 22123 253 42 3 0 0 qmgr [19382.179419] [ 3834] 0 3834 54158 254 40 3 0 0 rsyslogd [19382.186969] [ 3889] 0 3889 33259 157 19 3 0 0 crond [19382.194560] [ 3891] 0 3891 29297 27 10 3 0 0 agetty [19382.201981] [ 3901] 0 3901 29209 29 13 3 0 0 agetty [19382.208969] [ 3953] 0 3953 304259 12089 123 7 0 -999 dockerd [19382.216422] [ 4007] 0 4007 27711 251 56 3 0 -1000 sshd [19382.223794] [ 4045] 0 4045 296383 1843 63 6 0 -999 docker-containe [19382.231564] [ 4448] 0 4448 316050 11512 112 6 0 -999 kubelet [19382.238936] [ 4699] 0 4699 1878 204 8 5 0 -999 docker-containe [19382.246847] [ 4701] 0 4701 1878 182 8 5 0 -999 docker-containe [19382.254691] [ 4750] 0 4750 256 1 4 2 0 -998 pause [19382.262185] [ 4777] 0 4777 256 1 3 2 0 -998 pause [19382.269223] [ 4889] 0 4889 1878 185 8 5 0 -999 docker-containe [19382.276706] [ 4931] 0 4931 34732 1963 28 4 0 -998 kube-proxy [19382.284187] [ 5534] 0 5534 1878 146 9 5 0 -999 docker-containe [19382.292175] [ 5614] 0 5614 256 1 4 2 0 -998 pause [19382.299478] [28338] 0 28338 2230 140 9 5 0 -999 docker-containe [19382.307503] [28378] 0 28378 256 1 4 2 0 -998 pause [19382.314856] [28451] 0 28451 1878 246 9 5 0 -999 docker-containe [19382.322515] [28484] 0 28484 406 30 5 3 0 728 busybox [19382.331579] [28551] 0 28551 1458 219 6 3 0 728 nginx [19382.338903] [28552] 1000 28552 1572 313 6 3 0 728 nginx [19382.346179] [28553] 1000 28553 1572 328 6 3 0 728 nginx [19382.353515] [28554] 0 28554 7385 3784 17 3 0 728 bundle [19382.360947] [28557] 0 28557 22016 13522 46 3 0 728 bundle [19382.367891] [30170] 89 30170 22105 253 43 3 0 0 pickup [19382.375169] [32111] 0 32111 408 35 4 3 0 728 sh [19382.382225] [ 751] 0 751 133700 129589 260 4 0 728 python3 [19382.389084] [ 759] 0 759 133700 129589 258 3 0 728 python3 [19382.396597] [ 760] 0 760 133700 129589 259 3 0 728 python3 [19382.403964] [ 787] 0 787 133700 129589 260 4 0 728 python3 [19382.411256] [ 799] 0 799 133700 129589 260 4 0 728 python3 [19382.418787] [ 1198] 0 1198 407 43 5 3 0 728 sh [19382.425791] [ 1362] 0 1362 133700 129591 260 4 0 728 python3 [19382.432883] [ 1499] 0 1499 1878 193 9 5 0 -999 docker-containe [19382.440861] [ 1509] 0 1509 2230 126 9 5 0 -999 docker-containe [19382.448824] [ 1540] 0 1540 2905 54 11 3 0 999 bash [19382.455915] [ 1602] 0 1602 405 1 4 3 0 999 busybox [19382.463481] [ 1658] 0 1658 556 60 6 3 0 999 spot-monitor.sh [19382.471352] [ 1664] 0 1664 388 1 4 3 0 999 tail [19382.478198] [ 1682] 0 1682 35758 2360 32 4 0 999 aws-k8s-agent [19382.486020] [ 2021] 0 2021 388 1 4 3 0 999 sleep [19382.493191] [ 2022] 0 2022 92075 77703 159 4 0 728 python3 [19382.500626] Out of memory: Kill process 1682 (aws-k8s-agent) score 1000 or sacrifice child [19382.507562] Killed process 1682 (aws-k8s-agent) total-vm:143032kB, anon-rss:9440kB, file-rss:0kB, shmem-rss:0kB [19382.724904] oom_reaper: reaped process 1682 (aws-k8s-agent), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [19383.428541] python3 invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=728 [19383.439862] python3 cpuset=505c48dbdddb7a6c0ab880466281ce3860e3e459f37a15d8812a844301057e7b mems_allowed=0 [19383.449447] CPU: 0 PID: 2022 Comm: python3 Not tainted 4.14.123-111.109.amzn2.x86_64 #1 [19383.456449] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 [19383.461487] Call Trace: [19383.463975] dump_stack+0x5c/0x82 [19383.467049] dump_header+0x94/0x229 [19383.470399] ? get_page_from_freelist+0x525/0xba0 [19383.474590] oom_kill_process+0x223/0x420 [19383.478338] out_of_memory+0x2af/0x4d0 [19383.481896] __alloc_pages_slowpath+0x9b6/0xd20 [19383.485870] __alloc_pages_nodemask+0x25d/0x280 [19383.489898] filemap_fault+0x204/0x5f0 [19383.493587] ? page_add_file_rmap+0xd6/0x160 [19383.497576] __xfs_filemap_fault.constprop.8+0x49/0x120 [xfs] [19383.502607] __do_fault+0x57/0xd9 [19383.505730] __handle_mm_fault+0xdd2/0x12b0 [19383.509364] handle_mm_fault+0xaa/0x1e0 [19383.512884] __do_page_fault+0x23e/0x4c0 [19383.516471] ? page_fault+0x2f/0x50 [19383.519976] page_fault+0x45/0x50 [19383.523399] RIP: 4429e880:0x7ffc4a081280 [19383.527073] RSP: 0003:00005643e8c01c10 EFLAGS: 00000006 [19383.527114] Mem-Info: [19383.534186] active_anon:909531 inactive_anon:39 isolated_anon:0 [19383.534186] active_file:35 inactive_file:26 isolated_file:0 [19383.534186] unevictable:0 dirty:0 writeback:0 unstable:0 [19383.534186] slab_reclaimable:10459 slab_unreclaimable:14564 [19383.534186] mapped:41 shmem:140 pagetables:2964 bounce:0 [19383.534186] free:14859 free_pcp:0 free_cma:0 [19383.558686] Node 0 active_anon:3638124kB inactive_anon:156kB active_file:140kB inactive_file:104kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:164kB dirty:0kB writeback:0kB shmem:560kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes [19383.579131] Node 0 DMA free:15024kB min:184kB low:228kB high:272kB active_anon:860kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [19383.600063] lowmem_reserve[]: 0 3714 3714 3714 [19383.604058] Node 0 DMA32 free:44412kB min:44868kB low:56084kB high:67300kB active_anon:3637044kB inactive_anon:156kB active_file:64kB inactive_file:240kB unevictable:0kB writepending:0kB present:3915776kB managed:3829972kB mlocked:0kB kernel_stack:4736kB pagetables:11856kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [19383.626118] lowmem_reserve[]: 0 0 0 0 [19383.629660] Node 0 DMA: 1*4kB (M) 0*8kB 0*16kB 2*32kB (UM) 2*64kB (U) 2*128kB (UM) 1*256kB (U) 0*512kB 2*1024kB (UM) 0*2048kB 3*4096kB (M) = 15044kB [19383.640002] Node 0 DMA32: 1773*4kB (UME) 1062*8kB (UME) 534*16kB (UE) 425*32kB (UME) 103*64kB (UME) 6*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45092kB [19383.651450] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [19383.658669] 165 total pagecache pages [19383.662243] 0 pages in swap cache [19383.665423] Swap cache stats: add 0, delete 0, find 0/0 [19383.670000] Free swap = 0kB [19383.672939] Total swap = 0kB [19383.675896] 982941 pages RAM [19383.678946] 0 pages HighMem/MovableOnly [19383.682578] 21472 pages reserved [19383.685849] 0 pages hwpoisoned [19383.688926] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [19383.696054] [ 1690] 0 1690 10291 99 25 3 0 0 systemd-journal [19383.703823] [ 1875] 0 1875 29206 67 26 3 0 0 lvmetad [19383.711245] [ 2629] 0 2629 10653 231 24 3 0 -1000 systemd-udevd [19383.718824] [ 2933] 0 2933 15562 147 29 3 0 -1000 auditd [19383.726200] [ 2994] 0 2994 24997 79 17 3 0 0 irqbalance [19383.733766] [ 3015] 0 3015 3286 27 13 3 0 0 rngd [19383.740838] [ 3058] 81 3058 14584 136 34 3 0 -900 dbus-daemon [19383.748638] [ 3089] 32 3089 18457 179 38 3 0 0 rpcbind [19383.756242] [ 3109] 0 3109 52089 162 39 3 0 0 gssproxy [19383.763526] [ 3122] 0 3122 6609 91 18 3 0 0 systemd-logind [19383.771333] [ 3157] 999 3157 30140 126 29 4 0 0 chronyd [19383.778860] [ 3426] 0 3426 26363 501 50 3 0 0 dhclient [19383.786353] [ 3515] 0 3515 26363 494 49 3 0 0 dhclient [19383.794045] [ 3683] 0 3683 22084 259 40 3 0 0 master [19383.801268] [ 3685] 89 3685 22123 253 42 3 0 0 qmgr [19383.808131] [ 3834] 0 3834 54158 254 40 3 0 0 rsyslogd [19383.815647] [ 3889] 0 3889 33259 157 19 3 0 0 crond [19383.823512] [ 3891] 0 3891 29297 27 10 3 0 0 agetty [19383.832171] [ 3901] 0 3901 29209 29 13 3 0 0 agetty [19383.841011] [ 3953] 0 3953 304259 12089 123 7 0 -999 dockerd [19383.849607] [ 4007] 0 4007 27711 251 56 3 0 -1000 sshd [19383.858035] [ 4045] 0 4045 296383 1843 63 6 0 -999 docker-containe [19383.867339] [ 4448] 0 4448 316050 11561 112 6 0 -999 kubelet [19383.876002] [ 4699] 0 4699 1878 204 8 5 0 -999 docker-containe [19383.885240] [ 4701] 0 4701 1878 182 8 5 0 -999 docker-containe [19383.894375] [ 4750] 0 4750 256 1 4 2 0 -998 pause [19383.902901] [ 4777] 0 4777 256 1 3 2 0 -998 pause [19383.911437] [ 4889] 0 4889 1878 185 8 5 0 -999 docker-containe [19383.920654] [ 4931] 0 4931 34732 1963 28 4 0 -998 kube-proxy [19383.929570] [ 5534] 0 5534 1878 146 9 5 0 -999 docker-containe [19383.938668] [ 5614] 0 5614 256 1 4 2 0 -998 pause [19383.949377] [28338] 0 28338 2230 140 9 5 0 -999 docker-containe [19383.958812] [28378] 0 28378 256 1 4 2 0 -998 pause [19383.967541] [28451] 0 28451 1878 246 9 5 0 -999 docker-containe [19383.977181] [28484] 0 28484 406 30 5 3 0 728 busybox [19383.985706] [28551] 0 28551 1458 219 6 3 0 728 nginx [19383.994335] [28552] 1000 28552 1572 313 6 3 0 728 nginx [19384.002833] [28553] 1000 28553 1572 328 6 3 0 728 nginx [19384.011494] [28554] 0 28554 7385 3784 17 3 0 728 bundle [19384.020229] [28557] 0 28557 22016 13522 46 3 0 728 bundle [19384.028841] [30170] 89 30170 22105 253 43 3 0 0 pickup [19384.037663] [32111] 0 32111 408 35 4 3 0 728 sh [19384.045955] [ 751] 0 751 133700 129589 260 4 0 728 python3 [19384.054625] [ 759] 0 759 133700 129589 258 3 0 728 python3 [19384.063345] [ 760] 0 760 133700 129589 259 3 0 728 python3 [19384.072167] [ 787] 0 787 133700 129589 260 4 0 728 python3 [19384.080799] [ 799] 0 799 133700 129589 260 4 0 728 python3 [19384.088247] [ 1198] 0 1198 407 43 5 3 0 728 sh [19384.095215] [ 1362] 0 1362 133700 129591 260 4 0 728 python3 [19384.102218] [ 1499] 0 1499 1878 193 9 5 0 -999 docker-containe [19384.109893] [ 1509] 0 1509 2230 126 9 5 0 -999 docker-containe [19384.117373] [ 1540] 0 1540 2905 54 11 3 0 999 bash [19384.124044] [ 1602] 0 1602 405 1 4 3 0 999 busybox [19384.131406] [ 1658] 0 1658 556 60 6 3 0 999 spot-monitor.sh [19384.138928] [ 1664] 0 1664 388 1 4 3 0 999 tail [19384.145731] [ 1684] 0 1682 35758 0 32 4 0 999 aws-k8s-agent [19384.153406] [ 2021] 0 2021 388 1 4 3 0 999 sleep [19384.160821] [ 2022] 0 2022 94123 80303 164 4 0 728 python3 [19384.168785] Out of memory: Kill process 1658 (spot-monitor.sh) score 998 or sacrifice child [19384.175899] Killed process 2021 (sleep) total-vm:1552kB, anon-rss:4kB, file-rss:0kB, shmem-rss:0kB [19384.205523] oom_reaper: reaped process 2021 (sleep), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [19385.101716] python3 invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=728 [19385.111057] python3 cpuset=505c48dbdddb7a6c0ab880466281ce3860e3e459f37a15d8812a844301057e7b mems_allowed=0 [19385.118567] CPU: 0 PID: 2022 Comm: python3 Not tainted 4.14.123-111.109.amzn2.x86_64 #1 [19385.125388] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 [19385.130467] Call Trace: [19385.132991] dump_stack+0x5c/0x82 [19385.136046] dump_header+0x94/0x229 [19385.139239] ? get_page_from_freelist+0x525/0xba0 [19385.143303] oom_kill_process+0x223/0x420 [19385.146759] out_of_memory+0x2af/0x4d0 [19385.150254] __alloc_pages_slowpath+0x9b6/0xd20 [19385.154277] __alloc_pages_nodemask+0x25d/0x280 [19385.158060] filemap_fault+0x204/0x5f0 [19385.161353] ? page_add_file_rmap+0xd6/0x160 [19385.164946] __xfs_filemap_fault.constprop.8+0x49/0x120 [xfs] [19385.169792] __do_fault+0x57/0xd9 [19385.173002] __handle_mm_fault+0xdd2/0x12b0 [19385.176758] handle_mm_fault+0xaa/0x1e0 [19385.180121] __do_page_fault+0x23e/0x4c0 [19385.183559] ? page_fault+0x2f/0x50 [19385.186760] page_fault+0x45/0x50 [19385.189867] RIP: 4429e880:0x7ffc4a081280 [19385.193613] RSP: 0003:00005643e8c01c10 EFLAGS: 00000006 [19385.193631] Mem-Info: [19385.200660] active_anon:909527 inactive_anon:39 isolated_anon:0 [19385.200660] active_file:1 inactive_file:28 isolated_file:0 [19385.200660] unevictable:0 dirty:0 writeback:0 unstable:0 [19385.200660] slab_reclaimable:10459 slab_unreclaimable:14564 [19385.200660] mapped:54 shmem:140 pagetables:2933 bounce:0 [19385.200660] free:14959 free_pcp:109 free_cma:0 [19385.224523] Node 0 active_anon:3638108kB inactive_anon:156kB active_file:4kB inactive_file:112kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:116kB dirty:0kB writeback:0kB shmem:560kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [19385.243563] Node 0 DMA free:15020kB min:184kB low:228kB high:272kB active_anon:860kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [19385.262985] lowmem_reserve[]: 0 3714 3714 3714 [19385.266991] Node 0 DMA32 free:44816kB min:44868kB low:56084kB high:67300kB active_anon:3637248kB inactive_anon:156kB active_file:0kB inactive_file:8kB unevictable:0kB writepending:0kB present:3915776kB managed:3829972kB mlocked:0kB kernel_stack:4672kB pagetables:11732kB bounce:0kB free_pcp:604kB local_pcp:0kB free_cma:0kB [19385.289112] lowmem_reserve[]: 0 0 0 0 [19385.292640] Node 0 DMA: 1*4kB (M) 1*8kB (M) 0*16kB 1*32kB (U) 2*64kB (U) 2*128kB (UM) 1*256kB (U) 0*512kB 2*1024kB (UM) 0*2048kB 3*4096kB (M) = 15020kB [19385.303191] Node 0 DMA32: 1702*4kB (UE) 1065*8kB (UME) 535*16kB (UME) 426*32kB (UE) 102*64kB (UME) 6*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44816kB [19385.314720] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [19385.321974] 164 total pagecache pages [19385.325525] 0 pages in swap cache [19385.328766] Swap cache stats: add 0, delete 0, find 0/0 [19385.333192] Free swap = 0kB [19385.336151] Total swap = 0kB [19385.339175] 982941 pages RAM [19385.342235] 0 pages HighMem/MovableOnly [19385.345994] 21472 pages reserved [19385.349293] 0 pages hwpoisoned [19385.352302] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [19385.359326] [ 1690] 0 1690 10291 99 25 3 0 0 systemd-journal [19385.367167] [ 1875] 0 1875 29206 67 26 3 0 0 lvmetad [19385.374553] [ 2629] 0 2629 10653 231 24 3 0 -1000 systemd-udevd [19385.382203] [ 2933] 0 2933 15562 147 29 3 0 -1000 auditd [19385.389532] [ 2994] 0 2994 24997 79 17 3 0 0 irqbalance [19385.397238] [ 3015] 0 3015 3286 27 13 3 0 0 rngd [19385.404268] [ 3058] 81 3058 14584 136 34 3 0 -900 dbus-daemon [19385.411774] [ 3089] 32 3089 18457 179 38 3 0 0 rpcbind [19385.418992] [ 3109] 0 3109 52089 162 39 3 0 0 gssproxy [19385.426265] [ 3122] 0 3122 6609 91 18 3 0 0 systemd-logind [19385.434569] [ 3157] 999 3157 30140 126 29 4 0 0 chronyd [19385.442088] [ 3426] 0 3426 26363 501 50 3 0 0 dhclient [19385.449951] [ 3515] 0 3515 26363 494 49 3 0 0 dhclient [19385.457716] [ 3683] 0 3683 22084 259 40 3 0 0 master [19385.465186] [ 3685] 89 3685 22123 253 42 3 0 0 qmgr [19385.472347] [ 3834] 0 3834 54158 254 40 3 0 0 rsyslogd [19385.479791] [ 3889] 0 3889 33259 157 19 3 0 0 crond [19385.487167] [ 3891] 0 3891 29297 27 10 3 0 0 agetty [19385.494340] [ 3901] 0 3901 29209 29 13 3 0 0 agetty [19385.501645] [ 3953] 0 3953 304259 12089 123 7 0 -999 dockerd [19385.508986] [ 4007] 0 4007 27711 251 56 3 0 -1000 sshd [19385.516170] [ 4045] 0 4045 296383 1843 63 6 0 -999 docker-containe [19385.524122] [ 4448] 0 4448 316050 11561 112 6 0 -999 kubelet [19385.531684] [ 4699] 0 4699 1878 204 8 5 0 -999 docker-containe [19385.539500] [ 4701] 0 4701 1878 182 8 5 0 -999 docker-containe [19385.547329] [ 4750] 0 4750 256 1 4 2 0 -998 pause [19385.554704] [ 4777] 0 4777 256 1 3 2 0 -998 pause [19385.563729] [ 4889] 0 4889 1878 185 8 5 0 -999 docker-containe [19385.571739] [ 4931] 0 4931 34732 1963 28 4 0 -998 kube-proxy [19385.579255] [ 5534] 0 5534 1878 146 9 5 0 -999 docker-containe [19385.586865] [ 5614] 0 5614 256 1 4 2 0 -998 pause [19385.594365] [28338] 0 28338 2230 140 9 5 0 -999 docker-containe [19385.602255] [28378] 0 28378 256 1 4 2 0 -998 pause [19385.609360] [28451] 0 28451 1878 246 9 5 0 -999 docker-containe [19385.617380] [28484] 0 28484 406 30 5 3 0 728 busybox [19385.624824] [28551] 0 28551 1458 219 6 3 0 728 nginx [19385.632092] [28552] 1000 28552 1572 313 6 3 0 728 nginx [19385.639478] [28553] 1000 28553 1572 328 6 3 0 728 nginx [19385.646495] [28554] 0 28554 7385 3784 17 3 0 728 bundle [19385.653623] [28557] 0 28557 22016 13522 46 3 0 728 bundle [19385.660984] [30170] 89 30170 22105 253 43 3 0 0 pickup [19385.668297] [32111] 0 32111 408 35 4 3 0 728 sh [19385.675311] [ 751] 0 751 133700 129589 260 4 0 728 python3 [19385.682919] [ 759] 0 759 133700 129589 258 3 0 728 python3 [19385.690259] [ 760] 0 760 133700 129589 259 3 0 728 python3 [19385.697587] [ 787] 0 787 133700 129589 260 4 0 728 python3 [19385.705036] [ 799] 0 799 133700 129589 260 4 0 728 python3 [19385.712268] [ 1198] 0 1198 407 43 5 3 0 728 sh [19385.719241] [ 1362] 0 1362 133700 129591 260 4 0 728 python3 [19385.726576] [ 1499] 0 1499 1878 193 9 5 0 -999 docker-containe [19385.734421] [ 1509] 0 1509 2230 126 9 5 0 -999 docker-containe [19385.742259] [ 1540] 0 1540 2905 54 11 3 0 999 bash [19385.749525] [ 1602] 0 1602 405 1 4 3 0 999 busybox [19385.757104] [ 1658] 0 1658 556 59 6 3 0 999 spot-monitor.sh [19385.764837] [ 1664] 0 1664 388 1 4 3 0 999 tail [19385.772022] [ 2022] 0 2022 94187 80303 164 4 0 728 python3 [19385.779516] Out of memory: Kill process 1658 (spot-monitor.sh) score 998 or sacrifice child [19385.786615] Killed process 1658 (spot-monitor.sh) total-vm:2224kB, anon-rss:236kB, file-rss:0kB, shmem-rss:0kB [19385.844461] oom_reaper: reaped process 1658 (spot-monitor.sh), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [19386.191057] docker-containe invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=-999 [19386.202031] docker-containe cpuset=/ mems_allowed=0 [19386.206334] CPU: 1 PID: 15644 Comm: docker-containe Not tainted 4.14.123-111.109.amzn2.x86_64 #1 [19386.213713] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 [19386.218453] Call Trace: [19386.220980] dump_stack+0x5c/0x82 [19386.224320] dump_header+0x94/0x229 [19386.227774] ? get_page_from_freelist+0x525/0xba0 [19386.231966] oom_kill_process+0x223/0x420 [19386.235696] out_of_memory+0x2af/0x4d0 [19386.239317] __alloc_pages_slowpath+0x9b6/0xd20 [19386.243396] __alloc_pages_nodemask+0x25d/0x280 [19386.247384] filemap_fault+0x204/0x5f0 [19386.250980] ? page_add_file_rmap+0xd6/0x160 [19386.254886] __xfs_filemap_fault.constprop.8+0x49/0x120 [xfs] [19386.259562] __do_fault+0x57/0xd9 [19386.262605] __handle_mm_fault+0xdd2/0x12b0 [19386.266194] handle_mm_fault+0xaa/0x1e0 [19386.269760] __do_page_fault+0x23e/0x4c0 [19386.273395] ? page_fault+0x2f/0x50 [19386.276666] page_fault+0x45/0x50 [19386.279971] RIP: 00f7:0x77 [19386.282800] RSP: 204a8180:00007fe522ffcba0 EFLAGS: 700070007 [19386.282868] Mem-Info: [19386.289952] active_anon:909556 inactive_anon:39 isolated_anon:0 [19386.289952] active_file:34 inactive_file:5 isolated_file:0 [19386.289952] unevictable:0 dirty:0 writeback:0 unstable:0 [19386.289952] slab_reclaimable:10458 slab_unreclaimable:14564 [19386.289952] mapped:36 shmem:140 pagetables:2927 bounce:0 [19386.289952] free:14851 free_pcp:182 free_cma:0 [19386.315309] Node 0 active_anon:3638224kB inactive_anon:156kB active_file:136kB inactive_file:20kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:144kB dirty:0kB writeback:0kB shmem:560kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [19386.339624] Node 0 DMA free:15020kB min:184kB low:228kB high:272kB active_anon:884kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [19386.363851] lowmem_reserve[]: 0 3714 3714 3714 [19386.368784] Node 0 DMA32 free:44384kB min:44868kB low:56084kB high:67300kB active_anon:3637364kB inactive_anon:156kB active_file:40kB inactive_file:240kB unevictable:0kB writepending:0kB present:3915776kB managed:3829972kB mlocked:0kB kernel_stack:4656kB pagetables:11708kB bounce:0kB free_pcp:784kB local_pcp:728kB free_cma:0kB [19386.395417] lowmem_reserve[]: 0 0 0 0 [19386.399358] Node 0 DMA: 1*4kB (M) 1*8kB (M) 0*16kB 1*32kB (U) 2*64kB (U) 2*128kB (UM) 1*256kB (U) 0*512kB 2*1024kB (UM) 0*2048kB 3*4096kB (M) = 15020kB [19386.411147] Node 0 DMA32: 1756*4kB (UME) 1065*8kB (UME) 536*16kB (UME) 429*32kB (UME) 94*64kB (UME) 6*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44632kB [19386.424803] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [19386.432937] 159 total pagecache pages [19386.436449] 0 pages in swap cache [19386.440185] Swap cache stats: add 0, delete 0, find 0/0 [19386.445626] Free swap = 0kB [19386.449098] Total swap = 0kB [19386.452112] 982941 pages RAM [19386.455106] 0 pages HighMem/MovableOnly [19386.458596] 21472 pages reserved [19386.461848] 0 pages hwpoisoned [19386.464813] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [19386.472085] [ 1690] 0 1690 10291 99 25 3 0 0 systemd-journal [19386.479882] [ 1875] 0 1875 29206 67 26 3 0 0 lvmetad [19386.486833] [ 2629] 0 2629 10653 231 24 3 0 -1000 systemd-udevd [19386.494198] [ 2933] 0 2933 15562 147 29 3 0 -1000 auditd [19386.501564] [ 2994] 0 2994 24997 79 17 3 0 0 irqbalance [19386.508884] [ 3015] 0 3015 3286 27 13 3 0 0 rngd [19386.515718] [ 3058] 81 3058 14584 136 34 3 0 -900 dbus-daemon [19386.523535] [ 3089] 32 3089 18457 179 38 3 0 0 rpcbind [19386.530764] [ 3109] 0 3109 52089 162 39 3 0 0 gssproxy [19386.538212] [ 3122] 0 3122 6609 91 18 3 0 0 systemd-logind [19386.545902] [ 3157] 999 3157 30140 131 29 4 0 0 chronyd [19386.553072] [ 3426] 0 3426 26363 501 50 3 0 0 dhclient [19386.560555] [ 3515] 0 3515 26363 494 49 3 0 0 dhclient [19386.568029] [ 3683] 0 3683 22084 259 40 3 0 0 master [19386.575227] [ 3685] 89 3685 22123 253 42 3 0 0 qmgr [19386.582210] [ 3834] 0 3834 54158 254 40 3 0 0 rsyslogd [19386.589627] [ 3889] 0 3889 33259 157 19 3 0 0 crond [19386.596943] [ 3891] 0 3891 29297 27 10 3 0 0 agetty [19386.604079] [ 3901] 0 3901 29209 29 13 3 0 0 agetty [19386.611068] [ 3953] 0 3953 304259 12089 123 7 0 -999 dockerd [19386.618404] [ 4007] 0 4007 27711 251 56 3 0 -1000 sshd [19386.625568] [ 4045] 0 4045 296383 1843 63 6 0 -999 docker-containe [19386.633372] [ 4448] 0 4448 316050 11561 112 6 0 -999 kubelet [19386.640862] [ 4699] 0 4699 1878 204 8 5 0 -999 docker-containe [19386.648650] [ 4701] 0 4701 1878 182 8 5 0 -999 docker-containe [19386.656359] [ 4750] 0 4750 256 1 4 2 0 -998 pause [19386.665409] [ 4777] 0 4777 256 1 3 2 0 -998 pause [19386.672824] [ 4889] 0 4889 1878 185 8 5 0 -999 docker-containe [19386.680508] [ 4931] 0 4931 34732 1963 28 4 0 -998 kube-proxy [19386.688034] [ 5534] 0 5534 1878 146 9 5 0 -999 docker-containe [19386.696098] [ 5614] 0 5614 256 1 4 2 0 -998 pause [19386.703286] [28338] 0 28338 2230 140 9 5 0 -999 docker-containe [19386.710903] [28378] 0 28378 256 1 4 2 0 -998 pause [19386.718164] [28451] 0 28451 1878 246 9 5 0 -999 docker-containe [19386.726120] [28484] 0 28484 406 30 5 3 0 728 busybox [19386.733501] [28551] 0 28551 1458 219 6 3 0 728 nginx [19386.740686] [28552] 1000 28552 1572 313 6 3 0 728 nginx [19386.747736] [28553] 1000 28553 1572 328 6 3 0 728 nginx [19386.755067] [28554] 0 28554 7385 3784 17 3 0 728 bundle [19386.762148] [28557] 0 28557 22016 13522 46 3 0 728 bundle [19386.769492] [30170] 89 30170 22105 253 43 3 0 0 pickup [19386.776852] [32111] 0 32111 408 35 4 3 0 728 sh [19386.783961] [ 751] 0 751 133700 129589 260 4 0 728 python3 [19386.791256] [ 759] 0 759 133700 129589 258 3 0 728 python3 [19386.798658] [ 760] 0 760 133700 129589 259 3 0 728 python3 [19386.805948] [ 787] 0 787 133700 129589 260 4 0 728 python3 [19386.813043] [ 799] 0 799 133700 129589 260 4 0 728 python3 [19386.820474] [ 1198] 0 1198 407 43 5 3 0 728 sh [19386.827955] [ 1362] 0 1362 133700 129591 260 4 0 728 python3 [19386.835250] [ 1499] 0 1499 1878 193 9 5 0 -999 docker-containe [19386.843137] [ 1509] 0 1509 2230 126 9 5 0 -999 docker-containe [19386.851154] [ 1540] 0 1540 2905 54 11 3 0 999 bash [19386.858052] [ 1602] 0 1602 405 1 4 3 0 999 busybox [19386.865338] [ 1664] 0 1664 388 1 4 3 0 999 tail [19386.872656] [ 2022] 0 2022 94251 80410 164 4 0 728 python3 [19386.879887] Out of memory: Kill process 1540 (bash) score 998 or sacrifice child [19386.886275] Killed process 1540 (bash) total-vm:11620kB, anon-rss:212kB, file-rss:4kB, shmem-rss:0kB [19387.659760] oom_reaper: reaped process 1540 (bash), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [19387.684586] python3 invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=728 [19387.693731] python3 cpuset=505c48dbdddb7a6c0ab880466281ce3860e3e459f37a15d8812a844301057e7b mems_allowed=0 [19387.701668] CPU: 0 PID: 2022 Comm: python3 Not tainted 4.14.123-111.109.amzn2.x86_64 #1 [19387.708131] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 [19387.713199] Call Trace: [19387.715851] dump_stack+0x5c/0x82 [19387.719022] dump_header+0x94/0x229 [19387.722361] ? get_page_from_freelist+0x525/0xba0 [19387.726326] oom_kill_process+0x223/0x420 [19387.730005] out_of_memory+0x2af/0x4d0 [19387.733518] __alloc_pages_slowpath+0x9b6/0xd20 [19387.737649] __alloc_pages_nodemask+0x25d/0x280 [19387.741822] filemap_fault+0x204/0x5f0 [19387.745251] ? page_add_file_rmap+0xd6/0x160 [19387.749054] ? alloc_set_pte+0x2b3/0x5e0 [19387.752788] __xfs_filemap_fault.constprop.8+0x49/0x120 [xfs] [19387.757377] __do_fault+0x57/0xd9 [19387.760657] __handle_mm_fault+0xdd2/0x12b0 [19387.764462] handle_mm_fault+0xaa/0x1e0 [19387.768165] __do_page_fault+0x23e/0x4c0 [19387.771979] ? page_fault+0x2f/0x50 [19387.775070] page_fault+0x45/0x50 [19387.778204] RIP: 4429e880:0x7ffc4a081280 [19387.781706] RSP: 0003:00005643e8c01c70 EFLAGS: 00000006 [19387.781728] Mem-Info: [19387.789257] active_anon:909577 inactive_anon:39 isolated_anon:0 [19387.789257] active_file:0 inactive_file:2 isolated_file:0 [19387.789257] unevictable:0 dirty:0 writeback:0 unstable:0 [19387.789257] slab_reclaimable:10458 slab_unreclaimable:14564 [19387.789257] mapped:4 shmem:140 pagetables:2927 bounce:0 [19387.789257] free:14885 free_pcp:235 free_cma:0 [19387.814537] Node 0 active_anon:3638308kB inactive_anon:156kB active_file:0kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:16kB dirty:0kB writeback:0kB shmem:560kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes [19387.836617] Node 0 DMA free:15020kB min:184kB low:228kB high:272kB active_anon:884kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [19387.856674] lowmem_reserve[]: 0 3714 3714 3714 [19387.860796] Node 0 DMA32 free:44520kB min:44868kB low:56084kB high:67300kB active_anon:3637424kB inactive_anon:156kB active_file:0kB inactive_file:8kB unevictable:0kB writepending:0kB present:3915776kB managed:3829972kB mlocked:0kB kernel_stack:4656kB pagetables:11708kB bounce:0kB free_pcp:940kB local_pcp:708kB free_cma:0kB [19387.882473] lowmem_reserve[]: 0 0 0 0 [19387.886125] Node 0 DMA: 1*4kB (M) 1*8kB (M) 0*16kB 1*32kB (U) 2*64kB (U) 2*128kB (UM) 1*256kB (U) 0*512kB 2*1024kB (UM) 0*2048kB 3*4096kB (M) = 15020kB [19387.896825] Node 0 DMA32: 1708*4kB (UME) 1073*8kB (UME) 533*16kB (UE) 427*32kB (UE) 96*64kB (UME) 6*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44520kB [19387.908322] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [19387.915592] 142 total pagecache pages [19387.919310] 0 pages in swap cache [19387.922722] Swap cache stats: add 0, delete 0, find 0/0 [19387.927390] Free swap = 0kB [19387.930342] Total swap = 0kB [19387.933417] 982941 pages RAM [19387.936758] 0 pages HighMem/MovableOnly [19387.940702] 21472 pages reserved [19387.944034] 0 pages hwpoisoned [19387.947275] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [19387.954793] [ 1690] 0 1690 10291 99 25 3 0 0 systemd-journal [19387.962540] [ 1875] 0 1875 29206 67 26 3 0 0 lvmetad [19387.969987] [ 2629] 0 2629 10653 231 24 3 0 -1000 systemd-udevd [19387.977413] [ 2933] 0 2933 15562 147 29 3 0 -1000 auditd [19387.984805] [ 2994] 0 2994 24997 79 17 3 0 0 irqbalance [19387.992592] [ 3015] 0 3015 3286 27 13 3 0 0 rngd [19387.999957] [ 3058] 81 3058 14584 136 34 3 0 -900 dbus-daemon [19388.007604] [ 3089] 32 3089 18457 179 38 3 0 0 rpcbind [19388.015052] [ 3109] 0 3109 52089 162 39 3 0 0 gssproxy [19388.022826] [ 3122] 0 3122 6609 91 18 3 0 0 systemd-logind [19388.030671] [ 3157] 999 3157 30140 131 29 4 0 0 chronyd [19388.038450] [ 3426] 0 3426 26363 501 50 3 0 0 dhclient [19388.046047] [ 3515] 0 3515 26363 494 49 3 0 0 dhclient [19388.053357] [ 3683] 0 3683 22084 259 40 3 0 0 master [19388.060547] [ 3685] 89 3685 22123 253 42 3 0 0 qmgr [19388.067781] [ 3834] 0 3834 54158 254 40 3 0 0 rsyslogd [19388.075165] [ 3889] 0 3889 33259 157 19 3 0 0 crond [19388.082493] [ 3891] 0 3891 29297 27 10 3 0 0 agetty [19388.089906] [ 3901] 0 3901 29209 29 13 3 0 0 agetty [19388.097306] [ 3953] 0 3953 304259 12089 123 7 0 -999 dockerd [19388.104720] [ 4007] 0 4007 27711 251 56 3 0 -1000 sshd [19388.112131] [ 4045] 0 4045 296383 1843 63 6 0 -999 docker-containe [19388.119904] [ 4448] 0 4448 316050 11561 112 6 0 -999 kubelet [19388.127330] [ 4699] 0 4699 1878 204 8 5 0 -999 docker-containe [19388.136978] [ 4701] 0 4701 1878 182 8 5 0 -999 docker-containe [19388.144833] [ 4750] 0 4750 256 1 4 2 0 -998 pause [19388.152206] [ 4777] 0 4777 256 1 3 2 0 -998 pause [19388.159305] [ 4889] 0 4889 1878 185 8 5 0 -999 docker-containe [19388.166967] [ 4931] 0 4931 34732 1963 28 4 0 -998 kube-proxy [19388.174598] [ 5534] 0 5534 1878 146 9 5 0 -999 docker-containe [19388.182521] [ 5614] 0 5614 256 1 4 2 0 -998 pause [19388.189733] [28338] 0 28338 2230 140 9 5 0 -999 docker-containe [19388.197597] [28378] 0 28378 256 1 4 2 0 -998 pause [19388.204772] [28451] 0 28451 1878 246 9 5 0 -999 docker-containe [19388.212033] [28484] 0 28484 406 30 5 3 0 728 busybox [19388.219369] [28551] 0 28551 1458 219 6 3 0 728 nginx [19388.226810] [28552] 1000 28552 1572 313 6 3 0 728 nginx [19388.233905] [28553] 1000 28553 1572 328 6 3 0 728 nginx [19388.241286] [28554] 0 28554 7385 3784 17 3 0 728 bundle [19388.248703] [28557] 0 28557 22016 13522 46 3 0 728 bundle [19388.255896] [30170] 89 30170 22105 253 43 3 0 0 pickup [19388.263198] [32111] 0 32111 408 35 4 3 0 728 sh [19388.270545] [ 751] 0 751 133700 129589 260 4 0 728 python3 [19388.278230] [ 759] 0 759 133700 129589 258 3 0 728 python3 [19388.285607] [ 760] 0 760 133700 129589 259 3 0 728 python3 [19388.293108] [ 787] 0 787 133700 129589 260 4 0 728 python3 [19388.300453] [ 799] 0 799 133700 129589 260 4 0 728 python3 [19388.307788] [ 1198] 0 1198 407 43 5 3 0 728 sh [19388.315083] [ 1362] 0 1362 133700 129591 260 4 0 728 python3 [19388.322421] [ 1499] 0 1499 1878 193 9 5 0 -999 docker-containe [19388.330345] [ 1509] 0 1509 2230 126 9 5 0 -999 docker-containe [19388.338451] [ 1602] 0 1602 405 1 4 3 0 999 busybox [19388.345949] [ 1664] 0 1664 388 1 4 3 0 999 tail [19388.353192] [ 2022] 0 2022 94315 80506 164 4 0 728 python3 [19388.360494] Out of memory: Kill process 1602 (busybox) score 998 or sacrifice child [19388.367252] Killed process 1664 (tail) total-vm:1552kB, anon-rss:4kB, file-rss:0kB, shmem-rss:0kB [19388.473626] oom_reaper: reaped process 1664 (tail), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [19388.481752] dockerd invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=-999 [19388.493451] dockerd cpuset=/ mems_allowed=0 [19388.497819] CPU: 0 PID: 3981 Comm: dockerd Not tainted 4.14.123-111.109.amzn2.x86_64 #1 [19388.505545] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 [19388.511700] Call Trace: [19388.514760] dump_stack+0x5c/0x82 [19388.518454] dump_header+0x94/0x229 [19388.522397] ? get_page_from_freelist+0x525/0xba0 [19388.527248] oom_kill_process+0x223/0x420 [19388.531562] out_of_memory+0x2af/0x4d0 [19388.535750] __alloc_pages_slowpath+0x9b6/0xd20 [19388.540458] __alloc_pages_nodemask+0x25d/0x280 [19388.544400] filemap_fault+0x204/0x5f0 [19388.547795] ? core_sys_select+0x24b/0x280 [19388.551518] __xfs_filemap_fault.constprop.8+0x49/0x120 [xfs] [19388.556175] __do_fault+0x57/0xd9 [19388.559353] __handle_mm_fault+0xdd2/0x12b0 [19388.563074] handle_mm_fault+0xaa/0x1e0 [19388.566626] __do_page_fault+0x23e/0x4c0 [19388.570290] ? page_fault+0x2f/0x50 [19388.574439] page_fault+0x45/0x50 [19388.578165] RIP: 00f1:0x11 [19388.581370] RSP: 0000:00007f852c6bcc28 EFLAGS: 00000000 [19388.581413] Mem-Info: [19388.589665] active_anon:909574 inactive_anon:39 isolated_anon:0 [19388.589665] active_file:0 inactive_file:2 isolated_file:0 [19388.589665] unevictable:0 dirty:0 writeback:0 unstable:0 [19388.589665] slab_reclaimable:10458 slab_unreclaimable:14564 [19388.589665] mapped:4 shmem:140 pagetables:2916 bounce:0 [19388.589665] free:14935 free_pcp:177 free_cma:0 [19388.619133] Node 0 active_anon:3638296kB inactive_anon:156kB active_file:0kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:16kB dirty:0kB writeback:0kB shmem:560kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [19388.643064] Node 0 DMA free:15020kB min:184kB low:228kB high:272kB active_anon:884kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [19388.666959] lowmem_reserve[]: 0 3714 3714 3714 [19388.671666] Node 0 DMA32 free:44720kB min:44868kB low:56084kB high:67300kB active_anon:3637412kB inactive_anon:156kB active_file:0kB inactive_file:8kB unevictable:0kB writepending:0kB present:3915776kB managed:3829972kB mlocked:0kB kernel_stack:4656kB pagetables:11664kB bounce:0kB free_pcp:708kB local_pcp:708kB free_cma:0kB [19388.697615] lowmem_reserve[]: 0 0 0 0 [19388.701487] Node 0 DMA: 1*4kB (M) 1*8kB (M) 0*16kB 1*32kB (U) 2*64kB (U) 2*128kB (UM) 1*256kB (U) 0*512kB 2*1024kB (UM) 0*2048kB 3*4096kB (M) = 15020kB [19388.712384] Node 0 DMA32: 1748*4kB (UME) 1079*8kB (UME) 536*16kB (UME) 428*32kB (UME) 96*64kB (UME) 6*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44808kB [19388.723751] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [19388.730908] 142 total pagecache pages [19388.734357] 0 pages in swap cache [19388.737711] Swap cache stats: add 0, delete 0, find 0/0 [19388.742024] Free swap = 0kB [19388.744957] Total swap = 0kB [19388.747872] 982941 pages RAM [19388.750804] 0 pages HighMem/MovableOnly [19388.754212] 21472 pages reserved [19388.757213] 0 pages hwpoisoned [19388.760204] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [19388.767258] [ 1690] 0 1690 10291 99 25 3 0 0 systemd-journal [19388.775230] [ 1875] 0 1875 29206 67 26 3 0 0 lvmetad [19388.782619] [ 2629] 0 2629 10653 231 24 3 0 -1000 systemd-udevd [19388.790454] [ 2933] 0 2933 15562 147 29 3 0 -1000 auditd [19388.797584] [ 2994] 0 2994 24997 79 17 3 0 0 irqbalance [19388.804974] [ 3015] 0 3015 3286 27 13 3 0 0 rngd [19388.811868] [ 3058] 81 3058 14584 136 34 3 0 -900 dbus-daemon [19388.819388] [ 3089] 32 3089 18457 179 38 3 0 0 rpcbind [19388.826808] [ 3109] 0 3109 52089 162 39 3 0 0 gssproxy [19388.834155] [ 3122] 0 3122 6609 91 18 3 0 0 systemd-logind [19388.841867] [ 3157] 999 3157 30140 131 29 4 0 0 chronyd [19388.849254] [ 3426] 0 3426 26363 501 50 3 0 0 dhclient [19388.856623] [ 3515] 0 3515 26363 494 49 3 0 0 dhclient [19388.863852] [ 3683] 0 3683 22084 259 40 3 0 0 master [19388.871183] [ 3685] 89 3685 22123 253 42 3 0 0 qmgr [19388.878390] [ 3834] 0 3834 54158 254 40 3 0 0 rsyslogd [19388.885708] [ 3889] 0 3889 33259 157 19 3 0 0 crond [19388.893354] [ 3891] 0 3891 29297 27 10 3 0 0 agetty [19388.900761] [ 3901] 0 3901 29209 29 13 3 0 0 agetty [19388.907637] [ 3953] 0 3953 304259 12089 123 7 0 -999 dockerd [19388.914911] [ 4007] 0 4007 27711 251 56 3 0 -1000 sshd [19388.922070] [ 4045] 0 4045 296383 1843 63 6 0 -999 docker-containe [19388.929800] [ 4448] 0 4448 316050 11561 112 6 0 -999 kubelet [19388.937140] [ 4699] 0 4699 1878 204 8 5 0 -999 docker-containe [19388.945069] [ 4701] 0 4701 1878 182 8 5 0 -999 docker-containe [19388.954640] [ 4750] 0 4750 256 1 4 2 0 -998 pause [19388.961854] [ 4777] 0 4777 256 1 3 2 0 -998 pause [19388.968941] [ 4889] 0 4889 1878 185 8 5 0 -999 docker-containe [19388.976352] [ 4931] 0 4931 34732 1963 28 4 0 -998 kube-proxy [19388.983989] [ 5534] 0 5534 1878 146 9 5 0 -999 docker-containe [19388.991846] [ 5614] 0 5614 256 1 4 2 0 -998 pause [19388.999153] [28338] 0 28338 2230 140 9 5 0 -999 docker-containe [19389.006923] [28378] 0 28378 256 1 4 2 0 -998 pause [19389.014239] [28451] 0 28451 1878 246 9 5 0 -999 docker-containe [19389.021711] [28484] 0 28484 406 30 5 3 0 728 busybox [19389.029084] [28551] 0 28551 1458 219 6 3 0 728 nginx [19389.037048] [28552] 1000 28552 1572 313 6 3 0 728 nginx [19389.044348] [28553] 1000 28553 1572 328 6 3 0 728 nginx [19389.051619] [28554] 0 28554 7385 3784 17 3 0 728 bundle [19389.058750] [28557] 0 28557 22016 13522 46 3 0 728 bundle [19389.065967] [30170] 89 30170 22105 253 43 3 0 0 pickup [19389.073005] [32111] 0 32111 408 35 4 3 0 728 sh [19389.079899] [ 751] 0 751 133700 129589 260 4 0 728 python3 [19389.087188] [ 759] 0 759 133700 129589 258 3 0 728 python3 [19389.094474] [ 760] 0 760 133700 129589 259 3 0 728 python3 [19389.101853] [ 787] 0 787 133700 129589 260 4 0 728 python3 [19389.109314] [ 799] 0 799 133700 129589 260 4 0 728 python3 [19389.116300] [ 1198] 0 1198 407 43 5 3 0 728 sh [19389.123058] [ 1362] 0 1362 133700 129591 260 4 0 728 python3 [19389.130384] [ 1499] 0 1499 1878 193 9 5 0 -999 docker-containe [19389.138104] [ 1509] 0 1509 2230 126 9 5 0 -999 docker-containe [19389.145891] [ 1602] 0 1602 405 1 4 3 0 999 busybox [19389.153278] [ 2022] 0 2022 94315 80506 164 4 0 728 python3 [19389.160611] Out of memory: Kill process 1602 (busybox) score 998 or sacrifice child [19389.167155] Killed process 1602 (busybox) total-vm:1620kB, anon-rss:4kB, file-rss:0kB, shmem-rss:0kB [19389.183473] oom_reaper: reaped process 1602 (busybox), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [19390.028897] python3 invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=728 [19390.038996] python3 cpuset=505c48dbdddb7a6c0ab880466281ce3860e3e459f37a15d8812a844301057e7b mems_allowed=0 [19390.047549] CPU: 0 PID: 2022 Comm: python3 Not tainted 4.14.123-111.109.amzn2.x86_64 #1 [19390.054042] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 [19390.058992] Call Trace: [19390.061605] dump_stack+0x5c/0x82 [19390.064812] dump_header+0x94/0x229 [19390.068049] ? get_page_from_freelist+0x525/0xba0 [19390.072065] oom_kill_process+0x223/0x420 [19390.075470] out_of_memory+0x2af/0x4d0 [19390.078838] __alloc_pages_slowpath+0x9b6/0xd20 [19390.082706] __alloc_pages_nodemask+0x25d/0x280 [19390.086626] filemap_fault+0x204/0x5f0 [19390.090071] ? page_add_file_rmap+0xd6/0x160 [19390.093776] __xfs_filemap_fault.constprop.8+0x49/0x120 [xfs] [19390.098477] __do_fault+0x57/0xd9 [19390.101545] __handle_mm_fault+0xdd2/0x12b0 [19390.105082] handle_mm_fault+0xaa/0x1e0 [19390.108454] __do_page_fault+0x23e/0x4c0 [19390.112100] ? page_fault+0x2f/0x50 [19390.115291] page_fault+0x45/0x50 [19390.118429] RIP: 440ff5aa:0x7f5a44176b1e [19390.121990] RSP: 43f2ba20:00007f5a43eec088 EFLAGS: 7f5a2d465030 [19390.122058] Mem-Info: [19390.129227] active_anon:909559 inactive_anon:39 isolated_anon:0 [19390.129227] active_file:20 inactive_file:2 isolated_file:0 [19390.129227] unevictable:0 dirty:0 writeback:0 unstable:0 [19390.129227] slab_reclaimable:10458 slab_unreclaimable:14564 [19390.129227] mapped:26 shmem:140 pagetables:2918 bounce:0 [19390.129227] free:14963 free_pcp:0 free_cma:0 [19390.152840] Node 0 active_anon:3638236kB inactive_anon:156kB active_file:80kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:104kB dirty:0kB writeback:0kB shmem:560kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [19390.171894] Node 0 DMA free:15020kB min:184kB low:228kB high:272kB active_anon:884kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [19390.190847] lowmem_reserve[]: 0 3714 3714 3714 [19390.194867] Node 0 DMA32 free:44832kB min:44868kB low:56084kB high:67300kB active_anon:3637352kB inactive_anon:156kB active_file:0kB inactive_file:252kB unevictable:0kB writepending:0kB present:3915776kB managed:3829972kB mlocked:0kB kernel_stack:4640kB pagetables:11672kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [19390.215720] lowmem_reserve[]: 0 0 0 0 [19390.219198] Node 0 DMA: 1*4kB (M) 1*8kB (M) 0*16kB 1*32kB (U) 2*64kB (U) 2*128kB (UM) 1*256kB (U) 0*512kB 2*1024kB (UM) 0*2048kB 3*4096kB (M) = 15020kB [19390.229969] Node 0 DMA32: 1858*4kB (UME) 1087*8kB (UME) 537*16kB (UME) 430*32kB (UME) 98*64kB (UME) 6*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45520kB [19390.241358] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [19390.248651] 164 total pagecache pages [19390.252235] 0 pages in swap cache [19390.255569] Swap cache stats: add 0, delete 0, find 0/0 [19390.260052] Free swap = 0kB [19390.262942] Total swap = 0kB [19390.265837] 982941 pages RAM [19390.268864] 0 pages HighMem/MovableOnly [19390.272450] 21472 pages reserved [19390.275657] 0 pages hwpoisoned [19390.278816] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [19390.285858] [ 1690] 0 1690 10291 99 25 3 0 0 systemd-journal [19390.293707] [ 1875] 0 1875 29206 67 26 3 0 0 lvmetad [19390.301101] [ 2629] 0 2629 10653 231 24 3 0 -1000 systemd-udevd [19390.308893] [ 2933] 0 2933 15562 147 29 3 0 -1000 auditd [19390.316262] [ 2994] 0 2994 24997 79 17 3 0 0 irqbalance [19390.323905] [ 3015] 0 3015 3286 27 13 3 0 0 rngd [19390.331040] [ 3058] 81 3058 14584 136 34 3 0 -900 dbus-daemon [19390.338568] [ 3089] 32 3089 18457 179 38 3 0 0 rpcbind [19390.346021] [ 3109] 0 3109 52089 162 39 3 0 0 gssproxy [19390.353548] [ 3122] 0 3122 6609 91 18 3 0 0 systemd-logind [19390.361376] [ 3157] 999 3157 30140 131 29 4 0 0 chronyd [19390.368602] [ 3426] 0 3426 26363 501 50 3 0 0 dhclient [19390.375957] [ 3515] 0 3515 26363 494 49 3 0 0 dhclient [19390.383204] [ 3683] 0 3683 22084 259 40 3 0 0 master [19390.390569] [ 3685] 89 3685 22123 253 42 3 0 0 qmgr [19390.397558] [ 3834] 0 3834 54158 254 40 3 0 0 rsyslogd [19390.405152] [ 3889] 0 3889 33259 157 19 3 0 0 crond [19390.412681] [ 3891] 0 3891 29297 27 10 3 0 0 agetty [19390.419921] [ 3901] 0 3901 29209 29 13 3 0 0 agetty [19390.427269] [ 3953] 0 3953 304259 12089 123 7 0 -999 dockerd [19390.434611] [ 4007] 0 4007 27711 251 56 3 0 -1000 sshd [19390.441547] [ 4045] 0 4045 296383 1843 63 6 0 -999 docker-containe [19390.449539] [ 4448] 0 4448 316050 11561 112 6 0 -999 kubelet [19390.456937] [ 4699] 0 4699 1878 204 8 5 0 -999 docker-containe [19390.464743] [ 4701] 0 4701 1878 182 8 5 0 -999 docker-containe [19390.472705] [ 4750] 0 4750 256 1 4 2 0 -998 pause [19390.481798] [ 4777] 0 4777 256 1 3 2 0 -998 pause [19390.489107] [ 4889] 0 4889 1878 185 8 5 0 -999 docker-containe [19390.496951] [ 4931] 0 4931 34732 1963 28 4 0 -998 kube-proxy [19390.504469] [ 5534] 0 5534 1878 146 9 5 0 -999 docker-containe [19390.512384] [ 5614] 0 5614 256 1 4 2 0 -998 pause [19390.519676] [28338] 0 28338 2230 140 9 5 0 -999 docker-containe [19390.527241] [28378] 0 28378 256 1 4 2 0 -998 pause [19390.534634] [28451] 0 28451 1878 246 9 5 0 -999 docker-containe [19390.542358] [28484] 0 28484 406 30 5 3 0 728 busybox [19390.549880] [28551] 0 28551 1458 219 6 3 0 728 nginx [19390.557155] [28552] 1000 28552 1572 313 6 3 0 728 nginx [19390.564469] [28553] 1000 28553 1572 328 6 3 0 728 nginx [19390.571908] [28554] 0 28554 7385 3784 17 3 0 728 bundle [19390.579093] [28557] 0 28557 22016 13522 46 3 0 728 bundle [19390.586600] [30170] 89 30170 22105 253 43 3 0 0 pickup [19390.594068] [32111] 0 32111 408 35 4 3 0 728 sh [19390.601250] [ 751] 0 751 133700 129589 260 4 0 728 python3 [19390.608348] [ 759] 0 759 133700 129589 258 3 0 728 python3 [19390.615562] [ 760] 0 760 133700 129589 259 3 0 728 python3 [19390.623013] [ 787] 0 787 133700 129589 260 4 0 728 python3 [19390.630470] [ 799] 0 799 133700 129589 260 4 0 728 python3 [19390.637643] [ 1198] 0 1198 407 43 5 3 0 728 sh [19390.644695] [ 1362] 0 1362 133700 129591 260 4 0 728 python3 [19390.652027] [ 1499] 0 1499 1878 193 9 5 0 -999 docker-containe [19390.659911] [ 1509] 0 1509 2230 126 9 5 0 -999 docker-containe [19390.667447] [ 2022] 0 2022 94315 80506 164 4 0 728 python3 [19390.674564] [ 2023] 0 2023 1171 13 6 3 0 0 systemd-cgroups [19390.682363] [ 2024] 0 2024 2121 13 8 3 0 0 systemd-cgroups [19390.690125] Out of memory: Kill process 1362 (python3) score 862 or sacrifice child [19390.696837] Killed process 1362 (python3) total-vm:534800kB, anon-rss:518364kB, file-rss:0kB, shmem-rss:0kB [19390.807199] oom_reaper: reaped process 1362 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB