What is VM min_free_kbytes

Linux OOM kill why?

. Greetings, fellow administrators. I'm posting in the hope of shedding light on the OOM kills that are plaguing one of my company's machines. I can't decide whether or not they are legitimate OOMs.

It's a Centos 6.x with kernel 2.6.32-279.1.1.el6.x86_64.

Ram 8 GB, CPU athlon2-x4.

The big processes are MySQL and VMware Player 4 with a maximum of 2 and 4 GB permanently allocated RAM plus some overhead. (VMware is being killed for its size.)

Demons other than this 2 are running, but they are very small and very lightly loaded so I don't understand where the remaining 1 / 1.5 gig of RAM would go without counting the huge swap ...

In addition, the kills occur only on, while the backup cron jobs are running at night. (These are simple shell scripts with standard tools that back up some databases and compress some web and mailbox directories.)

Here, for example, it did a mysqldump - and this is the first time that it has happened with this particular task. It happened nearly every time I ran rsyncs or zips on a large directory tree (~ 1 million small files), also in conjunction with a 'page allocation error'. BUT I moved it all to another machine with zfs: After this operation, the killer left me alone for a while.

I hate that after months of juggling the subject, reading and rereading every thread on the internet, I still can't relate the information to my case. There is trade, why doesn't it trade instead of killing? And who takes the whole ram anyway? (There were a few legitimate memory leaks in the beginning and I got them.) Can't be fragmentation either as the failed requests are on the order of zero.

I'll add some data before the actual kill logs:

vm.swappiness = 100 vm.vfs_cache_pressure = 5000 vm.min_free_kbytes = 262144

(I added these to fix the problem. They're probably a bit extreme, but it still runs fine.)

I also experimented with overcommit_memory = 2 in vain. Isn't that supposed to deactivate the killer?

This is the normal memory status of the system. Note that the RAM of vmware counts as cache due to the mmap-ed vmem. By the way, vmware is set so that the VM memory can be reclaimed / exchanged. And it never does.

Total Free Shared Buffers Used Cached Mem: 7800792 7400032 400760 0 61 100 4449196 - / + Buffer / Cache: 2889736 4911056 Swap: 8388600 761588 7627012 SwapCached: 286648 kB Side Tables: 40200 kB CommitLimit: 15409312 kB Committed_BASB Node: 80251260 kB Committed_BAS: 80991260 kB 0, Zone DMA 4 1 1 3 1 1 0 0 1 1 3 Node 0, Zone DMA32 378 1476 2541 1491 328 240 74 28 8 0 0 Node 0, Zone Normal 1555 124 956 1825 659 175 54 31 15 0 0

Finally the OOM:

January 2nd 21:37:38: vmware-vmx called oom-kill: gfp_mask = 0xd0, order = 0, oom_adj = 0, oom_score_adj = 0 January 2nd 21:37:38: vmware-vmx cpuset = / mems_allowed = 0 Jan 2nd 21:37:38: Pid: 19635, comm: vmware-vmx Not Corrupted 2.6.32-279.1.1.el6.x86_64 # 1 Jan 2nd 21:37:38: Call tracking: Jan 2nd 21: 37:38: []? cpuset_print_task_mems_allowed + 0x91 / 0xb0 January 2nd 21:37:38: []? dump_header + 0x90 / 0x1b0 January 2nd 21:37:38: []? __delayacct_freepages_end + 0x2e / 0x30 January 2nd 21:37:38: []? security_real_capable_noaudit + 0x3c / 0x70 January 2nd 21:37:38: []? oom_kill_process + 0x82 / 0x2a0 January 2nd 21:37:38: []? select_bad_process + 0xe1 / 0x120 January 2nd 21:37:38: []? out_of_memory + 0x220 / 0x3c0 January 2nd 21:37:38: []? __alloc_pages_nodemask + 0x89e / 0x940 January 2nd 21:37:38: []? alloc_pages_current + 0xaa / 0x110 January 2nd 21:37:38: []? __get_free_pages + 0xe / 0x50 January 2nd 21:37:38: []? __pollwait + 0xb4 / 0xf0 January 2nd 21:37:38: []? eventfd_poll + 0x7d / 0x80 January 2nd 21:37:38: []? do_sys_poll + 0x29b / 0x520 January 2nd 21:37:38: []? __pollwait + 0x0 / 0xf0 January 2nd 21:37:38: []? Pollwake + 0x0 / 0x60 January 2nd 21:37:38: []? Pollwake + 0x0 / 0x60 January 2nd 21:37:38: []? Pollwake + 0x0 / 0x60 January 2nd 21:37:38: []? Pollwake + 0x0 / 0x60 January 2nd 21:37:38: []? Pollwake + 0x0 / 0x60 January 2nd 21:37:38: []? Pollwake + 0x0 / 0x60 January 2nd 21:37:38: []? Pollwake + 0x0 / 0x60 January 2nd 21:37:38: []? Pollwake + 0x0 / 0x60 January 2nd 21:37:38: []? Pollwake + 0x0 / 0x60 January 2nd 21:37:38: []? read_tsc + 0x9 / 0x20 January 2nd 21:37:38: []? ktime_get_ts + 0xa9 / 0xe0 January 2nd 21:37:38: []? poll_select_set_timeout + 0x8d / 0xa0 January 2nd 21:37:38: []? sys_ppoll + 0x4c / 0x180 January 2nd 21:37:38: []? system_call_fastpath + 0x16 / 0x1b January 2nd 21:37:38: Mem-Info: January 2nd 21:37:38: Node 0 DMA per CPU: January 2nd 21:37:38: CPU 0: hi: 0, btch : 1 usd: 0 January 2nd 21:37:38: CPU 1: hi: 0, btch: 1 usd: 0 January 2nd 21:37:38: CPU 2: hi: 0, btch: 1 usd: 0 2nd January 21:37:38: CPU 3: hi: 0, btch: 1 usd: 0 January 2 21:37:38: Node 0 DMA32 per CPU: January 2 21:37:38: CPU 0: hi: 186, btch: 31 usd: 175 Jan 2nd 21:37:38: CPU 1: hi: 186, btch: 31 usd: 40 Jan 2nd 21:37:38: CPU 2: hi: 186, btch: 31 usd : 180 January 2nd 21:37:38: CPU 3: hi: 186, btch: 31 usd: 152 January 2nd 21:37:38: Node 0 normal per CPU: January 2nd 21:37:38: CPU 0 : hi: 186, btch: 31 usd: 170 January 2nd 21:37:38: CPU 1: hi: 186, btch: 31 usd: 67 January 2nd 21:37:38: CPU 2: hi: 186, btch : 31 usd: 108 Jan 2nd 21:37:38: CPU 3: hi: 186, btch: 31 usd: 63 Jan 2nd 21:37:38: active_anon: 1467089 inactive_anon: 263165 isolated_anon: 64 Jan 2nd 21: 37:38: active_file: 12404 inactive_file: 65792 isolated_file: 96 January 2nd 21:37:38 : unpredictable: 2 dirty: 66080 write back: 1 unstable: 0 January 2nd 21:37:38: free: 73888 slab_reclaimable: 8971 slab_unreclaimable: 10661 January 2nd 21:37:38: assigned: 780904 shmem: 1035969 pagetables: 10566 bounce: 0 January 2nd 21:37:38: Node 0 DMA free: 15688 kB min: 500 kB low: 624 kB high: 748 kB active_anon: 0 kB inactive_anon: 0 kB active_file: 0 kB inactive_file: 0 kB unpredictable: 0 kB isolated (anon): 0 kB isolated (file): 0 kB available: 15284kB mlocked: 0kB dirty: 0kB writeback: 0kB assigned: 0kB shmem: 0kB slab_reclaimable: 0kB slab_unreclaimable: 0kB kernel_stack: 0kB pagetables: 0kB pagetables: 0kB : 0kB writeback_tmsc: 0kB bounce: 0kB writeback_tmsc: Yes January 2nd 21:37:38: lowmem_reserve []: 0 3254 7799 7799 January 2nd 21:37:38: Node 0 DMA32 free: 127608 kB min: 109 180 kB low: 136472 kB high: 163768 kB active_anon: 2209600 kB inactive_anon: 441980 kB active_file: 35784 kB inactive_file: 207948 kB not foreseeable: 8 kB isol t (anon): available: 3333024kB mlocked: 8kB dirty: 211888kB writeback: 0kB assigned: 955900kB shmem: 1376604kB slab_reclaimable: 20276kB slab_unreclaimable: 6460kB kernel_stack: 488kB pagetables] 21:3756:38e 2nd January : 0 0 4545 4545 January 2nd 21:37:38: Node 0 Normally free: 152 256 kB min: 152 456 kB low: 190 568 kB high: 228 684 kB active_anon: 3658 756 kB inactive_anon: 610 680 kB active_file: 13832 kB inactive_file: 55220 kB not predictable: 0 kB isolated (ank): available: 4654080kB mlocked: 0kB dirty: 52432kB writeback: 4kB assigned: 2167716kB shmem: 2767272kB slab_reclaimable: 15608kB slab_unreclaimable: 36184kB : lowmem_reserve []: 0 0 0 0 January 2nd 21:37:38: Node 0 DMA: 4 * 4 kB 1 * 8 kB 1 * 16 kB 3 * 32 kB 1 * 64 kB 1 * 128 kB 0 * 256 kB 0 * 512 kB 1 * 1024 kB 1 * 2048 kB 3 * 4096 kB = 15688 kB January 2nd 21:37:38: Node 0 DMA32: 240 * 4 kB 925 * 8 kB 1553 * 16 kB 748 * 32 kB 253 * 64 kB 152 * 128 kB 56 * 256 kB 28 * 512 kB 6 * 1024 kB 0 * 2048 kB 0 * 4096 kB = 127608 kB January 2nd 21:37:38: Node 0 normal: 1155 * 4th kB 1362 * 8 kB 2962 * 16 kB 860 * 32 kB 330 * 64 kB 113 * 128 kB 23 * 256 kB 18 * 512 kB 9 * 1024 kB 1 * 2048 kB 0 * 4096 kB = 152,380 kB January 2nd 21:37 : 38: 1162322 total page cache pages Jan 2nd 21:37:38: 48040 pages in swap cache Jan 2nd 21:37:38: Exchange cache statistics: add 3148787, delete 3100747, find 1726176/2015458 2 January 21:37:38: Free swap = 7750696kB January 2nd 21:37:38: Total Swap = 8388600kB January 2nd 21:37:38: 2031600 pages RAM January 2nd 21:37:38: 81,402 pages reserved January 21:37:38: 1987047 pages divided January 2nd 21:37:38: 707527 pages not divided January 2nd 21:37:38: [pid] uid tgid total_vm rss cpu oom_adj oom_score_adj name January 2nd 21:37 : 38: [457] 0 457 2673 90 1 -17 -1000 udevd Jan 2nd 21:37:38: [1261] 0 1261 62271 293 0 0 0 rsyslogd Jan 2nd 21:37:38: [1275] 0 1275 11 71 91 0 0 0 mdadm Jan 2 21:37:38: [1284] 81 1284 5382 159 1 0 0 dbus-daemon Jan 2 21:37:38: [1295] 70 1295 6946 160 2 0 0 Avahi daemon Jan 2nd 21:37:38: [1296] 70 1296 6914 27 1 0 0 Avahi daemon Jan 2nd 21:37:38: [1344] 68 1344 6292 346 0 0 0 Jan 2nd 21:37:38 : [1345] 0 1345 4524 152 0 0 0 Haldrunner January 2nd 21:37:38: [1391] 0 1391 5053 87 2 0 0 hald-addon-inpu January 2nd 21:37:38: [1406] 0 1406 6014 103 2 0 0 vmware-usbarbit January 2nd 21:37:38: [1420] 0 1420 16028 119 1 -17 -1000 sshd January 2nd 21:37:38: [1439] 38 1439 7539 146 0 0 0 ntpd Jan 2nd 21:37:38: [1483] 0 1483 29309 135 2 0 0 crond Jan 2nd 21:37:38: [1494] 0 1494 5362 47 2 0 0 atd Jan 2nd 21:37:38: [ 1511] 501 1511 25371 5751 1 0 0 Xvnc Jan 2 21:37:38: [1521] 501 1521 26513 66 0 0 0 sh Jan 2 21:37:38: [1532] 512 1532 21634 461 0 0 0 Xvnc January 2nd 21:37:38: [1541] 501 1541 5006 82 1 0 0 dbus-launch January 2nd 21:37:38: [1542] 501 1542 5382 141 2 0 0 dbus-daemon January 2nd 21:37 : 38: [1546] 501 15 46 5629 283 0 0 0 xfconfd Jan 2nd 21:37:38: [1551] 501 1551 28034 71 3 0 0 gpg agent Jan 2nd 21:37:38: [1561] 501 1561 57668 351 1 0 0 xfce4- Session Jan 2nd 21:37:38: [1565] 501 1565 49604 290 2 0 0 xfsettingsd Jan 2nd 21:37:38: [1567] 501 1567 55385 549 1 0 0 xfwm4 Jan 2nd 21:37:38: [1569] 501 1569 59719 1148 2 0 0 xfce4-panel January 2 21:37:38: [1571] 501 1571 56642 311 1 0 0 Thunar January 2 21:37:38: [1573] 501 1573 81 823 695 3 0 0 xfdesktop Jan 2nd 21:37:38: [1582] 501 1582 56799 339 1 0 0 xfce4-settings- Jan 2nd 21:37:38: [1584] 0 1584 520550 188 1 0 0 console-kit-dae Jan 2nd 21:37:38: [1650] 501 1650 55456 487 3 0 0 Panel-6-Systray Jan 2nd 21:37:38: [1654] 512 1654 26513 65 0 0 0 sh Jan 2nd 21:37 : 38: [1669] 512 1669 5006 68 2 0 0 dbus-launch January 2nd 21:37:38: [1670] 512 1670 5383 135 0 0 0 dbus-daemon January 2nd 21:37:38: [1674] 512 1674 5629 264 3 0 0 xfconfd Jan 2nd 21:37:38: [1680] 512 1680 28034 70 3 0 0 gpg agent Jan 2nd 21:37:38: [1683] 500 1683 2nd 7549 6909 2 0 0 Xvnc Jan 2nd 21:37:38: [1694] 512 1694 57667 346 2 0 0 xfce4 session Jan 2nd 21:37:38: [1699] 512 1699 55386 461 1 0 0 xfwm4 2nd January 21:37:38: [1701] 512 1701 66152 1404 2 0 0 xfce4-panel January 2 21:37:38: [1703] 512 1703 56617 235 0 0 0 Thunar January 2 21:37:38: [ 1705] 512 1705 85537 529 1 0 0 xfdesktop Jan 2 21:37:38: [1707] 512 1707 49604 285 1 0 0 xfsettingsd Jan 2 21:37:38: [1715] 512 1715 56 799 312 0 0 0 xfce4 -settings- January 2nd 21:37:38: [1717] 512 1717 55456 438 3 0 0 Panel-4-Systray January 2nd 21:37:38: [1721] 500 1721 26513 66 0 0 0 sh January 2nd 21:37:38: [1740] 500 1740 5006 68 1 0 0 dbus-start January 2nd 21:37:38: [1741] 500 1741 5383 167 2 0 0 dbus-daemon January 2nd 21:37:38: [1745] 500 1745 5629 275 1 0 0 xfconfd Jan 2nd 21:37:38: [1757] 500 1757 28070 139 3 0 0 gpg agent Jan 2nd 21:37:38: [1758] 0 1758 1542 83 3rd 0 0 pptpd Jan 2nd 21:37:38: [1774] 500 1774 57667 356 0 0 0 xfce4 session Jan 2nd 21:37:38: [1779] 500 1779 55674 785 2 0 0 xfwm4 Jan 2nd 21:37:38: [1781] 500 1781 65790 1363 1 0 0 xfce4-panel Jan 2nd 21:37:38: [1783] 500 1783 82 194 451 0 0 0 Thunar Jan 2nd 21:37 : 38: [1785] 500 1785 85642 813 2 0 0 xfdesktop Jan 2nd 21:37:38: [1790] 500 1790 49604 283 2 0 0 xfsettingsd Jan 2nd 21:37:38: [1800] 500 1800 38863 313 3 0 0 xterm January 2nd 21:37:38: [1807] 500 1807 56798 353 3 0 0 xfce4-settings- January 2nd 21:37:38: [1808] 500 1808 55456 470 1 0 0 Panel-6- Systray Jan 2nd 21:37:38: [1811] 500 1811 27074 69 2 0 0 Bash Jan 2nd 21:37:38: [1823] 0 1823 4704 145 0 0 0 smartd Jan 2nd 21:37:38: [1831] 0 1831 1014 48 2 0 0 mingetty January 2 21:37:38: [1833] 0 1833 1014 48 0 0 0 mingetty January 2 21:37:38: [1835] 0 1835 1014 48 1 0 0 mingetty Jan 2 21:37:38: [1837] 0 1837 1014 48 2 0 0 mingetty Jan 2 21:37:38: [1839] 0 1839 1014 49 3 0 0 mingetty Jan 2 21:37:38: [1843] 0 1843 1014 48 0 0 0 mingetty January 2 21:37:38: [2025] 0 2025 25340 59 2 0 0 vmnet-bridge January 2 21:37:38: [2033] 0 203 3 25333 15 1 0 0 vmnet-netifup January 2nd 21:37:38: [2058] 0 2058 27069 101 0 0 0 vmnet-natd January 2nd 21:37:38: [2060] 0 2060 25333 15 1 0 0 vmnet-netifup January 2nd 21:37:38: [2097] 0 2097 30 105 82 2 0 0 vmware-authdlau January 2nd 21:37:38: [2981] 500 2981 36335 76 0 0 0 su January 2nd 21: 37:38: [2984] 0 2984 27074 233 1 0 0 Bash Jan 2nd 21:37:38: [6347] 500 6347 39207 406 2 0 0 xterm Jan 2nd 21:37:38: [6349] 500 6349 27074 70 0 0 0 Bash January 2nd 21:37:38: [6407] 500 6407 36335 77 0 0 0 su January 2nd 21:37:38: [6410] 0 6410 27074 251 0 0 0 Bash January 2nd 21: 37:38: [6481] 0 6481 57857 154 0 0 0 mysql Jan 2nd 21:37:38: [6911] 0 6911 19820 120 1 0 0 master Jan 2nd 21:37:38: [6914] 89 6914 19889 122 0 0 0 qmgr January 2nd 21:37:38: [6918] 89 6918 19839 141 0 0 0 tlsmgr January 2nd 21:37:38: [17572] 0 17572 103460 2142 3 0 0 Thunar January 2nd 21: 37:38: [21227] 500 21227 38801 594 0 0 0 xterm Jan 2nd 21:37:38: [21229] 500 21229 27074 73 0 0 0 Bash Jan 2nd 21:37:38: [29713] 500 29713 3687 0 214 3 0 0 lftp January 2nd 21:37:38: [32170] 500 32170 38815 184 0 0 0 xterm January 2nd 21:37:38: [32172] 500 32172 27074 77 1 0 0 bash January 2nd 21st : 37: 38: [32189] 500 32189 36335 86 1 0 0 su 2nd Jan 21:37:38: [32197] 0 32197 27074 93 1 0 0 Bash 2nd Jan 21:37:38: [16025] 0 16025 2070 89 3 0 0 pptpctrl Jan 2nd 21:37:38: [16026] 0 16026 5544 108 1 0 0 pppd Jan 2nd 21:37:38: [31174] 0 31174 27073 175 1 0 0 mysqld_safe Jan 2nd 21st : 37: 38: [31909] 27 31909 1143356 587238 1 0 0 mysqld Jan 2nd 21:37:38: [32037] 0 32037 26546 169 0 0 0 mysqld_safe Jan 2nd 21:37:38: [32437] 495 32437 136 524 7673 1 0 0 mysqld Jan 2nd 21:37:38: [32449] 0 32449 26546 169 2 0 0 mysqld_safe Jan 2nd 21:37:38: [368] 493 368 211813 3831 0 0 0 mysqld Jan 2nd 21st : 37: 38: [884] 500 884 27074 310 1 0 0 Bash Jan 2nd 21:37:38: [1065] 501 1065 122130 2881 3 0 0 vmplayer Jan 2nd 21:37:38: [2031] 500 2031 38570 281 0 0 0 xterm January 2nd 21:37:38: [2034] 500 2034 27074 180 0 0 0 Bash January 2nd 21:37:38: [2051] 500 2051 36335 140 0 0 0 see 2nd January 21:37:38: [2055] 0 2055 27074 181 2 0 0 Bash 2nd January 21:37:38: [16591] 501 16591 77851 712 3 0 0 vmware-unit -he January 2 21:37:38: [16803] 0 16803 26883 237 1 0 0 January 2 21:37:38: [19635] 501 19635 1693624 793343 1 0 0 vmware-vmx January 2 21:37 : 38: [2186] 0 2186 38139 158 0 0 0 proftpd January 2nd 21:37:38: [5289] 500 5289 38992 979 3 0 0 xterm January 2nd 21:37:38: [5291] 500 5291 27074 188 2 0 0 Bash January 2nd 21:37:38: [5344] 500 5344 36335 148 1 0 0 su January 2nd 21:37:38: [5361] 0 5361 27074 350 0 0 0 Bash January 2nd 21:37 : 38: [18529] 500 18529 26514 227 0 0 0 mysql-workbench January 2nd 21:37:38: [18534] 500 18534 26514 226 1 0 0 catchsegv January 2nd 21:37:38: [18536] 500 18536 26514 77 2 0 0 catchsegv January 2nd 21:37:38: [18537] 500 18537 227088 7571 2 0 0 mysql-workbench January 2nd 21:37:38: [409] 0 409 131 527 1556 1 0 0 geany 2nd. January 21:37:38: [410] 0 410 2054 92 2 0 0 gnome-pty-helpe January 2 21:37:38: [411] 0 411 27074 238 1 0 0 Bas h January 2nd 21:37:38: [5750] 0 5750 2672 92 0 -17 -1000 udevd January 2nd 21:37:38: [5753] 0 5753 2672 87 0 -17 -1000 udevd January 2nd 21: 37:38: [5788] 0 5788 10640 594 3 0 0 openvpn January 2nd 21:37:38: [5792] 0 5792 10640 598 3 0 0 openvpn January 2nd 21:37:38: [5800] 99 5800 11135 587 3 0 0 openvpn January 2nd 21:37:38: [21552] 0 21552 110137 2111 1 0 0 httpd January 2nd 21:37:38: [21555] 48 21555 139593 7684 3 0 0 httpd January 2nd 21: 37:38: [21558] 48 21558 140002 8513 3 0 0 httpd Jan 2nd 21:37:38: [23283] 497 23283 9846 193 1 0 0 dkim-filter Jan 2nd 21:37:38: [23284] 497 23284 33979 524 1 0 0 dkim-filter January 2 21:37:38: [6819] 0 6819 2070 152 3 0 0 pptpctrl January 2 21:37:38: [6820] 0 6820 5544 237 1 0 0 pppd 2 Jan 2nd 21:37:39: [17208] 48 17208 112903 4566 0 0 0 httpd Jan 2nd 21:37:39: [17209] 48 17209 138359 5895 0 0 0 httpd Jan 2nd 21:37:39: [17210 ] 48 17210 138693 7341 3 0 0 httpd Jan 2nd 21:37:39: [1255] 0 1255 24571 713 0 0 0 sshd Jan 2nd 21:37:39: [1278] 0 1278 13874 396 1 0 0 SFTP server Jan 2nd 21:37:39: [14064] 48 14064 138202 6622 3 0 0 httpd Jan 2nd 21:37:39: [14065] 48 14065 139625 7776 1 0 0 httpd Jan 2nd 21:37:39: [16899] 48 16899 138543 7523 3 0 0 httpd January 2nd 21:37:39: [32639] 89 32639 19924 722 0 0 0 pickup 2.January 21:37:39: [4973] 48 4973 136179 4973 3 0 0 httpd January 2 21:37:39: [4976] 48 4976 138478 7371 0 0 0 httpd January 2 21:37:39: [4977] 48 4977 136173 4777 3 0 0 httpd January 2nd 21:37:39: [5662] 0 5662 35030 336 0 0 0 crond January 2nd 21:37:39: [5663] 0 5663 2297 282 2 0 0 sh 2nd January 21:37:39: [5664] 0 5664 2298 302 0 0 0 Bash January 2nd 21:37:39: [5665] 0 5665 15910 437 0 0 0 mutt January 2nd 21:37:39: [5947] 0 5947 2298 337 1 0 0 Bash Jan 2nd 21:37:39: [6416] 48 6416 110 170 2070 3 0 0 httpd Jan 2nd 21:37:39: [6625] 48 6625 110 170 1895 0 0 0 httpd 2nd Jan 21:37:39: [6642] 0 6642 32679 1632 2 0 0 mysqldump Jan 2nd 21:37:39: Insufficient memory: Kill process 19635 (vmware-vmx) with 199 or sacrifice the child Jan 2nd 21st : 37: 39: Aborted process 19635, UID 501, (vmware-vmx) total-vm: 6774496 kB, anon-rss: 74020 kB, file-rss: 3099352 kB