My machine has 4GB of ram and 4GB of swap.  The amount of usable memory is about 3GB.  The machine is running an iscsi target, squid, munin, unbound, and other services.However, the memory usage indicates that most of the 3GB being used is for buffers/cache.  If that is the case, why does the system still throw an OOM and not reduce the amount used by buffers / cache?  Furthermore, can't swap be utilized if needed (it is barely touched):If I echo 3 into vm.drop_caches, the number if buffers / caches drops significantly and I don't get any OOM errors for a while.I perceive the problem to be from lack of freeing up buffers / caches as well as not addressing swap space that is available.              total        used        free      shared  buff/cache   availableMem:           3502         169          29           9        3303        3274Swap:          4095          11        4084[Sat Nov 12 22:20:03 2016] Out of memory: Kill process 19741 (squid) score 3 or sacrifice child[Sat Nov 12 22:20:03 2016] Killed process 19742 (log_file_daemon) total-vm:12500kB, anon-rss:136kB, file-rss:1204kB, shmem-rss:0kB[Sat Nov 12 22:20:03 2016] tcp invoked oom-killer: gfp_mask=0x27080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), order=2, oom_score_adj=0[Sat Nov 12 22:20:03 2016] tcp cpuset=/ mems_allowed=0[Sat Nov 12 22:20:03 2016] CPU: 0 PID: 20616 Comm: tcp Not tainted 4.7.4-hardened #1[Sat Nov 12 22:20:03 2016] Hardware name: Dell Inc.                 OptiPlex GX620               /0HH807, BIOS A07 03/31/2006[Sat Nov 12 22:20:03 2016]  0000000000000286 0000000000000286 ffff8800642a3b10 ffffffff812d1bbe[Sat Nov 12 22:20:03 2016]  0000000000000000 ffff8800642a3cb0 ffff8800d4799e00 ffff8800642a3b78[Sat Nov 12 22:20:03 2016]  ffffffff811a57ca ffff8800ab41b200 0000000000000015 0000000000000206[Sat Nov 12 22:20:03 2016] Call Trace:[Sat Nov 12 22:20:03 2016]  [<ffffffff812d1bbe>] dump_stack+0x4e/0x79[Sat Nov 12 22:20:03 2016]  [<ffffffff811a57ca>] dump_header+0x5e/0x1e7[Sat Nov 12 22:20:03 2016]  [<ffffffff8114981d>] oom_kill_process+0x95/0x34d[Sat Nov 12 22:20:03 2016]  [<ffffffff81149eeb>] out_of_memory+0x3a1/0x3ca[Sat Nov 12 22:20:03 2016]  [<ffffffff8114e163>] __alloc_pages_nodemask+0x93a/0xaaa[Sat Nov 12 22:20:03 2016]  [<ffffffff8114e312>] __alloc_pages_node+0x3f/0x54[Sat Nov 12 22:20:03 2016]  [<ffffffff8114e517>] alloc_kmem_pages_node+0x2d/0x7f[Sat Nov 12 22:20:03 2016]  [<ffffffff8107a097>] copy_process.part.48+0x191/0x1f9e[Sat Nov 12 22:20:03 2016]  [<ffffffff8107c07c>] _do_fork+0xd7/0x2db[Sat Nov 12 22:20:03 2016]  [<ffffffff8107c351>] sys_clone+0x39/0x50[Sat Nov 12 22:20:03 2016]  [<ffffffff81001a6f>] do_syscall_64+0x55/0x74[Sat Nov 12 22:20:03 2016]  [<ffffffff81733913>] entry_SYSCALL64_slow_path+0x25/0x25[Sat Nov 12 22:20:03 2016] Mem-Info:[Sat Nov 12 22:20:03 2016] active_anon:13586 inactive_anon:19647 isolated_anon:24                        active_file:666416 inactive_file:138007 isolated_file:0                        unevictable:0 dirty:1007 writeback:1 unstable:0                        slab_reclaimable:36881 slab_unreclaimable:4204                        mapped:6840 shmem:2218 pagetables:1185 bounce:0                        free:8281 free_pcp:0 free_cma:0[Sat Nov 12 22:20:03 2016] Node 0 DMA free:13984kB min:32kB low:44kB high:56kB active_anon:24kB inactive_anon:260kB active_file:472kB inactive_file:528kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:40kB slab_reclaimable:368kB slab_unreclaimable:52kB kernel_stack:16kB pagetables:12kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4 all_unreclaimable? no[Sat Nov 12 22:20:03 2016] lowmem_reserve[]: 0 3486 3486 3486[Sat Nov 12 22:20:03 2016] Node 0 [Sat Nov 12 22:20:03 2016] DMA32 free:19140kB min:7528kB low:11088kB high:14648kB active_anon:54320kB inactive_anon:78328kB active_file:2665192kB inactive_file:551500kB unevictable:0kB isolated(anon):96kB isolated(file):0kB present:3643928kB managed:3570484kB mlocked:0kB dirty:4028kB writeback:4kB mapped:27360kB shmem:8832kB slab_reclaimable:147156kB slab_unreclaimable:16764kB kernel_stack:3712kB pagetables:4728kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4 all_unreclaimable? no[Sat Nov 12 22:20:03 2016] lowmem_reserve[]:[Sat Nov 12 22:20:03 2016]  0[Sat Nov 12 22:20:03 2016]  0[Sat Nov 12 22:20:03 2016]  0 0[Sat Nov 12 22:20:03 2016] Node 0 DMA: 26*4kB (UMEH) 17*8kB (UMEH) 23*16kB (UMEH) 14*32kB (MEH) 10*64kB (UMH) 4*128kB (UMH) 2*256kB (EH) 2*512kB (ME) 2*1024kB (EH) 0*2048kB 2*4096kB (M) = 13984kB[Sat Nov 12 22:20:03 2016] Node 0 [Sat Nov 12 22:20:03 2016] DMA32: [Sat Nov 12 22:20:03 2016] 2313*4kB [Sat Nov 12 22:20:03 2016] (UME) [Sat Nov 12 22:20:03 2016] 1248*8kB [Sat Nov 12 22:20:03 2016] (UME) [Sat Nov 12 22:20:03 2016] 0*16kB [Sat Nov 12 22:20:03 2016] 0*32kB [Sat Nov 12 22:20:03 2016] 0*64kB [Sat Nov 12 22:20:03 2016] 0*128kB [Sat Nov 12 22:20:03 2016] 0*256kB [Sat Nov 12 22:20:03 2016] 0*512kB [Sat Nov 12 22:20:03 2016] 0*1024kB [Sat Nov 12 22:20:03 2016] 0*2048kB [Sat Nov 12 22:20:03 2016] 0*4096kB [Sat Nov 12 22:20:03 2016] = 19236kB[Sat Nov 12 22:20:03 2016] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB[Sat Nov 12 22:20:03 2016] 809484 total pagecache pages[Sat Nov 12 22:20:03 2016] 2843 pages in swap cache[Sat Nov 12 22:20:03 2016] Swap cache stats: add 3997, delete 1154, find 3819274/3819343[Sat Nov 12 22:20:03 2016] Free swap  = 4182696kB[Sat Nov 12 22:20:03 2016] Total swap = 4194300kB[Sat Nov 12 22:20:03 2016] 914981 pages RAM[Sat Nov 12 22:20:03 2016] 0 pages HighMem/MovableOnly[Sat Nov 12 22:20:03 2016] 18383 pages reserved[Sat Nov 12 22:20:03 2016] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name[Sat Nov 12 22:20:03 2016] [ 2521]     0  2521     5671      400      13       3       38         -1000 udevd[Sat Nov 12 22:20:03 2016] [ 2789]     0  2789    23964       64      14       3       11             0 lvmetad[Sat Nov 12 22:20:03 2016] [ 3259]     0  3259     8535       21      19       3      127             0 syslog-ng[Sat Nov 12 22:20:03 2016] [ 3260]     0  3260    49135      818      31       3      135             0 syslog-ng[Sat Nov 12 22:20:03 2016] [ 3328]     0  3328     3240      430      11       3      103             0 crond[Sat Nov 12 22:20:03 2016] [ 3644]   103  3644     3315      501      10       3       14             0 dbus-daemon[Sat Nov 12 22:20:03 2016] [ 3676]     0  3676   140206     1988      55       3      162             0 NetworkManager[Sat Nov 12 22:20:03 2016] [ 3730]     0  3730     2610        0      10       3       33             0 atd[Sat Nov 12 22:20:03 2016] [ 3746]   104  3746    90640     1097      37       3      561             0 polkitd[Sat Nov 12 22:20:03 2016] [ 3748]     0  3748     7529     1035      21       3      107             0 wpa_supplicant[Sat Nov 12 22:20:03 2016] [ 3888]     0  3888     2997      464      11       3      606             0 haveged[Sat Nov 12 22:20:03 2016] [ 3922]     0  3922    12795     2711      29       3      936             0 munin-node[Sat Nov 12 22:20:03 2016] [ 4006]     0  4006     4196     2125      12       3       10             0 dhclient[Sat Nov 12 22:20:03 2016] [ 4145]     0  4145     2369       31       9       3        0             0 rngd[Sat Nov 12 22:20:03 2016] [ 5773]     0  5773     3027      457      10       3        0             0 agetty[Sat Nov 12 22:20:03 2016] [ 5774]     0  5774     3026      437      11       3        0             0 agetty[Sat Nov 12 22:20:03 2016] [ 5775]     0  5775     3026      424      11       3        2             0 agetty[Sat Nov 12 22:20:03 2016] [ 5776]     0  5776     3026      435      11       3        0             0 agetty[Sat Nov 12 22:20:03 2016] [10460]  1000 10460     3805       86      12       3        0             0 ssh-agent[Sat Nov 12 22:20:03 2016] [14256]   113 14256   282464     1904      47       5        0             0 privoxy[Sat Nov 12 22:20:03 2016] [14290]   113 14290   259927     1876      39       4        0             0 privoxy[Sat Nov 12 22:20:03 2016] [14761]     0 14761    69836     1005      46       4        4         -1000 tgtd[Sat Nov 12 22:20:03 2016] [14763]     0 14763     2691       42      11       3        0             0 tgtd[Sat Nov 12 22:20:03 2016] [15840]   106 15840     4471     2188      13       3       10             0 dhcpd[Sat Nov 12 22:20:03 2016] [16782]   501 16782    24486      220      18       3        0             0 stunnel[Sat Nov 12 22:20:03 2016] [16788]     0 16788     6045      562      16       3        0         -1000 sshd[Sat Nov 12 22:20:03 2016] [18585]   118 18585    29712     4475      29       3       23             0 unbound[Sat Nov 12 22:20:03 2016] [20456]     0 20456     3026      451      11       3        0             0 agetty[Sat Nov 12 22:20:03 2016] [22476]     0 22476     3805       87      10       3        0             0 ssh-agent[Sat Nov 12 22:20:03 2016] [24989]     0 24989    14459      987      33       3        0             0 sshd[Sat Nov 12 22:20:03 2016] [24991]  1000 24991    14497      683      32       3        0             0 sshd[Sat Nov 12 22:20:03 2016] [24992]  1000 24992    14459      582      32       3        0             0 sshd[Sat Nov 12 22:20:03 2016] [26146]     0 26146    14459      967      33       3        0             0 sshd[Sat Nov 12 22:20:03 2016] [26152]     0 26152    14459      643      30       3        0             0 sshd[Sat Nov 12 22:20:03 2016] [ 7433]     0  7433    12775     1601      30       3        0             0 cupsd[Sat Nov 12 22:20:03 2016] [ 9717]     0  9717    14459     1201      34       3        0             0 sshd[Sat Nov 12 22:20:03 2016] [ 9721]  1000  9721    14459      736      32       3        0             0 sshd[Sat Nov 12 22:20:03 2016] [ 9722]  1000  9722     7638     1372      19       3        0             0 zsh[Sat Nov 12 22:20:03 2016] [ 9899]   107  9899    13293     1473      28       3        0             0 squid[Sat Nov 12 22:20:03 2016] [17193]   112 17193    29139     1167      37       3        0             0 minidlnad[Sat Nov 12 22:20:03 2016] [19741]   107 19741    19303     6751      40       3        0             0 squid[Sat Nov 12 22:20:03 2016] [20415]     0 20415     7439      571      19       3       87             0 crond[Sat Nov 12 22:20:03 2016] [20420]   177 20420     2815      671      11       3        0             0 sh[Sat Nov 12 22:20:03 2016] [20423]   177 20423     2814      656      10       3        0             0 munin-cron[Sat Nov 12 22:20:03 2016] [20424]   177 20424     1447      427       8       3        0             0 logger[Sat Nov 12 22:20:03 2016] [20425]   177 20425    37465     6280      61       3        0             0 munin-update[Sat Nov 12 22:20:03 2016] [20440]   177 20440    38644     5840      63       3        0             0 /usr/libexec/mu[Sat Nov 12 22:20:03 2016] [20441]     0 20441    12795     2639      28       3      858             0 /usr/sbin/munin[Sat Nov 12 22:20:03 2016] [20616] 65534 20616     2408      600      10       3        0             0 tcp[Sat Nov 12 22:20:03 2016] [20617] 65534 20617     2408       62       7       3        0             0 tcp[Sat Nov 12 22:20:03 2016] Out of memory: Kill process 19741 (squid) score 3 or sacrifice child[Sat Nov 12 22:20:03 2016] Killed process 19741 (squid) total-vm:77212kB, anon-rss:17284kB, file-rss:9640kB, shmem-rss:80kB
how to prevent OOM, use swap
linux;out of memory