For a while, I encounter RAM-shortages on my Debian webserver (VPS/virtual machine). This would not be unusual, if they happend on a regular basis. But they do not. Here's a chart from Munin:                 To solve such riddles, I tracked my system with atop. Here're two snapshots from 7:00AM and 9:00AM - during and after the RAM shortage (using the -m option to see the memory-related information):ATOP - <snip>                           2014/09/10  07:00:02                           ------                             10m0s elapsed<snip>MEM |  tot     2.0G |  free   79.1M  | cache 102.4M  |  dirty   0.1M |  buff   53.2M  |  slab   90.8M |                |               |SWP |  tot     2.0G |  free    2.0G  |               |               |                |               |  vmcom 748.1M  | vmlim   3.0G  |DSK |           sda |  busy      1%  | read     917  |  write   1695 |  KiB/w     13  |  MBr/s   0.01 |  MBw/s   0.04  | avio 1.22 ms  |<snip>  PID     MINFLT     MAJFLT     VSTEXT      VSIZE      RSIZE      VGROW      RGROW     RUID         EUID          MEM     CMD       1/1513717        102         18     10709K     874.5M     206.2M         0K       128K     mysql        mysql         10%     mysqld 4086        166          0       450K     228.1M     21896K         0K         0K     www-data     www-data       1%     apache219131       1659         99       450K     225.5M     19604K     -2652K     -2292K     www-data     www-data       1%     apache2 1469        608          0       450K     222.6M     18508K       256K        64K     www-data     www-data       1%     apache223038        347          0       450K     222.3M     18496K         0K         0K     www-data     www-data       1%     apache2 4085        721          0       450K     222.1M     18308K         0K         0K     www-data     www-data       1%     apache210639        790          0       450K     224.9M     18284K       768K       932K     www-data     www-data       1%     apache219158        199          1       450K     222.1M     18064K         0K        52K     www-data     www-data       1%     apache2 1895        330          0       450K     221.8M     18020K         0K         0K     www-data     www-data       1%     apache2 6661       3346         22       450K     224.0M     17700K      1512K      -780K     www-data     www-data       1%     apache212570        808          0       450K     221.7M     17668K       512K       508K     www-data     www-data       1%     apache219817          0          0       450K     214.5M     15336K         0K         0K     root         root           1%     apache218209       3996          0      2277K     55592K     14728K     55592K     14728K     till         till           1%     python18210       2760          0         4K     43292K     10544K     43292K     10544K     munin        munin          1%     munin-update11976        506          0       149K     18788K      6512K         0K         0K     root         root           0%     atop 1934        175          0         4K     52228K      5852K         0K         0K     root         root           0%     munin-node17993          0          0         4K     67020K      5712K         0K         0K     postgrey     postgrey       0%     /usr/sbin/post 2000          0          0       346K     244.3M      5668K         0K         0K     root         root           0%     rsyslogd14557          0          0      7163K     234.9M      5284K         0K         0K     root         root           0%     php5-fpm14558          0          0      7163K     234.9M      4564K         0K         0K     www-data     www-data       0%     php5-fpm14559          0          0      7163K     234.9M      4564K         0K         0K     www-data     www-data       0%     php5-fpm  328          0          0       134K     572.6M      2932K         0K         0K     root         root           0%     console-kit-da<snip>And...ATOP - vmd1989                           2014/09/10  09:00:02                           ------                             10m0s elapsed<snip>MEM |  tot     2.0G |  free    1.5G  | cache  88.8M  |  dirty   0.1M |  buff   19.2M  |  slab   25.8M |                |               |SWP |  tot     2.0G |  free    2.0G  |               |               |                |               |  vmcom 748.0M  | vmlim   3.0G  |DSK |           sda |  busy      0%  | read     453  |  write   1991 |  KiB/w     12  |  MBr/s   0.01 |  MBw/s   0.04  | avio 1.01 ms  |<snip>  PID     MINFLT     MAJFLT     VSTEXT      VSIZE      RSIZE      VGROW      RGROW     RUID         EUID          MEM     CMD       1/1613717        189          0     10709K     874.5M     206.3M         0K         0K     mysql        mysql         10%     mysqld23038        743          7       450K     222.6M     18620K         0K        40K     www-data     www-data       1%     apache223930        692          0       450K     220.6M     18568K         0K         0K     www-data     www-data       1%     apache228738       4784          0         4K     126.4M     18328K     126.4M     18328K     munin        munin          1%     munin-update26990        392          1       450K     220.5M     18088K         0K       112K     www-data     www-data       1%     apache226552       1150          2       450K     220.3M     17788K       512K       576K     www-data     www-data       1%     apache228744       1443          0         4K     129.1M     17636K     129.1M     17636K     munin        munin          1%     /usr/share/mun27424        602          0       450K     219.8M     17504K         8K       240K     www-data     www-data       1%     apache227000        216          0       450K     219.8M     17308K         8K       104K     www-data     www-data       1%     apache228290       2977          0       450K     219.9M     17200K     219.9M     17200K     www-data     www-data       1%     apache219817         68          0       450K     214.5M     15340K         0K         0K     root         root           1%     apache228287        429          1       450K     215.0M     10384K     215.0M     10384K     www-data     www-data       1%     apache228727        184          0       450K     214.5M      9300K     214.5M      9300K     www-data     www-data       0%     apache228728        191          0       450K     214.5M      9300K     214.5M      9300K     www-data     www-data       0%     apache211976        490          0       149K     18788K      6512K         0K         0K     root         root           0%     atop 1934        428          0         4K     52228K      5852K         0K         0K     root         root           0%     munin-node 2000          0          0       346K     244.3M      5668K         0K         0K     root         root           0%     rsyslogd28745       1036          0         4K     52228K      5580K     52228K      5580K     root         root           0%     munin-node [::14557          0          0      7163K     234.9M      5284K         0K         0K     root         root           0%     php5-fpm17993          0          0         4K     67020K      4844K         0K         0K     postgrey     postgrey       0%     /usr/sbin/post14558          0          0      7163K     234.9M      4564K         0K         0K     www-data     www-data       0%     php5-fpm14559          0          0      7163K     234.9M      4564K         0K         0K     www-data     www-data       0%     php5-fpm  328          0          0       134K     572.6M      2932K         0K         0K     root         root           0%     console-kit-da<snip>Sorry for the long lists - just do not want to miss the cause. Yet, my problem is: I do not see the cause. There is significantly less free memory in the status (top), but no process that would explain why, where is the memory going...Is my thinking incorrect with this?UpdateAccording to Patrick's advice, I collected /proc/meminfo - during a phase of RAM shortage and later. In sake of easy visibility, I put the content into one table:               mem-shortage   a bit laterMemTotal:        2060776 kB    2060776 kBMemFree:          252896 kB    1608532 kB   *Buffers:           15464 kB      12060 kBCached:            71864 kB      62800 kBSwapCached:         4160 kB       4160 kBActive:           268020 kB     253368 kBInactive:         134988 kB     132300 kBActive(anon):     225940 kB     220872 kBInactive(anon):    97296 kB     220872 kB   *Active(file):      42080 kB      32496 kBInactive(file):    37692 kB      29116 kBUnevictable:        6540 kB       6680 kBMlocked:            6540 kB       6680 kBSwapTotal:       2096476 kB    2096476 kBSwapFree:        2081568 kB    2081568 kBDirty:                 0 kB        116 kBWriteback:             0 kB          0 kBAnonPages:        318084 kB     313364 kBMapped:            20692 kB      20408 kBShmem:              4208 kB       9896 kBSlab:              24336 kB      23936 kBSReclaimable:      10252 kB       9316 kBSUnreclaim:        14084 kB      14620 kBKernelStack:        1464 kB       1544 kBPageTables:         8396 kB       9544 kBNFS_Unstable:          0 kB          0 kBBounce:                0 kB          0 kBWritebackTmp:          0 kB          0 kBCommitLimit:     3126864 kB    3126864 kBCommitted_AS:     744764 kB     761812 kBVmallocTotal:   34359738367 kB  34359738367 kBVmallocUsed:         272976 kB       272976 kBVmallocChunk:   34359464431 kB  34359464431 kBHardwareCorrupted:     0 kB          0 kBAnonHugePages:         0 kB          0 kBHugePages_Total:       0             0HugePages_Free:        0             0HugePages_Rsvd:        0             0HugePages_Surp:        0             0Hugepagesize:       2048 kB       2048 kBDirectMap4k:      282560 kB     282560 kBDirectMap2M:     1814528 kB    1814528 kBI only see two signficant (not in the statistical sense) differences, marked with an asterisk (*), but I do not think, they tell me where the RAM went.I also checked for shared memory (as good as I could) ... and found none.# ipcs -m------ Shared Memory Segments --------key        shmid      owner      perms      bytes      nattch     statusI also check for hidden processes using unhide. But except for a false positive (known issue with Debian), there seem not to be any hidden processes.Any more ideas why 1.2 GB RAM are in use - and then not? Could this be another issue caused by the virtual server architecture?UpdateI followed Sergio's hint to consult lsmod and check for memory ballooning. The size column does not tell anything helpful, but there's a process vmw_balloon - so it seems to actually be an issue of shifting memory between the virtual machines. Question answered :)# During high RAM usage (removed middle part)$ lsmod | sort -r -k 2,2nModule                  Size  Used bycrc16                  12343  1 ext4crc_t10dif             12348  1 sd_modlibcrc32c              12426  2 xfs,btrfsmperf                  12453  0ata_generic            12490  0pcspkr                 12632  0vmw_balloon            12657  0           <=ac                     12668  0i2c_piix4              12704  0coretemp               12898  0<snip>reiserfs              193501  0drm                   211856  2 ttm,vmwgfxext4                  381419  1xfs                   628913  0btrfs                 641551  0
Where has my RAM gone & how to interpret atop's memory output?
debian;virtual machine;webserver;ram;atop
Probably your virtual machine is suffering some kind of memory ballooning operation ordered from the virtualization platform. You can try to confirm this by looking for a related module with lsmod (the name changes from one virtualization platform to another, but it should be pretty distinctive).When memory ballooning is enabled, a virtualization Host can move memory resources from one VM to another, when needed. At the request of said Host, the kernel module from the Guest reserves the indicated amount of physical RAM (physical from the viewpoint of OS running on the Guest), to be sure that no other process can make use of it. Then the Host reassigns the real physical resources to another Guest.The effect on the Guest is exactly what you're seeing, a lot of memory used memory with no apparent owner.If you don't have control of that virtualization platform, you should ask your provider for information about the actual configuration of the ballooning parameters for your virtual machine.