I develop a system (x86_64 based) that runs Linux (Ubuntu 14.04.3) and has several pieces of custom built hardware connected.  I have written drivers and control software for the custom hardware.  After running tests over an extended period of time (several days) on this system I noticed that the amount of free memory (as shown by cat /proc/meminfo and the free command) was steadily declining.  The amount of memory used by my user space processes was well within reason. I  cleared the page cache but it had little effect on the amount of free memory.I rebooted the system and then wrote a script to run a simple test in a loop 500 times, clear the page cache, and then write the output of /proc/meminfo to a file. I then ran this script over several days to collect some data points.  After analyzing the resulting data I noticed that the amount of free memory (MemFree, MemAvailable also) steadily declines in a linear fashion while the other categories remained roughly flat. I collected way too much data to post here, but here are the first and last data points:                       _Run 1_     _Run 650_MemTotal:              65738276    65738276MemFree:               65182220    30881420MemAvailable:          65124632    30824008Buffers:                   2292        2064Cached:                  101204      100816SwapCached:                   0           0Active:                  174772      195008Inactive:                 82924       82444Active(anon):            154304      174712Inactive(anon):           66624       66628Active(file):             20468       20296Inactive(file):           16300       15816Unevictable:                  0           0Mlocked:                      0           0SwapTotal:             66978812    66978812SwapFree:              66978812    66978812Dirty:                      188         184Writeback:                    0           0AnonPages:               154296      180848Mapped:                   99760       99360Shmem:                    66672       66676Slab:                     46836       48352SReclaimable:             17000       18008SUnreclaim:               29836       30344KernelStack:               4176        4128PageTables:                7244        6680NFS_Unstable:                 0           0Bounce:                       0           0WritebackTmp:                 0           0CommitLimit:           99847948    99847948Committed_AS:            433008      417576VmallocTotal:       34359738367 34359738367VmallocUsed:            1886988     1886956VmallocChunk:       34357817344 34357817344HardwareCorrupted:            0           0AnonHugePages:           106496      126976CmaTotal:                     0           0CmaFree:                      0           0HugePages_Total:              0           0HugePages_Free:               0           0HugePages_Rsvd:               0           0HugePages_Surp:               0           0Hugepagesize:              2048        2048DirectMap4k:              93872       93872DirectMap2M:            1894400     1894400DirectMap1G:           67108864    67108864Notice that a tiny bit more than half of the system's 64GB of memory is consumed somehow while there doesn't appear to be a significant increase in any other category.Here are some of the things I've done to try to figure out the issue:I immediately suspected a kernel memory leak probably cause by one of my drivers, so I audited all of my driver code. I didn't find any obvious problems. I also enabled the kernel's kmemleak checker and reran the previously mentioned test script.  It did not find any leaks.Most recently I enabled the page_owner debug feature of the kernel and reran my script while collecting periodic output. After sorting and then diffing the outputs I see a few deltas (by my calculation, the total delta is 10,872 pages or ~42MB) but nothing anywhere near the ~30 GB that is consumed.I am thoroughly stumped and fast running out of ideas. Does anyone have any idea as to what is going on and/or suggestions as to how to figure it out?
How do I figure out what is consuming memory?
linux;linux kernel;memory