In our research group we are running a computing server for deep learning with a number of NVIDIA Titan X graphics cards and quite some CPU cores. Given that it is a research lab and we have ~10 people using the machine the load on the CPU/GPU cores is almost always high. I am now in charge showing that the machine is overly used and I can propose hardware upgrades. To make an argument, I want to create a detailed history of the CPU/GPU/MEM usage on the machine. The problem is, I don't know the right tools for the job. Of course, I can do some scripting but I prefer off-the-shelf tools since I am not a system administrator :) For monitoring the CPU/GPU usage I typically use nvidia-smi and htop, but these are not suitable for generating long-time histories.Any recommendations on creating such histories? 
Creating CPU/GPU/Memory load average history
memory;nvidia;cpu;load