Q:

mongod non-mapped virtual memory doubled suddenly, has not changed since

The non-mapped virtual memory stat for our mongo primary has always been constant, and we never gave it much thought before yesterday.  Yesterday, a series of accidental full-collection scans from a poorly designed query resulted in a big slowdown, where the mongod process was using 100% CPU, and every query was taking tens of seconds.
After offloading the offending query to our secondaries, the performance problems disappeared, but the non-mapped virtual memory more than doubled, and hasn't gone down since.  It used to hold at about 600MB; now it's at about 1.4GB.  The increase was immediate, and exactly correlates to the slowdown, and it hasn't changed at all since.
The number of connections has been completely constant throughout, so we can be sure it isn't that.
What might cause this? Is it a problem?  Should we be concerned?
Running on Ubuntu 12.04 64-bit on an EC2 instance.

A:

The answer is it depends but it doesn't appear to be a problem and you shouldn't be too concerned if this is the only behaviour you have seen.
The storage level in MongoDB use memory mapped files so the total virtual memory used can be the size of all your the DB data on the disk.
The resident memory on the system will represent the actual working set used by MongoDB and depending on your usage / access patterns, it will typically grow over time to the total physical RAM on your host. The OS will only page out this data when it sees a need otherwise it will simply stay put (it uses a Least Recently Used / LRU and Least Frequently Used / LFR approach on the data). The virtual memory usage will grow in tandem with this at a little over twice the rate of resident usage as MongoDB maintains two views of resident memory in the virtual memory space.
When MongoDB is restarted it begins from scratch and given a specific time frame for your working data set with regular, predictable usage you should get a good representation of what your working set size is in terms of RAM.
MongoDB's docs contain some further information that help describe how it uses memory:
http://docs.mongodb.org/manual/faq/storage/#faq-storage-memory-mapped-files
A deeper technical view of MongoDB's storage layer is presented here:
http://www.mongodb.com/presentations/understanding-mongodb-storage-performance-and-data-safety

A:

Because virtual memory is effectively free, nobody bothers to clean it up or minimize its usage. So long as the resident set size is reasonable, I wouldn't worry about it.

