How do I judge the novelty of my paper in the area of algorithms and system design? I have been working on a research project and have submitted the paper to two different venues(not simultaneously), and both times I have gotten reviews which criticize the paper for a lack of novelty. What exactly makes an algorithm or system design novel? How can I judge the novelty of my solution?
The novelty of a scientific result is defined in terms of its relationship to previously published results.  In order to perform such a comparison you need:


to have a good idea of what related works have been published by the community, typically obtained by some combination of reading papers and attending conferences, and 
to compare your your system against the most similar prior systems and demonstrate its quantitative or qualitative superiority (e.g. "my system is three times faster" or "my system can process widgets AND frobs, and all previous systems could only do one or the other").


The "amount" of novelty is then how much better you are, relative to the interests of the community, and is very community dependent: a 2% improvement in the cost of manufacturing concrete is worth billions of dollars, while a 2% improvement in the speed of a personal computer program is generally unnoticeably tiny.