Is there any research done on ideal proctoring algorithms? I am a grad student TA and I proctor exams. I want to know, what is the best way to proctor? As in, paths to take while walking, when to stand, when to walk, etc.

Is there any rigorous studies on what catches the most cheating? How does the algorithm change for larger or smaller classrooms?

I am interested in scientific studies, not anecdotes. Are there any rigorous studies for this?
I believe that you can formulate the problem mathematically in a way which can lead to some well-known optimisation problem.

However, I think the mathematical model you described is very restricted and the solution is not likely to work very well in practice, even if you find it. Your model only includes moving and observing a certain part of a room - plain and simple. However, in practice, cheating is prevented not only by someone who "polices" the situation. It is a more complex process, which includes some aspects of ethics, culture, community values, etc. Creating a proper culture within your School / University is much better way of preventing cheating, than walking around students following a super-optimal path prescribed by a supercomputer.

Also, the efficiency of a proctor depends not only on their path through the room, but on a lot of psychological  factors, e.g. how this person is perceived by students. A lazily walking person will not be very efficient in preventing cheating whatever the path is. A really "scary" proctor (with heavy looks, etc) can prevent all cheating sitting on their chair in a corner of the room. There is much more factors to this problem than trajectory only, and that's why I don't think the model you suggest is very useful for this problem.