Should I mention professors in a Masters application? I am wondering if I am supposed to mention in my statement of purpose/research objective/reason for the masters program that I like the research of Prof X.

Lets assume for example a Master in Computational Science Engineering. The area is relatively specific: stochastic, machine learning and a bit of other stuff. Should I still mention a professor and his work, or does it not matter at all for the application? 

However, if I would apply for a "regular" masters program where I have to choose a department that I prefer, should I mention a professor and some of his work in this case?
You can certainly mention a professor and his/her research in your statement of purpose as long as it is genuine.  This can show that you are a) familiar with the school/the department and its faculty and b) you have a good understanding of the field and your interests within it.  There is certainly nothing wrong with saying 'One of the reasons that I chose this school is because Prof. X's work on XYZ is very interesting, and I would love the opportunity to learn from him/her and his/her work.'  Don't, however, try to name-drop for the sake of it.  

You should also make sure that you do a little research to make sure that that professor is actually still going to be doing work during your time at that school.  It looks kind of bad if you say 'I really want to work with Prof. A,' but Prof A is going to be on sabbatical or something like that.  This kind of information is not always readily available.

In short, don't be afraid of mentioning a prof if you really mean it and are really excited, but don't look up someone from each school you are applying to for the sake of putting a name in your paper.