Master thesis advisor gave me an ill-defined question. I worked on it for a year! My master thesis advisor gave me an ill-defined question. His expertise is in mathematical optimization, but he wanted to shift to heuristic optimization so he assigned all his graduate students topics in that field although he actually doesn't know much about it. 

He assigned me a problem. I had a feeling that it was ill-defined. I worked on the problem he assigned me for a year but now the numerical results affirms that I was right. 

I am not sure how to deal with the situation. Should I waste more time trying  to re-formulate the statement of the problem? I feel it's unfair for me to be both the student and the mentor! What is his job, really? If I'm to do everything on my own...

He doesn't know anything about this field he forced his students to work in. 

However, if I quit probably no one is going to believe that it is his fault as a mentor and won't find anyone in the department to write me a recommendation letter to start fresh somewhere else.

So, what are my options now?

Is it advisable to tell the truth to other professors? Will they believe me?
You (and some of the commenters) have a misconception about mathematics, and even research in general. You need to dispel that and present what you have done and learned in a way that satisfies your advisor so that you can finish your degree.
Comments about research in general
If your advisor knew the answer before giving you the problem, then it wouldn't be research. Research is an exploration of the unknown. All research questions are &quot;ill formed&quot; at the start. People notice an anomaly or a missing piece of theory and ask &quot;What in the world is going on here?&quot;. There are not immediate answers. Einstein said “If we knew what it was we were doing, it would not be called research, would it?”.
Since you are exploring the unknown the results could come out to be anything at all. You start with an idea that &quot;might be true&quot;. If you set out to &quot;prove that it is true&quot; then you are doing propaganda and not research. You want to take the attitude that &quot;How do I show whether this is true or not?&quot;. Starting with preconceptions can easily lead you astray into scientific misconduct. Reality is brutal.
And no, in doctoral research you don't need &quot;positive&quot; results. You need to discover the truth. Learning and establishing that some hypothesis is false is just as valuable as knowledge as knowing that it is true.
Comments about mathematics in particular
Mathematical research shares a lot with research in general. It is a search into the unknown to find truth. But establishing that, for example, Fermat's Last Theorem was false would be precisely as valuable as showing that it is true. Possibly even more so, if it led to new insights. But it is what it is, not what you want it to be.
And, again, if your advisor knew the answer to the question posed to you, then you wouldn't be doing research but only, perhaps, confirming something already known.
What to do
You actually have written your solution within your question. You say &quot;I worked on the problem he assigned me for a year now but now the numerical results affirms that I was right&quot;. That is a result. It is precisely what mathematics is about. Write that up. You have been successful at doing mathematics. You haven't failed. You have succeeded. You have insight into a problem. Perhaps you have insight that no one else in the world has at the moment.