What happens when a closed form solution to a research problem is not possible? Say I discover an “interesting topic” (at least from my perspective), and I do all sort of work necessary to formulate and tackle the problem (it is an optimization problem), nearly a month of work, and now I suddenly come to realize that the close-form analytical solution to this problem is not possible (or that the solution is too complicated).

What should I do? Should I continue to solve the problem using a numerical method (using computer simulation tools from Matlab), or should I give up my topic?
Most real world problems don't have closed-form solutions. Somehow, we manage.

You write:


  the close-form analytical solution to this problem is not possible


If the impossibility of a closed-form analytic solution is in itself a new result, then that may well be a publishable result. If you can also find an efficient method to get a numerical (approximate) solution, that's either a heavier-weight first paper (combined with the proof of the non-existence of closed-form solution), or it's a second paper in its own right.

You then wrote:


  OR the solution is too complicated.


If there may be a closed-form analytical solution, but it's just too complicated for you to find, then that's an entirely different matter. In which case, you've got nothing to publish. Just a very hard problem that you can either persevere with, or you can stop working on it for a while, and go do something more promising. There's no harm in stopping working on it for a while. Just keep the problem in the back of your mind, along with a handful of other unsolved problems: and every time you learn a new heuristic, algorithm, or solution pattern (or an enhancement to an old one), then try to apply it to the unsolved problems you've been storing up (kudos to the late Richard Feynman for this).