If research design, data analysis, and results only are good, where is the line between suggesting major revision and rejection? Disclaimer: I am quite fresh as a researcher, even more as a reviewer.

I am reviewing a paper that appears to report a well-conducted empirical study, with proper data analysis. The results of the article are interesting and relevant for the journal.

However, I am very concerned with how the paper was written. I will provide some of my concerns of the report here:

Besides many grammatical errors - which I forgive as English is not my main language as well - the article is badly structured and often confusing. It fails to clearly state the research questions. No all the hypotheses are well supported by the literature review. It omits several key studies in the literature review. It often makes strong claims without a proper reference. It often fails to quantify stuff, falling to the tedious "a lot", "many", "few", "circa", etc. It is quite mysterious on certain aspects of the research design-e.g., a control group is suddenly mentioned without really clarifying what constituted the control group, and the paper reports that the experiment is not even a controlled experiment.

In sum, for the first sections I have nearly a comment for each of the manuscript paragraphs. I will do my best as a reviewer. I will provide all possible suggestions for improving the manuscript. Whereas I am confident that the results of the study are interesting, and that the research design is sound, I don't know if failing to clearly write the manuscript is a reason for suggesting rejection or for recommending major revisions.

Where is the line in this case? The journal does not clarify this in its reviewer guidelines.

In case of suggesting a major revision, what if the authors still fail to provide a clearly written and organized manuscript in a second round? Could I still request for major revisions?
I would recommend emailing the journal editor to ask for some of these clarifications. There might be certain unwritten policies endemic to that particular journal.
In general, you can ask for multiple rounds of revisions (major or minor) until you are satisfied. Peer review is supposed to be an iterative conversation. However, as I mentioned earlier, there could be certain journal policies regarding the number of revisions which you might get addressed if you talk to the editor.
As a reviewer, if you feel that the article should be rejected, you are free to make that claim. However, personally, I always give at least one chance to a paper (however horrible it is) to redress my comments.