Is it wise to choose a better journal with lesser number of similar papers? I need to choose between two journals for my article. Both candidates are, in principle, suitable for the topic. One of them has a (considerably) larger number of articles similar to mine but the other one is a better journal in my opinion. The first one is a journal of both research articles and educational articles. Whereas the second one is a journal of purely new research.

My article is about the re-derivation of a very well-known result using fewer axioms. Thus, it is new research but might be more appreciated by a community engaged in both research and education. So, I am confused as to which one to choose. Personally, if both the journals were ready to accept my article, then I would choose the second one (the one with only research articles) for its more serious tone and thus, for the possibility of my article getting more serious attention.

But I think I need to make a choice incorporating the chances of acceptance as well.
Ask colleagues more experienced than you, ideally colleagues with experience of submitting to those journals, what they think of your chances.  

If there is a question of whether your article fits in the scope of a journal, you could also ask an editor whether they would welcome a submission such as yours.

Usually, though, it will end up being a judgement call. You could, of course, submit to your top choice, and then go with the other one if your paper gets rejected. This is quite commonly done, even if the top journal is a long shot. However, this takes a lot of time and effort, and in my opinion it is often not as good a use of your time as getting something punished quickly and moving on to new work.