Throwaway account

So in my friendship group there are 5 of us, one of them who we shall call "N" has slipped away and has become the odd one out of our group since he moved away. He isn't involved in our group chats on facebook and I have a feeling that the other 3 in my friend group don't really like him either. One of my other friends in the friend group is "J". J and I used to go to local boxing matches together for about a year, and we really enjoyed it. Now I'll admit, I've always had a soft spot for N and I let him take advantage of me sometimes, but I still like him as a friend, whereas J doesn't hate him, but finds him to be cringey and intrusive at times. J and N used to be really close, but like everyone else in the friendship group, J has drifted away from N recently. J made it clear to me that he didn't want N to start coming to the local boxing matches with us, but I decided to invite him along anyway as I still like N, a lot.. In fact, I've started gaining romantic feelings for N but he doesn't know, as I'm gay and he's straight, but I really like spending time with him anyway.

J ended up getting a bit upset with me, because he felt like I ruined the boxing matches by inviting N with us, because N isn't even really into boxing and it feels kind of awkward for J when he is there. We also have to take N home all the time as he doesn't drive, which adds about an extra 1.5 hours to the end of our night, which is when J and I used to hang out and watch cartoons at his place.

J is still coming to the boxing and likes it, but AITA for inviting N along when J told me he wasn't cool with it, or should I have a right to invite whoever I like?