So this all happened yesterday/today and he says I'm an asshole for not understanding what he is saying nor apologizing. I think I should have nothing to apologize for. 

So backstory a bit I am 19, my friend (we will call Gary) just hit 40 and he looks around in his mid thirties. Gary has had a rough time finding love and he got infatuated with this married women (apparently the married woman is on and off with her husband) and well apparently Gary went for it and she turned him down, because shes married, and he is super down. 

Well yesterday she walked by and to kind of cheer him up I said "dont worry shes ugly anyways" halfway through the work day. Well he has some anxiety and feels everybody hates him and doesnt want to step on relationships, so when I asked if he wanted to come over to watch some South Park because he has no internet, he replied with "I dont want your husband to hate me or get jealous". I said "dont worry when he saw you in person he said he had nothing to worry about because you look like you're thirty" as in like my husband knows I wouldnt date somebody older than me. 

Well today I walked into work and he gave me a huge attitude all day and then said "you called me ugly". I was honestly shook and was taken aback
 He said that both of my statements combined together meant that I was indirectly saying "Gary is so ugly that an ugly girl didnt even want him." He says I should apologize for calling him ugly and 'I'm showing how young I am by not understanding his side'.

I understand I probably worded the 'looked thirty' comment badly  but I just dont understand what I did wrong. When I tried talking to him he literally interrupted me everytime and said 'wow you really show how young you are' and 'just say you're sorry'. Tbh I'm stubborn and wont. 

Am I in the wrong? Am I the asshole? 