Ugh this whole thing is so petty, but please hear me out.

So the other day as I was heading home from work, I call up my fiance (let's call him Sam) to ask if he wants me to bring him anything to eat. He tells me that he's not at home and hanging out with some friends- I say hello to everyone (I was on speaker) and say I'll see him later then. Maybe 5 minutes later as I am driving, one of Sam's friends calls me and says something like "Tell Sam to let me see his phone! He was going to show me something on his phone, but I joked that I was going to go through his phone and now he won't let me see!!" Meanwhile, I hear my fiance in the background "Say no!" etc etc. So, obviously they're all being silly and just to poke fun I side with his friend and say stuff like "Let her see it, be nice!" Sam: "WOW, really?? You're so in the doghouse"-Mind you all of this (I thought) was all in fun, and not serious. 

Later that day when Sam gets home, come to find out he is actually legitimately upset that I didn't "back him up"/"take his side". He tells me "When \[friend\] threaten to call you, and told them to go ahead and call because I just knew you'd take my side" I tell him sorry I was just kidding around I didn't think this was a serious situation. Days later (today), he brings it up again saying things about how it hurt his feelings how I automatically didn't take his side, we're supposed to be a team, and he felt that our relationship was disrespected. I (once again) express that I didn't think the situation was that serious, this whole thing is ridiculous, I don't understand why he's so upset about this and I don't know what he wants from me. I also point out that I don't always have to "take his side" because we're allowed to have differing opinions. He claims it's not about always agreeing and that I am disregarding his feelings.  

So am I the asshole here?