(Sorry about formatting, I'm on mobile)
I'm a first year student in college, so I am living in a dorm with my roommate. When we first got to college, we made a roommate agreement form. At the time, we were both single and what not. One of the most relevant points on this form was something along the lines of "If you want to fuck anyone, let me know so I can either leave or not walk in on it."

Fast forward one month, and my roommate now has a girlfriend. One night, they are hanging out in the room, and I decide to go grab something from the dining hall. I tell them that I will be back in 5 to 10 minutes. I get my food and come back (I had my headphones in to listen to music), and when I come back, they were in the middle of sex. I tell them to text me when they are done so I can go back into my room, and leave. After they are done, I ask them to text me so that way I don't have to walk in on it again.

The next week, I come back from class, and they were fucking again, and yet again I had no text telling me not to come home. 

This continues for almost a month until one time, I was unable to finish my homework, since I left my backpack in the room. At that point, I decide to lay down the law. I tell them that if they do that again, I will not allow her to be in our room anymore. She technically needed both my roommate's and my consent to be there anyways, so I had that power.

That same night, I am trying to fall asleep around 1:30 AM. I shift a bit and one of my headphones falls out (I sleep with headphones for silence). I then hear moans and "daddy". I ignore it for the night, but the next day, I sit them down and have a talk. I tell them that she is no longer allowed in our room, since they repeatedly didn't follow the one thing I asked. They could have asked me to leave or texted me, and I wouldn't have cared, but they never did. Now they are mad at and they say I'm an asshole. Am I the asshole?