Okay so for context:  
I'm a freshman in HS, and I have a little brother who we're gonna call "Zam" for this interaction because I don't want to put his name out on the internet, despite it being a very common one.

Simply put, Zam is the kind of person to rage at video games. A lot. A few years ago when we used our PS3 a lot, he broke about 4 controllers in separate fits of rage, and during his fortnite addiction, I heard some loud banging from across the hallway. The next day, he started asking for my controller.

This controller is simply a wired Xbox controller made for people who want to play PC games using a controller but don't want to spend the 60 bucks on a wireless controller that I got for my birthday. I use this controller a lot for the sort of couch co-op or games that have very simple controls (Think GBA emulation, The Binding of Isaac, Super Treasure Arena, etc.).

So I say okay at first, but tell Zam he needs to save money up for a controller of his own. He says he will, and I'm alright with this deal.

That was three months ago, and he has yet to get a new controller, and I've heard him abuse mine more than once.

Eventually, I wanted to play some Isaac with my computer connected to the TV so I could sit on my bed while playing, and asked if I could use the controller once he was done with his round of fortnite. He muttered sure and continued playing. The round ends, and he doesn't give it back. Instead, he just queues up again. I tell him to give it back, and he says "I just started this round, let me finish R\*tard". I'm livid at this point, so I unplug the controller, turn off the xbox, take the controller, and walk out of the room. (This was probably an overreaction, but he used R\*tard specifically against me because I have Asperger's Syndrome). He calls me an asshole for doing that and he attempts to grab me and shove me against a wall. I shove him away, he stumbles, then he runs down to mom, crying, and tells her that I beat him up (He's 12, and he wasn't crying when he got stabbed by a fucking fish hook from his own stupidity). I got grounded for a week on account of "Treating my brother like crap", and he taunted me by using my controller the entire time I was grounded.

I had enough, and I've taken it; every time he asks for it I say no.

He still insists I'm an ass for doing so, and that's the question: Am I The Asshole? I know this is really petty and all, but it would be nice to know.