I got accepted into a math PhD program but I don't feel adequate enough to attend I got accepted into a school that I really wanted to go to for a math PhD. I finished my masters and I love math very much. It's something that I've always been extremely passionate about. I used to always go for representation theory seminars with an old professor that I did research with (we mainly did work in certain parts of representation theory) and I thoroughly enjoyed them even if I couldn't fully understand. 

But I suck at math. I think I only was accepted because I'm a woman and Indian. Everyone is so smart in all my classes. During my undergraduate degree I was amongst the top of the class but once I got to a master's, I was at the bottom of the barrel. I can't understand things the way everyone else does. Everyone is always able to engage in the lesson and it takes me 4 hours just to understand the smallest thing, it's depressing. 

I still got A's in almost all my classes but it wasn't hard to get an A in the classes. It's not like I deserved it and it probably deceived admissions officers. 

What makes me even more depressed is that on standardized tests, I can score in the 95th percentile in english/verbal without ever cracking open a book. But it's just not the same in math, I have to try so hard and even still I don't do nearly as well as I would like. 

My strengths aren't in math, I am very good with english, history and philosophy. My professors from my undergraduate degree would always encourage me to pursue these fields but I love math so much more even though I'm not good at it. I just don't know what to do, I enjoy those subjects recreationally but not something I want to pursue full-time right now. I would love to get a second master's in philosophy but it's not my priority and if I never got it, I think I'd be fine just reading the literature on my own. 

Anyways, even the most basic math facts I will forget. It's embarrassing to be a PhD student and forget things so easily. I don't even want to make friends because I'm scared they will start talking about math and I won't know what they're saying

Should I just quit?
First off, congratulations on being accepted to the PhD. That means the faculty have carefully evaluated your application materials and decided that you are a promising young mathematician. They are experts and they think you have what it takes to finish a PhD.

No one gets admitted just because of their gender and/or race. It would be a waste of the department's resources to admit unqualified students and it would be a disservice to those students -- setting them up to fail.

It sounds like you have Impostor Syndrome. The truth is that almost everyone, from time to time, feels like they don't belong. I have felt that way at many points.

No, you should not "just quit". You have a great opportunity in front of you. It sounds like you have other things that you want to consider doing. Perhaps you will decide that those other opportunities are better for you. But, if you do turn down the PhD, you should not think of it as quitting. You should think of it as taking a positive step in another direction.

That said, academia is hard. And I would not advise anyone to do a PhD unless they really enjoy research.

You should think seriously about what you want to do, as it sounds like you are uncertain about where your best option lies. Talk to people you know personally, as they can best advise you. 

One thing to note is that a PhD should give you some flexibility to study other things or, better yet, to combine different research topics. My PhD program allowed — required, in fact — me to take graduate courses in totally different fields.