Are there US universities that do not need GRE and TOEFL score for admission in PhD in Mathematics? I have completed my Bachelor's degree (3 years) in Mathematics and pursuing MSc(2 years). Currently I am in final year and expect to obtain MSc degree in next May. I am willing to do my PhD (in Mathematics) in USA. 

From the information obtained from my seniors and friends, I found that most of the USA institutions ask for GRE and TOEFL score. But are there universities of USA which do not need GRE and/or TOEFL score for admission in Phd in Mathematics?
The truth of the matter is that while you may find many schools that do not require the GRE, virtually every US school is going to require either the TOEFL or a similar test such as the IELTS.

The reason for this is that one of the important criteria for admission to most US graduate programs is the ability to do coursework and interact with fellow graduate students and faculty in their lingua franca, which in the US is English. Consequently, schools need to ensure that the students they accept have a certain level of proficiency in English before they enroll. (A number of grad school admissions officers have mentioned that they blacklisted candidates from certain countries because of problems related to fraud—candidates would hire other people to conduct telephone interviews with universities in the US, and the actual candidate would come with hardly any English whatsoever.)

So, for most schools, you will find it a requirement to be able to speak the dominant language of instruction—and not just in the United States. (For instance, all students at our university must show the ability to speak either German or English, depending on the program to which they apply.)