In statistics, why do you reject the null hypothesis when the p-value is less than the alpha value (the level of significance) This is a question that I've always wondered in statistics, but never had the guts to ask the professor. The professor would say that if the p-value is less than or equal to the level of significance (denoted by alpha) we reject the null hypothesis because the test statistic falls in the rejection region. When I first learned this, I did not understand why were comparing the p values to the alpha values. After all, the alpha values were brought in arbitrarily. What is the reason for comparing them to the alpha values and where do the alpha values of 0.05 and 0.10 come from? Why does the statement $ p_\text{value} \leq \alpha$ allow you to reject $H_0$?
Here's the idea: you have a hypothesis you want to test about a given population. How do you test it? You take data from a random sample, and then you determine how likely (this is the confidence level) it is that a population with that assumed hypothesis and an assumed distribution would produce such data. You decide: if this data has a probability less than, say $95$% of coming from this population, then you reject at this confidence level--so $95$% is your confidence level. How do you decide how likely it is for the data to come from a given population? You use a certain assumed distribution of the data, together with any parameters of the population that you may know.

A concrete example: You want to test the claim that the average adult male weight is $170 lbs$ . You know that adult weight is normally-distributed, with standard deviation, say, 10 pounds. You say: I will accept this hypothesis, if the sample data I get comes from this population with probability at least $95$% . How do you decide how likely the sample data is? You use the fact that the data is normally-distributed, with (population) standard deviation=$10$, and you assume the mean is $170$ . How do you determine how likely it is for the sample data to come from this population: the $z-$ value you get ( since this is a normally-distributed variable , and a table allows you to determine the probability. 

So, say the average of the random sample of adult male weights is $188lbs$. Do you accept the claim that the population mean is $170$? . Well, the decision comes down to : how likely (how probable) is it that a normally-distributed variable with mean $170$ and standard deviation $10$ would produce a sample value of $188lb$? . Since you have the necessary values for the distribution, you can test how likely this value of $188$ is, in a population $N(170,10)$ by finding its $z-$ value. If this $z-$ -value is less than the critical value, then the value you obtain is less likely than your willing to accept. Otherwise, you accept.
Let’s consider two significant levels ($\alpha$=0.05 and $\alpha$=0.01) for p=0.03.

p=0.03: there is a 3% chance of observing an effect, solely due to the sampling error, when there is actually none.

$\alpha$=0.05: we are up to 5% confident that any observed effect is due to random sampling (or 95% confident that it is not!).

Since we only have 3% chance of observing the effect due to sampling error, we cannot be up to 5% confident that the observed effect is due to random sampling and support $H_0$ at this level. Therefore, we reject $H_0$ at 5% level.

Following the same logic, we are $\alpha$=0.01 confident that the observed effect is random and $H_0$ is true. Therefore, we do not reject $H_0$ at 1% level.

I hope the reader a better understanding of why $H_0$
 is reject when $ p \leq \alpha$

I had the same question and this post too motivated me to write a short piece about it. The example above is taken from there