I am writing an optimization expression and in the constraints part, I want to limit the number of non-zero entries of the vector to a certain number R. 

Suppose if the vector is M dimensional, then I would like to have R entries to be non-zero and (M-R) entries to be zero. I want to have a vector expression or multiply of add or any function which I can write in the constraint part that ensure certain amount of non-zero entries or certain amount of zero entries. 

I know how to limit the entries to certain number of 1s. If you equate the square of the norm of the vector to the value of R, you would limit the non-zero entries to R provided the vector only contains 0 and 1. However in case I have real numbers and zeros and I want to limit the number of non-zero entries, what could be the vector expression of function ?
['https://math.stackexchange.com/questions/938470', 'https://math.stackexchange.com', 'https://math.stackexchange.com/users/176651/']
You are looking for the 0-norm, which is exactly the number of non-zero elements in a vector. So your constraint looks like

$$
\| x\|\_0 \leq R
$$

However, the term norm here is used loosely, since the 0-norm is not really a norm (it does not satisfy triangle inequality). In fact, the constraint above is non-convex, and problems involving the minimization of 0-norm are NP-hard.

Still, under certain conditions, the 0-norm can be relaxed into 1-norm (which is much easier to deal with), without changing the solution to the optimization problem (see compressive sensing). Whether this applies to your case depends on the structure of your optimization problem.
It is a bit problematic to say that exactly $R$ should be non-zero, because then you have to define first what constitutes non-zero in your model (is $10^{-15}$ nonzero? If so, you will have massive problems to express this as it is essentially numerical noise to a numerical solver)

If you mean *at most* $R$ elements non-zero then it is easily described, when $0\leq x \leq 1$, as $x\leq \delta, \sum \delta \leq R$ where $\delta$ is a binary vector of length $M$.

BTW, strict inequalities are not possible in practice. You would have to write that as a strict inequality using a margin $0\leq x \leq 1-\epsilon$ where $\epsilon$ is a small constant, but large enough not to drown in the general numerical tolerances of the solver (somewhere around $10^-7$ typically)