why do we use cosine as the expression of vector dot product? When we do vector products, we use two different methods.
One is the vector dot product, another is vector cross product.
The equation of the vector dot product is
$$\textbf A \cdot \textbf B =|\textbf A| | \textbf B| \cos\theta,$$
where $\theta$ is the angle between the vectors $\textbf A$ and $\textbf B$.

Why do we use cosine as the expression?
The dot product of two vectors $A$ and $B$ is just the product of the magnitude of one vector with the scalar projection of the other one on itself. Hence the $cos$ term. Also, note that the $cos$ function is greater for smaller angles, and lesser for larger ones, just like the length of the projection. (both are the same thing, actually.)
It is actually the definition of the dot product of two vectors.