Is it true that every finite dimensional matrix is a bounded linear operator? Is it true that every finite dimensional matrix is a bounded linear operator? 
How to calculate its norm then? Is it related to singular value?

Yes, every operator between finite dimensional vector spaces (which includes all finite matrices) is bounded. If the norm taken on the underlying spaces $\Bbb R^n$ and $\Bbb R^m$ is the usual Euclidean norm, then the induced operator norm on the matrix will be precisely the largest singular value.
In particular, we have
$$
\left(\sup_{\|x\|=1} \|Ax\|\right)^2=
\sup_{\|x\|=1} (Ax)^T(Ax)=
\sup_{\|x\|=1} x^T(A^TA)x
$$
and by the Rayleigh Ritz theorem, this coincides with $\sigma_1^2$.
