Do we need to use limits while finding definite integral of a piecewise function? Let $f:[a,b]\to \mathbb{R}$ be a piecewise function such that 
$$f(x) = \left\{ 
\begin{array}{c}
g(x) \hspace{1cm} a\le x<\alpha  \\ 
h(x) \hspace{1cm} \alpha \le x \le b
\end{array}
\right. $$
where $\alpha \in (a,b)$. If we need to find the definite integral
$$\begin{align} \int_{a}^{b} f(x) dx\end{align}$$
Is it equal to 
$$\int_a^{\alpha}g(x)dx + \int_{\alpha}^b h(x)dx$$
or since $g(x)$ in not defined for $\alpha$, do we need to use limit as the upper bound of the first integral approaches $\alpha$ i.e.
$$\lim_{h\to 0^+}\int_a^{\alpha - h}g(x)dx$$
Here is an example where the integral is computed without using any limit.

From the comments I understood that your main concern is that $g(\alpha)$ is not defined and hence we can't talk of the symbol $$\int_{a}^{\alpha}g(x)\,dx$$ You are right here. But we have to talk of $\int_{a}^{\alpha}f(x)\,dx$ and not about integral of $g$ here.
The problem is handled by introducing another function $a(x)$ on $[a,\alpha]$ such that $a(x) = g(x)$ for $x \in [a, \alpha)$ and $a(\alpha) = h(\alpha)$. Now we can talk of the integral $\int_{a}^{\alpha}a(x)\,dx$ and note that we have $f(x) = a(x)$ if $x\in [a, \alpha]$ and $f(x) = h(x)$ if $x \in [\alpha, b]$ and then we can write $$\int_{a}^{b}f(x)\,dx = \int_{a}^{\alpha}a(x)\,dx + \int_{\alpha}^{b}h(x)\,dx$$ provided that both integrals on the right exist.
In reality we are not given functions like $g(x), h(x)$ but rather expressions in $x$ like $x + 1, 2x^{2} + \sin x$ in place of $g(x), h(x)$ and for these functions we do know that they are defined for the intermediate point $\alpha$.

Still suppose that there is a function $f$ which is defined and bounded on $[a, b)$ and we don't have any definition of $f$ for point $b$. Then as per definition of Riemann integral we can't talk about $\int_{a}^{b}f(x)\,dx$. What can we do now? Well we can define $f(b)$ to take any value (as per our wish) and then then talk of $\int_{a}^{b}f(x)\,dx$. As I mentioned in my comments, the values of a function at a finite number of points in an interval do not affect the existence/value of the Riemann integral of the function, hence we are at a liberty to choose any value of $f(b)$ and talk about integral of $f$ on $[a, b]$. This is precisely the way we think about integrals like $$\int_{0}^{1}\cos (1/t)\,dt$$ Limits are used for a different setup which goes by the name of Improper Riemann Integral. This setup is used when either the interval of integration is unbounded or if the function is unbounded on a bounded interval. Examples of both types are $$\int_{0}^{\infty}\frac{\sin x}{x} \,dx = \lim_{t \to \infty}\int_{0}^{t}\frac{\sin x}{x}\,dx,\, \int_{0}^{1}\frac{dx}{\sqrt{1 - x^{2}}} = \lim_{t \to 1^{-}}\int_{0}^{t}\frac{dx}{\sqrt{1 - x^{2}}}$$
