A silly question with unprovability By Gödel's incompleteness theorems, we can get a true but unprovable sentance $\psi$. However, we know it is true since its falsity implies contradiction. Then, why couldn't we accept this "proof" since it shows that $\psi$ must be true? Does it mean that the law of excluded middle could not be written as an axiom of finite length?

We start with an appropriate theory $T$ and make the Gödel sentence $G_T$. When we argue that $G_T$ is true, we do not make that argument within $T$. We generally need to assume something beyond $T$ - such as "$T$ is consistent" - in order to show that $G_T$ is true.  But, if $T$ is an appropriate theory, the incompleteness theorems show that $T$ does not prove "$T$ is consistent" and so this argument can't work within $T$. 
So the real issue with the phrase in the question is not with the word "true", the issue is with the word "unprovable". In the motto "true but unprovable", the term "true" refers to the standard model, while the phrase "unprovable" only means "unprovable in $T$", not "unprovable in any system whatsoever".  Every statement is provable in some system, such as a system that includes the statement as an axiom.  
