Is there a way to determine the prefactor of the trace of the product of two $SU(N)$ matrices? While renormalizing QCD I found the follow equation:
$$tr(T_aT_b) = n(r)\delta_{ab},$$
where $T_a$ is the generic SU(N) generator and $n(r)$ is a factor depending on the algebra representation.
In the defining representation of $SU(3)$ the generators are
$$T_a = \frac{\lambda_a}{2},$$
where $\lambda_a$ are the eight well know Gell-Mann matrices; in this case it's easy to verify that $n(r) = \frac{1}{2}$. I was thus wondering: what is the formal demonstration of this relation? And is there a way to determine $n(r)$ for generic matricial representation and generic N, or we are bound to know the explicit form of the generators in order to obtain it?
Yes, following the  discussion below, all such indices for all reps can be gotten from self-consistency in tensoring the defining/fundamental with itself, its conjugate, the adjoint, etc. You do not need to know the explicit form of the generator matrices in a given representation--only its provenance out of fundamentals and antifundamentals. Here is how it works for your case of the fundamental.

It is a one-line proof, but you are meant to be familiar with the language, or some equivalent thereof. Let's call F the fundamental representation N×N matrices, A the adjoint (N²—1)×(N²—1) ones, and 0 the 1-dim singlet rep ones. 

The Kronecker product fundamental-antifundamental (N×N)⊗(N×N) representation matrices are given by the coproduct, 
$$
F^a\otimes 1\!\!1 + 1\!\!1 \otimes \bar F^a = A^a  \oplus 0 = A^a, 
$$
which you must check, indeed, satisfies the same Lie algebra, 
$$
[(F^a\otimes 1\!\!1 + 1\!\!1 \otimes \bar F^a) ,(F^b\otimes 1\!\!1 + 1\!\!1 \otimes \bar F^b)]=if^{abc}(F^c\otimes 1\!\!1 + 1\!\!1 \otimes \bar F^c).
$$

Now multiply this with its a ⟶ b analog, 
$$
(F^a\otimes 1\!\!1 + 1\!\!1 \otimes \bar F^a) (F^b\otimes 1\!\!1 + 1\!\!1 \otimes \bar F^b) = A^a    A^b= \\ 
F^a F^b\otimes 1\!\!1 + 1\!\!1 \otimes \bar F^a \bar F^b +  F^a\otimes \bar F^b + F^b\otimes \bar F^a, 
$$
and trace. The trace of the direct product amounts to a plain product of the separate traces of each tensor factor, of course, so the last two terms vanish, as the F s are traceless.

You thus establish in a blink that 
$$
2N\operatorname{tr} F^a F^b = \operatorname{tr} A^a A^b .
$$
If the trace on the right is N, the trace on the left must be 1/2, your sought answer. 

In physics, the r.h.s. is normalized to be N, as the structure constants providing the adjoint rep are chosen to be such. (Not in mathematics!)

Mutatis mutandis you know the trace of all irreps must be fixed w.r.t. the adjoint normalization:  scaling up the generators of the algebra results in an identical rescaling of the structure constants, which we already fixed. 

The reason for these normalization choices, despised by mathematicians (who normalize the linchpin structure constant f  with 1), is because  physicists want all SU(N) s to include an isospin SU(2) with its conventional spin 1/2 normalizations, for ease in particle physics applications. (The mayhem started with flavor SU(3) and lives on.)

To  reassure yourself are in control of the algorithm, try composing ${\bf 3}\otimes {\bf 3}= {\bf 6}\oplus \bar{\bf 3}$ in SU(3) and compute this trace for the sextet: You should find 3 — 1/2=5/2. (It is half the representation index you find in math texts.) 

Standard  physics (QFT) texts like Peskin &amp; Schroeder, An Introduction To Quantum Field Theory (1995), §15.4, review the language.
You have to know the form of the generators, because it depends on them.

Simply put, if you assume $a=b$, then it's $tr(T_a T_b) = n(r)$ (which is obvious
but the closest you can get for a Formula for $n(r)$, where its implicitly assumed that $tr(T_a T_a) = tr(T_bT_b) \quad\forall a,b$, which is no problem after you scale the basis vectors (''Generators'') correctly).

I am not sure about the other Questions of yours (in the sense that I don't know
what answer you're seeking for).
What might help you is getting familiar with the so called ''Cartan-Weyl-Basis'',
and judging from the style in which your question was written, I assume you
are a physicist, so the following book might help you: 
''Shattered Symmetry Group Theory from the Eightfold Way to the Perdiodic Table'',
by Pieter Thyssen and Arnout Ceulemans. 

Finally, observe that in the equation you have written, there does not appear to be any given representation (besides, maybe, the fundamental one), so the asking for it in an arbitrary representation seems kind of pointless to me...