Precisely, what is a primitive recursive definition? If I understand correctly, the language of primitive recursive arithmetic has a distinguished function symbol $S$, together with a function symbol for each primitive recursive definition (hereafter PRD).
For example, the PRD $$f(x,0) = x, \quad f(x,S(y)) = S(f(x,y))$$
corresponds to a function symbol that we can denote $+$.
Similarly, the PRD
$$f(x,0) = 0, \quad f(x,S(y)) = f(x,y)+x$$
corresponds to a function symbol that we can denote $\times$.

Question. How do we precisely specify the concept of a PRD?

For example, I'd like to know if there's a formal system that generates all valid definitions by primitive recursion, but no non-valid definitions.

There's a nice treatment in an old book that should be any decent library (also very cheaply available as a Dover reprint), Joel W. Robbin’s Mathematical Logic: A First Course (W. A Benjamin, 1969, Dover reprint 2006: pp. 212)
Robbin defines the primitive recursive functions in the usual way -- $f$ is primitive recursive if there is a finite sequence of functions $f_0, f_1, f_2, \ldots f_n = f$ where each $f_i$ is either an initial function or defined from earlier functions in the sequence by primitive recursion or composition); and then he defines a language which has a function expression for each p.r. function $f$.  The key idea is to have a complex function expression built up to reflect a full definition of $f$ by primitive recursion and/or composition ultimately in terms of the initial functions. So; we have primitive expressions for each initial function, and two forms of constructing new expressions from old corresponding to definitions by primitive recursion and composition. 
The result is that, in your words, in the formal language we can construct a formal expression that fully encapsulates any informal primitive recursive definition. And any expression constructed in the described way will indeed express a proper definitions by primitive recursion (and there be no constructible "non-valid" definitions, just as you want).
Robbin then introduces a system of arithmetic which he calls RA which has axioms for the logic plus axioms governing the expressions for the initial functions, and then there are axioms for dealing with complex functional expressions in terms of their constituents. RA ends up stronger than PRA since Robbin allows stronger induction, but for the purposes of understanding how you get function expressions for each primitive recursive function, his treatment is exemplary, as far as I recall.
