1. A speech processing apparatus for carrying out text-to-speech synthesis, comprising: an input unit to which a plurality of segments obtained by delimiting a phonological sequence corresponding to a target speech in units of synthesis and prosodic information on the respective segments corresponding to the target speech are entered; a unit selector configured to select a plurality of first speech units from a group of speech units on the basis of the prosodic information for each of the plurality of segments; a decomposer configured to decompose each of the plurality of first speech units into periodic components and aperiodic components for each of the plurality of segments; a periodic component fusing unit configured to generate a second speech unit by fusing the periodic components of the plurality of first speech units for each of the plurality of segments; an aperiodic component fusing unit configured to generate a third speech unit by fusing the aperiodic components of the plurality of first speech units for each of the plurality of segments; and a generator configured to generate a synthesized speech by adding speech waveforms obtained respectively from the second speech unit and the third speech unit generated for each of the plurality of segments and concatenating the same among the segments.