1. A speech signal classification system, comprising: a speech frame input unit for generating a speech frame by converting a speech signal of a time domain to a speech signal of a frequency domain; a characteristic extractor for extracting characteristic information from the generated speech frame; a primary recognition unit for performing primary recognition using the extracted characteristic information to derive a primary recognition result to be used to determine if the speech frame is a voice sound, an non-voice sound, or background noise; a memory unit for storing characteristic information extracted from the speech frame and at least one other speech frame; a secondary statistical value calculator for calculating secondary statistical values using the stored characteristic information; a secondary recognition unit for performing secondary recognition using the determination result of the speech frame according to the primary recognition result and the secondary statistical values to derive a secondary recognition result to be used to determine if the speech frame is an non-voice sound or background noise; a controller for determining if the speech frame is a voice sound based on the primary recognition result voice sound, and if it is determined that the speech frame is not a voice sound, storing the characteristic information of the speech frame and at least one other speech frame, calculating the secondary statistical values using the stored characteristic information, performing the secondary recognition using the determination result of the speech frame based on the primary recognition result and the secondary statistical values, and determining if the speech frame is an non-voice sound or background noise based on the secondary recognition result; and a classification and output unit for classifying and outputting the speech frame as a voice sound, an non-voice sound, or background noise according to the determination results.