1. A method of recognizing gender or age of a speaker according to speech emotion or arousal, comprising steps of: A) segmentalizing speech signals into a plurality of speech segments; B) fetching the first speech segment from the speech segments to further acquire at least one of the emotional feature or the arousal degree of the speech segment; C) determining at least one of the emotional feature or the arousal degree of the speech segment; if the emotional feature is the object for determination, determine whether the emotional feature belongs to a specific emotion; if the arousal degree is the object for determination, determine whether the arousal degree of the speech segment is greater or less than a specific threshold; if either of the two answers is yes, proceed to the step D); if none of the two answers is yes, return to the step B) and then fetch the next speech segment; D) fetching the feature indicative of gender or age from the speech segment to further acquire at least one feature parameter corresponding to gender or age; and E) applying recognition to the at least one feature parameter according to a gender or age recognition measure to further determine the gender or age of the speaker in the currently-processed speech segment; next, apply the step B) to the next speech segment.