1. A method of recognizing gender or age of a speaker according to speech emotion or arousal, comprising steps of: A) segmentalizing speech signals into a plurality of speech segments; B) fetching the first speech segment from the speech segments to further acquire an arousal degree of the speech segment; B-1) after the first speech segment is fetched from the speech segments, applying a first classification to the arousal degree of the speech segment to enable the arousal to be classified as a high degree or a low degree of arousal; C) if a determination condition is set at a greater-than-threshold condition, proceeding the step D) when the arousal degree of the speech segment is determined greater than the specific threshold, or returning to the step B) when the arousal degree of the speech segment is determined less than or equal to the specific threshold; and if the determination condition is set at a less-than-threshold condition, proceeding to step D) when the arousal degree of the speech segment is determined less than the specific threshold, or returning to the step B) when the arousal degree of the speech segment is determined greater than or equal to the specific threshold; D) fetching a feature indicative of gender or age from the speech segment to further acquire at least one feature parameter corresponding to gender or age; and E) applying recognition to the at least one feature parameter according to a gender or age recognition measure to further determine the gender or age of the speaker in the currently-processed speech segment; next, apply the step B) to the next speech segment, wherein the steps A)-E) are executed by a computer.