1. A speaker indexing device comprising: a processor configured to: from a speech signal containing a conversation between a plurality of speakers, extract a plurality of features representing human speech features on a frame-by-frame basis, each frame having a predetermined time length; model a distribution of first feature sets that contain at least two of the plurality of features, by a mixture distribution that contains as many probability distributions as the number of speakers; for each of the probability distributions, select from among the first feature sets, either first feature sets located within a predetermined distance from the center of the probability distribution or a predetermined number of first feature sets in sequence starting from the first feature set closest to the center of the probability distribution, and select, from the plurality of features extracted for the frame corresponding to the selected first feature sets, at least one second feature as first training data for speaker of the plurality of speakers that corresponds to the probability distribution; train a speaker model for each of the plurality of speakers by using the first training data selected for the corresponding speaker, the speaker model representing the speech features of the speaker by a probability distribution of the second feature; and based on the speaker model of each of the plurality of speakers and on the second feature selected for each frame, append to each frame identification information for identifying the speaker speaking in the frame.