1. A speaker indexing device comprising: a processor configured to: from a speech signal containing a conversation between a plurality of speakers, extract at least one feature and a data set including at least two values on a frame-by-frame basis, each frame having a predetermined time length, the at least one feature and the data set representing human speech features; model a distribution of the data set extracted for each frame, by a mixture distribution that contains as many probability distributions as the number of speakers; for each of the probability distributions, select from among the data set extracted for each frame, either data sets located within a predetermined distance from the center of the probability distribution or a predetermined number of data sets in sequence starting from the data set closest to the center of the probability distribution, and select the at least one feature extracted for the frame corresponding to each selected data set, as first training data for speaker of the plurality of speakers that corresponds to the probability distribution; train a speaker model for each of the plurality of speakers by using the first training data selected for the corresponding speaker, the speaker model representing the speech features of the speaker by a probability distribution of the at least one feature; and based on the speaker model of each of the plurality of speakers and on the at least one feature for each frame, append to each frame identification information for identifying the speaker speaking in the frame.