1. A voice activity segmentation device comprising: a processor; and a memory, wherein the memory is configured to store and the processor is configured to implement: a first voice activity segmentation unit which determines a voice-active segment, which is a first voice-active segment, and a voice-inactive segment, which is a first voice-inactive segment, in a time-series of input sound by comparing a threshold value and a feature value of the time-series of the input sound; a second voice activity segmentation unit which determines, after a reference speech acquired from a reference speech storage unit has been superimposed on a time-series of the first voice-inactive segment, a voice-active segment and a voice-inactive segment in the time-series of the superimposed first voice-inactive segment by comparing the threshold value and a feature value of the time-series of the superimposed first voice-inactive segment; and a threshold value update unit which updates the threshold value in such a way that a discrepancy rate between the determination result of the second voice activity segmentation unit and a correct segmentation calculated from the reference speech is decreased.