1. An HMM (Hidden Markov Model) learning device comprising: learning means configured to learn a state transition probability as the function of actions that an agent can execute, with learning with HMM performed based on actions that said agent has executed, and time series information made up of an observation signal observed as a result of said actions; and storage means configured to store learning results by said learning means as internal model data including a state transition probability table and an observation probability table; wherein said learning means calculate a frequency variable equivalent to frequency to be used for estimation calculation of an HMM state transition probability, and a frequency variable equivalent to frequency to be used for estimation calculation of an HMM observation probability; and wherein said storage means hold said frequency variable corresponding to each of state transition probabilities, and said frequency variable corresponding to each of observation probabilities, of said state transition probability table; and wherein said learning means use said frequency variables held by said storage means to perform learning by an additional learning method, and estimate said state transition probability and said observation probability based on said frequency variables.