Many sequential data often have complex temporal structure which requires both hierarchical and multi-scale information to be modeled properly. In language modeling, a long sentence is often composed of many phrases which further can be decomposed by words. Meanwhile, in action recognition, an action category can be described by many sub-actions. For instance, `long jump' contains `running', `jumping' and `landing'. As stated in [1]}, a promising approach to model such hierarchical representation is the multi-scale RNN. One popular approach of implementing multi-scale RNN is to treat the hierarchical timescales as pre-defined parameters. For example, Wang et al. [2]} implemented a multi-scale architecture by building a multiple layers LSTM in which higher layer skip several time steps. In their paper, the skipped number of time steps is the parameter to be pre-defined. However, it is often impractical to pre-define such timescales without learning, which also leads to poor generalization capability. Chung et al. [1]} proposed a novel RNN structure, Hierarchical Multi-scale Recurrent Neural Network (HM-RNN), to learn time boundaries from data automatically. These temporal boundaries are like rules described by discrete variables inside RNN cells. Normally, it is difficult to implement training algorithms for discrete variables.
Popular approaches include unbiased estimator with the aid of REINFORCE [4]}. In this paper, we re-implement the HM-RNN by applying a recently proposed Gumbel-sigmoid function [5]} [6]} to realize the training of stochastic neurons due to its efficiency in practice [7]}.
