One important reason for applying soft attention instead of its hard version is that the stochastic hard attention mechanism is difficult to train. Although the REINFORCE-like algorithms [1]} are unbiased estimators to train stochastic units, their gradients have high variants. To solve the problem, recently, Jang et al. [2]} proposed a novel categorical re-parameterization techniques using Gumbel-softmax distribution. The Gumbel-softmax is a superior estimator for categorical discrete units [2]}. It has been proved to be efficient and has high performance in [2]}.
