The idea of allowing dynamic program halting behavior in recurrent neural networks was initially explored in [1]}.
Here, a special `halting' unit, when activated, resulted in termination of the program and emission of program outputs.
Similarly, the Adaptive Computation Time (ACT) algorithm [2]} uses a sigmoidal `halting unit' (and associated weights) for halting program execution, which is regularized to penalize the amount of computation used.
The ACT algorithm for halting has later been adapted to the Transformer family of models as well [3]}.
PonderNet [4]} builds on ACT but crucially differs in its use of a probabilistic halting strategy.
It defines a probability distribution over all possible computation steps available and weights the output prediction loss at each step with their associated halting probability.
Further, PonderNet uses a KL-divergence between the posterior distribution over the number of computation steps and a prior distribution as a regularizer.
